Navigation

This document describes IPython from the perspective of developers. Most
importantly, it gives information for people who want to contribute to the
development of IPython. So if you want to help out, read on!

IPython development is done using Bazaar [Bazaar] and Launchpad [Launchpad].
This makes it easy for people to contribute to the development of IPython.
There are several ways in which you can join in.

If you have a small change that you want to send to the team, you can edit your
bazaar checkout of IPython (see below) in-place, and ask bazaar for the
differences:

$ cd /path/to/your/copy/of/ipython
$ bzr diff > my_fixes.diff

This produces a patch file with your fixes, which we can apply to the source
tree. This file should then be attached to a ticket in our bug tracker, indicating what it does.

This model of creating small, self-contained patches works very well and there
are open source projects that do their entire development this way. However,
in IPython we have found that for tracking larger changes, making use of
bazaar’s full capabilities in conjunction with Launchpad’s code hosting
services makes for a much better experience.

Making your own branch of IPython allows you to refine your changes over time,
track the development of the main team, and propose your own full version of
the code for others to use and review, with a minimum amount of fuss. The next
parts of this document will explain how to do this.

When working on IPython, you won’t actually make edits directly to the
lp:ipython branch. Instead, you will create a separate branch for your
changes. For now, let’s assume you want to do your work in a branch named
“ipython-mybranch”. Create this branch by doing:

$ bzr branch ipython ipython-mybranch

When you actually create a branch, you will want to give it a name that
reflects the nature of the work that you will be doing in it, like
“install-docs-update”.

Now you are ready to actually make edits in your ipython-mybranch
branch. Before doing this, it is helpful to install this branch so you can
test your changes as you work. This is easiest if you have setuptools
installed. Then, just do:

$ cd ipython-mybranch
$ python setupegg.py develop

Now, make some changes. After a while, you will want to commit your changes.
This let’s Bazaar know that you like the changes you have made and gives you
an opportunity to keep a nice record of what you have done. This looks like
this:

Please note that since we now don’t use an old-style linear ChangeLog (that
tends to cause problems with distributed version control systems), you should
ensure that your log messages are reasonably detailed. Use a docstring-like
approach in the commit messages (including the second line being left
blank):

Single line summary of changes being committed.
* more details when warranted ...
* including crediting outside contributors if they sent the
code/bug/idea!

As you work, you will repeat this edit/commit cycle many times. If you work on
your branch for a long time, you will also want to get the latest changes from
the lp:ipython branch. This can be done with the following sequence of
commands:

Once you are done with your edits, you should post your branch on Launchpad so
that other IPython developers can review the changes and help you merge your
changes into the main development branch. To post your branch on Launchpad,
do:

Then, go to the IPython Launchpad site, and you
should see your branch under the “Code” tab. If you click on your branch, you
can provide a short description of the branch as well as mark its status. Most
importantly, you should click the link that reads “Propose for merging into
another branch”. What does this do?

This let’s the other IPython developers know that your branch is ready to be
reviewed and merged into the main development branch. During this review
process, other developers will give you feedback and help you get your code
ready to be merged. What types of things will we be looking for:

All code is documented.

All code has tests.

The entire IPython test suite passes.

Once your changes have been reviewed and approved, someone will merge them
into the main development branch.

Some notes for core developers when merging third-party contributions¶

Core developers, who ultimately merge any approved branch (from themselves,
another developer, or any third-party contribution) will typically use
bzr merge to merge the branch into the trunk and push it to the main
Launcphad site. This is a short list of things to keep in mind when doing this
process, so that the project history is easy to understand in the long run, and
that generating release notes is as painless and accurate as possible.

When you merge any non-trivial functionality (from one small bug fix to a big
feature branch), please remember to always edit the changes_ file
accordingly. This file has one main section for each release, and if you
edit it as you go, noting what new features, bug fixes or API changes you
have made, the release notes will be almost finished when they are needed
later. This is much easier if done when you merge the work, rather than
weeks or months later by re-reading a massive Bazaar log.

When big merges are done, the practice of putting a summary commit message in
the merge is extremely useful. It makes this kind of job much nicer,
because that summary log message can be almost copy/pasted without changes,
if it was well written, rather than dissecting the next-level messages from
the individual commits.

It’s important that we remember to always credit who gave us something if
it’s not the committer. In general, we have been fairly good on this front,
this is just a reminder to keep things up. As a note, if you are ever
committing something that is completely (or almost so) a third-party
contribution, do the commit as:

$ bzr commit --author="Someone Else"

This way it will show that name separately in the log, which makes it even
easier to spot. Obviously we often rework third party contributions
extensively, but this is still good to keep in mind for cases when we don’t
touch the code too much.

All standalone documentation should be written in plain text (.txt) files
using reStructuredText [reStructuredText] for markup and formatting. All such
documentation should be placed in directory docs/source of the IPython
source tree. The documentation in this location will serve as the main source
for IPython documentation and all existing documentation should be converted
to this format.

To build the final documentation, we use Sphinx [Sphinx]. Once you have
Sphinx installed, you can build the html docs yourself by doing:

Good docstrings are very important. All new code should have docstrings that
are formatted using reStructuredText for markup and formatting, since it is
understood by a wide variety of tools. Details about using reStructuredText
for docstrings can be found here.

There are, however, some important exceptions to these rules. In some cases,
IPython code will interface with packages (Twisted, Wx, Qt) that use other
conventions. At some level this makes it impossible to adhere to our own
standards at all times. In particular, when subclassing classes that use other
naming conventions, you must follow their naming conventions. To deal with
cases like this, we propose the following policy:

If you are subclassing a class that uses different conventions, use its
naming conventions throughout your subclass. Thus, if you are creating a
Twisted Protocol class, used Twisted’s
namingSchemeForMethodsAndAttributes.

All IPython’s official interfaces should use our conventions. In some cases
this will mean that you need to provide shadow names (first implement
fooBar and then foo_bar=fooBar). We want to avoid this at all
costs, but it will probably be necessary at times. But, please use this
sparingly!

Implementation-specific private methods will use
_single_underscore_prefix. Names with a leading double underscore will
only be used in special cases, as they makes subclassing difficult (such
names are not easily seen by child classes).

Occasionally some run-in lowercase names are used, but mostly for very short
names or where we are implementing methods very similar to existing ones in a
base class (like runlines() where runsource() and runcode() had
established precedent).

The old IPython codebase has a big mix of classes and modules prefixed with an
explicit IP. In Python this is mostly unnecessary, redundant and frowned
upon, as namespaces offer cleaner prefixing. The only case where this approach
is justified is for classes which are expected to be imported into external
namespaces and a very generic name (like Shell) is too likely to clash with
something else. We’ll need to revisit this issue as we clean up and refactor
the code, but in general we should remove as many unnecessary IP/ip
prefixes as possible. However, if a prefix seems absolutely necessary the more
specific IPY or ipy are preferred.

It is extremely important that all code contributed to IPython has tests.
Tests should be written as unittests, doctests or as entities that the Nose
[Nose] testing package will find. Regardless of how the tests are written, we
will use Nose for discovering and running the tests. Nose will be required to
run the IPython test suite, but will not be required to simply use IPython.

Tests of Twisted using code need to follow two additional guidelines:

Twisted using tests should be written by subclassing the TestCase
class that comes with twisted.trial.unittest.

All Deferred instances that are created in the test must be
properly chained and the final one must be the return value of the test
method.

When these two things are done, Nose will be able to run the tests and the
twisted reactor will be handled correctly.

Each subpackage in IPython should have its own tests directory that
contains all of the tests for that subpackage. This allows each subpackage to
be self-contained. A good convention to follow is to have a file named
test_foo.py for each module foo.py in the package. This makes
it easy to organize the tests, though like most conventions, it’s OK to break
it if logic and common sense dictate otherwise.

If a subpackage has any dependencies beyond the Python standard library, the
tests for that subpackage should be skipped if the dependencies are not
found. This is very important so users don’t get tests failing simply because
they don’t have dependencies. We ship a set of decorators in the
IPython.testing package to tag tests that may be platform-specific or
otherwise may have restrictions; if the existing ones don’t fit your needs, add
a new decorator in that location so other tests can reuse it.

To run the IPython test suite, use the iptest command that is
installed with IPython (if you are using IPython in-place, without installing
it, you can find this script in the scripts directory):

$ iptest

This command colects all IPython tests into separate groups, and then calls
either Nose with the proper options and extensions, or Twisted’s
trial. This ensures that tests that need the Twisted reactor
management facilities execute separate of Nose. If any individual test group
fails, iptest will print what you need to type so you can rerun that
particular test group alone for debugging.

By default, iptest runs the entire IPython test
suite (skipping tests that may be platform-specific or which depend on tools
you may not have). But you can also use it to run only one specific test file,
or a specific test function. For example, this will run only the
test_magic file from the test suite:

Since iptest is based on nosetests, you can pass it any regular
nosetests option. For example, you can use --pdb or --pdb-failures to
automatically activate the interactive Pdb debugger on errors or failures. See
the nosetests documentation for further details.

You can write tests either as normal test files, using all the conventions that
Nose recognizes, or as doctests. Note that all IPython functions should have
at least one example that serves as a doctest, whenever technically feasible.
However, example doctests should only be in the main docstring if they are a
good example, i.e. if they convey useful information about the function. If
you simply would like to write a test as a doctest, put it in a separate test
file and write a no-op function whose only purpose is its docstring.

Note, however, that in a file named test_X, functions whose only test
is their docstring (as a doctest) and which have no test functionality of their
own, should be called doctest_foo instead of test_foo, otherwise they get
double-counted (the empty function call is counted as a test, which just
inflates tests numbers artificially). This restriction does not apply to
functions in files with other names, due to how Nose discovers tests.

You can use IPython examples in your docstrings. Those can make full use of
IPython functionality (magics, variable substitution, etc), but be careful to
keep them generic enough that they run identically on all Operating Systems.

The prompts in your doctests can be either of the plain Python >>> variety
or In[1]: IPython style. Since this is the IPython system, after all, we
encourage you to use IPython prompts throughout, unless you are illustrating a
specific aspect of the normal prompts (such as the %doctest_mode magic).

If a test isn’t safe to run inside the main nose process (e.g. because it loads
a GUI toolkit), consider running it in a subprocess and capturing its output
for evaluation and test decision later. Here is an example of how to do it, by
relying on the builtin _ip object that contains the public IPython api as
defined in IPython.ipapi:

deftest_obj_del():"""Test that object's __del__ methods are called on exit."""test_dir=os.path.dirname(__file__)del_file=os.path.join(test_dir,'obj_del.py')out=_ip.IP.getoutput('ipython %s'%del_file)nt.assert_equals(out,'object A deleted')

If a doctest contains input whose output you don’t want to verify identically
via doctest (random output, an object id, etc), you can mark a docstring with
#random. All of these test will have their code executed but no output
checking will be done:

>>> 1+3junk goes here... # random>>> 1+2again, anything goes #randomif multiline, the random mark is only needed once.>>> 1+2You can also put the random marker at the end:# random>>> 1+2# random.. or at the beginning.

In a case where you want an entire docstring to be executed but not verified
(this only serves to check that the code runs without crashing, so it should be
used very sparingly), you can put #all-random in the docstring.

Most of the release process is automated by the release script in the
tools directory. This is just a handy reminder for the release manager.

First, run build_release, which does all the file checking and
building that the real release script will do. This will let you do test
installations, check that the build procedure runs OK, etc. You may want to
disable a few things like multi-version RPM building while testing, because
otherwise the build takes really long.

Run the release script, which makes the tar.gz, eggs and Win32 .exe
installer. It posts them to the site and registers the release with PyPI.

Updating the website with announcements and links to the updated
changes.txt in html form. Remember to put a short note both on the news
page of the site and on Launcphad.

Drafting a short release announcement with i) highlights and ii) a link to
the html changes.txt.

There are no definite plans for porting of IPython to python 3. The major
issue is the dependency on twisted framework for the networking/threading
stuff. It is possible that it the traditional IPython interactive console
could be ported more easily since it has no such dependency. Here are a few
things that will need to be considered when doing such a port especially
if we want to have a codebase that works directly on both 2.x and 3.x.

The syntax for exceptions changed (PEP 3110). The old
except exc, var changed to except exc as var. At last
count there was 78 occurences of this usage in the codebase. This
is a particularly problematic issue, because it’s not easy to
implement it in a 2.5-compatible way.

Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we
will likely at some point put out a release that requires strictly 2.6 and
abandons 2.5 compatibility. This will then allow us to port the code to using
print() as a function, except exc as var syntax, etc. But as of
version 0.11 at least, we will retain Python 2.5 compatibility.