Playing with PyPy

»

I've been following the PyPy project since I first heard of it in 2003 or
so. The concept behind it is fascinating; it's a Python interpreter written
in (a subset of) Python. It's actually a lot more than that because the
language front-ends (e.g. Python) are quite separate from the backends
(e.g. C, JVM, CLI, Python). This makes it a unique platform for language
research because coding Python is typically easier than C, and so much
of the work is already done for you.

It's clear that PyPy is very useful for academic research, but it's also
quickly becoming a practical target for developing and deploying Python
code. At the PyPy Speed Center you can see that it's already several
times faster than CPython, and has the potential to fix most of the more
fundamental flaws of the CPython VM.

What's awesome right now:

PyPy has a modern garbage collector, not ref counting

PyPy's JIT can run string mangling and numerics code
very quickly, which removes the need for most C extensions

PyPy is already fast, and is getting faster all the time

How it can be more awesome (just my opinion, I don't speak for the PyPy
team and their implementation goals):

PyPy has an alpha quality cpyext that will allow you to use CPython
extensions (requires a recompile), and when that's polished it will be very
easy for CPython users to migrate en masse, even though they may have
complicated dependencies such as NumPy, SciPy, PIL, etc.

PyPy has the potential to eventually remove the GIL, and/or
have multiple VMs in the same OS process

PyPy could add M:N threading and concurrency constructs to the
language (some stackless support already exists, but is not
currently compatible with the JIT and doesn't take advantage
of multiple cores)

PyPy could simultaneously support Python 2.x and 3.x code in the
same process, making it practical to actually make the transition
(note: this is a crazy idea that would be terribly difficult)

Playing with PyPy

I've been working on helping the PyPy team with some real world benchmarks
for JSON, and helping sort out Mac OS X issues. I've also been tuning a
branch of simplejson to run efficiently on PyPy. I'll write more about
this in a follow up post, but here's how you can get started.

If you're a library author or an advanced user you should be experimenting
with PyPy right now. In these instructions we'll install PyPy in ~/opt
and create a virtualenv for it in ~/virtualenv.

# Make a ~/src to store all of these clones
mkdir -p ~/src
# Get the PyPy jitviewer application (install to ~/src/jitviewer)
(cd ~/src; hg clone https://bitbucket.org/pypy/jitviewer)
# Clone PyPy source, this will take a while
(cd ~/src; hg clone https://bitbucket.org/pypy/pypy)

Installing from binary

Follow the link on http://pypy.org/download.html to download one of the
Mac OS X binaries (either 32-bit or 64-bit), the current version at this
time is 1.4.1.

source ~/virtualenv/pypy-env/bin/activate
# now you can use PyPy! both "python" and "pypy" will work

Tuning for PyPy 1.4.1

On Mac OS X, PyPy 1.4.1 (and current default) does not choose optimal tuning
values for the GC. You will get ~30% better performance by setting this
environment variable:

export PYPY_GC_NURSERY=1M

Note that 1M is a machine specific value, so if your Mac isn't the same model
as mine there might be a better default for you. The value is very likely to
be dependent on the amount of L2/L3 cache and how
many physical cores you have, and you can get those values from sysctl:

The PyPy team is very interested in knowing what the sysctl values are for your
machine and the output of the GC benchmark, so if you get this far please send it
along to me or the PyPy mailing list! Having output from many different models of
Mac will help us come up with a better algorithm for choosing sane defaults.

Building PyPy from source

Make sure to install a binary first. Since translating PyPy is CPU bound,
this runs a lot faster if you use PyPy.

These commands will build PyPy, create a release based on the hg revision,
update ~/virtualenv/pypy-env, etc.: