Wednesday, December 22, 2010

PyPy 1.4.1

Release announcement

We're pleased to announce
the 1.4.1 release of PyPy.
This release consolidates all the bug fixes that occurred since the
previous release. To everyone that took the trouble to report
them, we want to say thank you.

What is PyPy

PyPy is a very compliant Python interpreter, almost a drop-in
replacement for CPython. Note that it still only emulates Python
2.5 by default; the fast-forward branch with Python 2.7
support is slowly getting ready but will only be integrated in
the next release.

In two words, the advantage of trying out PyPy instead of CPython
(the default implementation of Python) is, for now, the
performance. Not all programs are faster in PyPy, but we are
confident that any CPU-intensive task will be much faster, at
least if it runs for long enough (the JIT has a slow warm-up
phase, which can take several seconds or even one minute on the
largest programs).

Note again that we do support compiling and using C extension
modules from CPython (pypy setup.py install). However, this
is still an alpha feature, and the most complex modules typically
fail for various reasons; others work (e.g. PIL) but take a
serious performance hit. Also, for Mac OS X see below.

Please note also that PyPy's performance was optimized almost
exclusively on Linux. It seems from some reports that on Windows
as well as Mac OS X (probably for different reasons) the
performance might be lower. We did not investigate much so far.

More highlights

We migrated to Mercurial (thanks to Ronny Pfannschmidt and
Antonio Cuni) for the effort) and moved to bitbucket. The new
command to check out a copy of PyPy is:
hg clone http://bitbucket.org/pypy/pypy

In long-running processes, the assembler generated by old
JIT-compilations is now freed. There should be no more leak,
however long the process runs.

Improve a lot the performance of the binascii module, and
of hashlib.md5 and hashlib.sha.

Made sys.setrecursionlimit() a no-op. Instead, we rely purely
on the built-in stack overflow detection mechanism, which also
gives you a RuntimeError -- just not at some exact recursion
level.

Fix argument processing (now e.g. pypy -OScpass works like
it does on CPython --- if you have a clue what it does there
:-) )

cpyext on Mac OS X: it still does not seem to work. I get
systematically a segfault in dlopen(). Contributions welcome.

Fix two corner cases in the GC (one in minimark, one in
asmgcc+JIT). This notably prevented pypy translate.py -Ojit
from working on Windows, leading to crashes.

Fixed a corner case in the JIT's optimizer, leading to Fatal
RPython error: AssertionError.

Added some missing built-in functions into the 'os' module.

Fix ctypes (it was not propagating keepalive information from
c_void_p).

Release announcement

We're pleased to announce
the 1.4.1 release of PyPy.
This release consolidates all the bug fixes that occurred since the
previous release. To everyone that took the trouble to report
them, we want to say thank you.

What is PyPy

PyPy is a very compliant Python interpreter, almost a drop-in
replacement for CPython. Note that it still only emulates Python
2.5 by default; the fast-forward branch with Python 2.7
support is slowly getting ready but will only be integrated in
the next release.

In two words, the advantage of trying out PyPy instead of CPython
(the default implementation of Python) is, for now, the
performance. Not all programs are faster in PyPy, but we are
confident that any CPU-intensive task will be much faster, at
least if it runs for long enough (the JIT has a slow warm-up
phase, which can take several seconds or even one minute on the
largest programs).

Note again that we do support compiling and using C extension
modules from CPython (pypy setup.py install). However, this
is still an alpha feature, and the most complex modules typically
fail for various reasons; others work (e.g. PIL) but take a
serious performance hit. Also, for Mac OS X see below.

Please note also that PyPy's performance was optimized almost
exclusively on Linux. It seems from some reports that on Windows
as well as Mac OS X (probably for different reasons) the
performance might be lower. We did not investigate much so far.

More highlights

We migrated to Mercurial (thanks to Ronny Pfannschmidt and
Antonio Cuni) for the effort) and moved to bitbucket. The new
command to check out a copy of PyPy is:
hg clone http://bitbucket.org/pypy/pypy

In long-running processes, the assembler generated by old
JIT-compilations is now freed. There should be no more leak,
however long the process runs.

Improve a lot the performance of the binascii module, and
of hashlib.md5 and hashlib.sha.

Made sys.setrecursionlimit() a no-op. Instead, we rely purely
on the built-in stack overflow detection mechanism, which also
gives you a RuntimeError -- just not at some exact recursion
level.

Fix argument processing (now e.g. pypy -OScpass works like
it does on CPython --- if you have a clue what it does there
:-) )

cpyext on Mac OS X: it still does not seem to work. I get
systematically a segfault in dlopen(). Contributions welcome.

Fix two corner cases in the GC (one in minimark, one in
asmgcc+JIT). This notably prevented pypy translate.py -Ojit
from working on Windows, leading to crashes.

Fixed a corner case in the JIT's optimizer, leading to Fatal
RPython error: AssertionError.

Added some missing built-in functions into the 'os' module.

Fix ctypes (it was not propagating keepalive information from
c_void_p).

8 comments:

There are still a number of branches that have not been merged into trunk yet: at least fast-forward (Python 2.7), jit-unroll-loops (better JITting of arithmetic and short loops), arm-backend (JIT support on ARM) and jitypes2 (turn ctypes calls into real assembler-level calls with the JIT). There is also the stackless+JIT integration pending. Finally the sprint will also be a place to try out and run some applications. So it's not like we are out of work :-)

Actually, hashlib.sha was not the same as sha.sha: the former used to be a ctypes call to the OpenSSL lib, whereas the latter uses our built-in sha implementation. So hashlib.sha was faster in theory, but killed by the overhead of using ctypes. Now, at least in a default version of pypy, the hashlib.md5 and .sha are redirected to the built-in md5.md5 and sha.sha.

Another issue was that with the built-in md5.md5 and sha.sha, on 64-bit, there was a 1.5x speed impact due to the C compiler not recognizing an expression that was meant to be a 32-bit integer rotation.

I guess that http://speed.pypy.org don't show this because they use directly md5.md5 or sha.sha, and are on 32-bit.

Andrei: not directly. We played at some point with translating RPython code to Javascript, but it didn't give enough benefits (because it's not full Python that we can translate, just "RPython"). The alternative would be to translate the whole PyPy interpreter to Javascript, but that would give a result that is both huge (in term of download size) and horribly slow (100x slower than Javascript maybe).