Saturday, March 19, 2016

PyPy 5.0.1

We have released a bugfix for PyPy 5.0, after reports that the newly released
lxml 3.6.0, which now supports PyPy 5.0 +, can crash on large files.
Thanks to those who reported the crash. Please update, downloads are available
at

The changes between PyPy 5.0 and 5.0.1 are only two bug fixes: one in
cpyext, which fixes notably (but not only) lxml; and another for a
corner case of the JIT.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7. It’s fast (PyPy and CPython 2.7.x performance comparison)
due to its integrated tracing JIT compiler.
We also welcome developers of other
dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems
(Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD),
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and the
big- and little-endian variants of PPC64 running Linux.

Please update, and continue to help us make PyPy better.

Cheers
The PyPy Team

PyPy 5.0.1

We have released a bugfix for PyPy 5.0, after reports that the newly released
lxml 3.6.0, which now supports PyPy 5.0 +, can crash on large files.
Thanks to those who reported the crash. Please update, downloads are available
at

The changes between PyPy 5.0 and 5.0.1 are only two bug fixes: one in
cpyext, which fixes notably (but not only) lxml; and another for a
corner case of the JIT.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7. It’s fast (PyPy and CPython 2.7.x performance comparison)
due to its integrated tracing JIT compiler.
We also welcome developers of other
dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems
(Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD),
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and the
big- and little-endian variants of PPC64 running Linux.

PyPy 5.0

We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

Faster and Leaner

We continue to improve the warmup time and memory usage of JIT-related metadata. The exact effects depend vastly on the program you’re running and can range from insignificant to warmup being up to 30% faster and memory dropping by about 30%.

C-API Upgrade

We also merged a major upgrade to our C-API layer (cpyext), simplifying the interaction between c-level objects and PyPy interpreter level objects. As a result, lxml (prerelease) with its cython compiled component passes all tests on PyPy. The new cpyext is also much faster. This major refactoring will soon be followed by an expansion of our C-API compatibility.

Profiling with vmprof supported on more platforms

vmprof has been a go-to profiler for PyPy on linux for a few releases and we’re happy to announce that thanks to the cooperation with jetbrains, vmprof now works on Linux, OS X and Windows on both PyPy and CPython.

CFFI

While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. PyPy 5.0 ships with cffi-1.5.2 which now allows embedding PyPy (or CPython) in a C program.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (pypy and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, freebsd), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and 64 bit PowerPC hardware, specifically Linux running the big- and little-endian variants of ppc64.

Other Highlights (since 4.0.1 released in November 2015)

New features:

Support embedding PyPy in a C-program via cffi and static callbacks in cffi.
This deprecates the old method of embedding PyPy

Refactor vmprof to work cross-operating-system, deprecate using buggy
libunwind on Linux platforms. Vmprof even works on Windows now.

Support more of the C-API type slots, like tp_getattro, and fix C-API
macros, functions, and structs such as _PyLong_FromByteArray(),
PyString_GET_SIZE, f_locals in PyFrameObject, Py_NAN, co_filename in
PyCodeObject

Use a more stable approach for allocating PyObjects in cpyext. (seeblog post). Once the PyObject corresponding to a PyPy object is created,
it stays around at the same location until the death of the PyPy object.
Done with a little bit of custom GC support. It allows us to kill the
notion of “borrowing” inside cpyext, reduces 4 dictionaries down to 1, and
significantly simplifies the whole approach (which is why it is a new
feature while technically a refactoring) and allows PyPy to support the
populart lxml module (as of the next release) with no PyPy specific
patches needed

Backport always using os.urandom for uuid4 from cpython and fix the JIT as well
(issue #2202)

More completely support datetime, optimize timedelta creation

Fix for issue #2185 which caused an inconsistent list of operations to be
generated by the unroller, appeared in a complicated DJango app

Fix an elusive issue with stacklets on shadowstack which showed up when
forgetting stacklets without resuming them

Fix entrypoint() which now acquires the GIL

Fix direct_ffi_call() so failure does not bail out before setting CALL_MAY_FORCE

Fix (de)pickling long values by simplifying the implementation

Fix RPython rthread so that objects stored as threadlocal do not force minor
GC collection and are kept alive automatically. This improves perfomance of
short-running Python callbacks and prevents resetting such object between
calls

Support floats as parameters to itertools.isslice()

Check for the existence of CODESET, ignoring it should have prevented PyPy
from working on FreeBSD

Fix for corner case (likely shown by Krakatau) for consecutive guards with
interdependencies

Updates to numpy 1.10.2 (incompatibilities and not-implemented features
still exist)

Support dtype=((‘O’, spec)) union while disallowing record arrays with
mixed object, non-object values

Remove all traces of micronumpy from cpyext if –withoutmod-micronumpy option used

Support indexing filtering with a boolean ndarray

Support partition() as an app-level function, together with a cffi wrapper
in pypy/numpy, this now provides partial support for partition()

Performance improvements:

Optimize global lookups

Improve the memory signature of numbering instances in the JIT. This should
massively decrease the amount of memory consumed by the JIT, which is
significant for most programs. Also compress the numberings using variable-
size encoding

Optimize string concatenation

Use INT_LSHIFT instead of INT_MUL when possible

Improve struct.unpack by casting directly from the underlying buffer.
Unpacking floats and doubles is about 15 times faster, and integer types
about 50% faster (on 64 bit integers). This was then subsequently
improved further in optimizeopt.py.

PyPy 5.0

We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

Faster and Leaner

We continue to improve the warmup time and memory usage of JIT-related metadata. The exact effects depend vastly on the program you’re running and can range from insignificant to warmup being up to 30% faster and memory dropping by about 30%.

C-API Upgrade

We also merged a major upgrade to our C-API layer (cpyext), simplifying the interaction between c-level objects and PyPy interpreter level objects. As a result, lxml (prerelease) with its cython compiled component passes all tests on PyPy. The new cpyext is also much faster. This major refactoring will soon be followed by an expansion of our C-API compatibility.

Profiling with vmprof supported on more platforms

vmprof has been a go-to profiler for PyPy on linux for a few releases and we’re happy to announce that thanks to the cooperation with jetbrains, vmprof now works on Linux, OS X and Windows on both PyPy and CPython.

CFFI

While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. PyPy 5.0 ships with cffi-1.5.2 which now allows embedding PyPy (or CPython) in a C program.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (pypy and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, freebsd), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and 64 bit PowerPC hardware, specifically Linux running the big- and little-endian variants of ppc64.

Other Highlights (since 4.0.1 released in November 2015)

New features:

Support embedding PyPy in a C-program via cffi and static callbacks in cffi.
This deprecates the old method of embedding PyPy

Refactor vmprof to work cross-operating-system, deprecate using buggy
libunwind on Linux platforms. Vmprof even works on Windows now.

Support more of the C-API type slots, like tp_getattro, and fix C-API
macros, functions, and structs such as _PyLong_FromByteArray(),
PyString_GET_SIZE, f_locals in PyFrameObject, Py_NAN, co_filename in
PyCodeObject

Use a more stable approach for allocating PyObjects in cpyext. (seeblog post). Once the PyObject corresponding to a PyPy object is created,
it stays around at the same location until the death of the PyPy object.
Done with a little bit of custom GC support. It allows us to kill the
notion of “borrowing” inside cpyext, reduces 4 dictionaries down to 1, and
significantly simplifies the whole approach (which is why it is a new
feature while technically a refactoring) and allows PyPy to support the
populart lxml module (as of the next release) with no PyPy specific
patches needed

Backport always using os.urandom for uuid4 from cpython and fix the JIT as well
(issue #2202)

More completely support datetime, optimize timedelta creation

Fix for issue #2185 which caused an inconsistent list of operations to be
generated by the unroller, appeared in a complicated DJango app

Fix an elusive issue with stacklets on shadowstack which showed up when
forgetting stacklets without resuming them

Fix entrypoint() which now acquires the GIL

Fix direct_ffi_call() so failure does not bail out before setting CALL_MAY_FORCE

Fix (de)pickling long values by simplifying the implementation

Fix RPython rthread so that objects stored as threadlocal do not force minor
GC collection and are kept alive automatically. This improves perfomance of
short-running Python callbacks and prevents resetting such object between
calls

Support floats as parameters to itertools.isslice()

Check for the existence of CODESET, ignoring it should have prevented PyPy
from working on FreeBSD

Fix for corner case (likely shown by Krakatau) for consecutive guards with
interdependencies

Updates to numpy 1.10.2 (incompatibilities and not-implemented features
still exist)

Support dtype=((‘O’, spec)) union while disallowing record arrays with
mixed object, non-object values

Remove all traces of micronumpy from cpyext if –withoutmod-micronumpy option used

Support indexing filtering with a boolean ndarray

Support partition() as an app-level function, together with a cffi wrapper
in pypy/numpy, this now provides partial support for partition()

Performance improvements:

Optimize global lookups

Improve the memory signature of numbering instances in the JIT. This should
massively decrease the amount of memory consumed by the JIT, which is
significant for most programs. Also compress the numberings using variable-
size encoding

Optimize string concatenation

Use INT_LSHIFT instead of INT_MUL when possible

Improve struct.unpack by casting directly from the underlying buffer.
Unpacking floats and doubles is about 15 times faster, and integer types
about 50% faster (on 64 bit integers). This was then subsequently
improved further in optimizeopt.py.