The biggest news is the support for installing packages that useffi.verify() on machines without a C compiler. Arguably, this
lifts the last serious restriction for people to use CFFI.

Partial list of smaller changes:

mappings between 'wchar_t' and Python unicodes

the introduction of ffi.NULL

a possibly clearer API for ffi.new(): e.g. to allocate a single int and obtain a pointer to it, use ffi.new("int *") instead of the oldffi.new("int")

and of course a plethora of smaller bug fixes

CFFI uses pkg-config to install itself if available. This helps
locate libffi on modern Linuxes. Mac OS/X support is available too
(see the detailed installation instructions). Win32 should work out
of the box. Win64 has not been really tested yet.

Cheers,
Armin Rigo and Maciej Fijałkowski

Hi everybody,

We released CFFI 0.2.1 (expected to be 1.0 soon). CFFI is a way to call C from Python.

EDIT: Win32 was broken in 0.2. Fixed.

This release is only for CPython 2.6 or 2.7. PyPy support is coming in
the ffi-backend branch, but not finished yet. CPython 3.x would be
easy but requires the help of someone.

The biggest news is the support for installing packages that useffi.verify() on machines without a C compiler. Arguably, this
lifts the last serious restriction for people to use CFFI.

Partial list of smaller changes:

mappings between 'wchar_t' and Python unicodes

the introduction of ffi.NULL

a possibly clearer API for ffi.new(): e.g. to allocate a single int and obtain a pointer to it, use ffi.new("int *") instead of the oldffi.new("int")

and of course a plethora of smaller bug fixes

CFFI uses pkg-config to install itself if available. This helps
locate libffi on modern Linuxes. Mac OS/X support is available too
(see the detailed installation instructions). Win32 should work out
of the box. Win64 has not been really tested yet.

Friday, July 13, 2012

I'm proud to release the result of a Facebook-sponsored study on the feasibility of
using the RPython toolchain to produce a PHP interpreter. The rules were
simple: two months; one person; get as close to PHP as possible, implementing
enough warts and corner cases to be reasonably sure that it answers hard
problems in the PHP language. The outcome is called Hippy VM and implements
most of the PHP 1.0 language (functions, arrays, ints, floats and strings).
This should be considered an alpha release.

The resulting interpreter is obviously incomplete – it does not support all
modern PHP constructs (classes are completely unimplemented), builtin functions,
grammar productions, web server integration, builtin libraries
etc., etc.. It's just complete enough for me to reasonably be able to
say that – given some engineering effort – it's possible to provide a rock-solid
and fast PHP VM using PyPy technologies.

The result is available in a Bitbucket repo and is released under the MIT
license.

Performance

The table below shows a few benchmarks comparing Hippy VM to Zend (a standard
PHP interpreter available in Linux distributions) and HipHop VM (a PHP-to-C++
optimizing compiler developed by Facebook). The versions used were Zend 5.3.2
(Zend Engine v2.3.0) and HipHop VM heads/vm-0-ga4fbb08028493df0f5e44f2bf7c042e859e245ab
(note that you need to check out the vm branch to get the newest version).

The run was performed on 64-bit Linux running on a Xeon W3580 with 8M of
L2 cache, which was otherwise unoccupied.

Unfortunately, I was not able to run it on the JITted version of HHVM, the new effort by Facebook,
but people involved with the project told me it's usually slower or comparable with the compiled HipHop.
Their JITted VM is still alpha software, so I'll update it as soon as I have the info.

benchmark

Zend

HipHop VM

Hippy VM

Hippy / Zend

Hippy / HipHop

arr

2.771

0.508+-0%

0.274+-0%

10.1x

1.8x

fannkuch

21.239

7.248+-0%

1.377+-0%

15.4x

5.3x

heapsort

1.739

0.507+-0%

0.192+-0%

9.1x

2.6x

binary_trees

3.223

0.641+-0%

0.460+-0%

7.0x

1.4x

cache_get_scb

3.350

0.614+-0%

0.267+-2%

12.6x

2.3x

fib

2.357

0.497+-0%

0.021+-0%

111.6x

23.5x

fasta

1.499

0.233+-4%

0.177+-0%

8.5x

1.3x

The PyPy compiler toolchain provides a way to implement a dynamic
language interpreter in a high-level language called RPython. This is
a language which is lower-level than Python, but still higher-level than
C or C++: for example, RPython is a garbage-collected language. The killer
feature is that the toolchain will generate a JIT for your interpreter which
will be able to leverage most of the work that has been done on speeding up Python
in the PyPy project. The resulting JIT is generated for your interpreter, and is not Python-specific.
This was one of the toolchain's original design decisions – in contrast to e.g. the JVM,
which was initially only used to interpret Java and later adjusted to serve as a platform for
dynamic languages.

Another important difference is that there is no common bytecode to which you compile both your
language and Python, so you don't inherit problems presented when implementing language X on top of,
say, Parrot VM or the JVM. The PyPy toolchain does not impose constraints on the semantics of
your language, whereas the benefits of the JVM only apply to languages that map well onto Java concepts.

PHP deviations

The project's biggest deviation from the PHP specification is probably
that GC is no longer reference counting. That means that the object finalizer, when
implemented, will not be called directly at the moment of object death, but
at some later point. There are possible future developments to alleviate that
problem, by providing "refcounted" objects when leaving the current scope.
Research has to be done in order to achieve that.

Assessment

The RPython toolchain seems to be a cost-effective choice for writing
dynamic language VMs. It both provides a fast JIT and gives you
access to low-level primitives when you need them. A good example is
in the directory hippy/rpython which contains the implementation
of an ordered dictionary. An ordered dictionary is not a primitive
that RPython provides – it's not necessary for the goal of
implementing Python. Now, implementing it on top of a normal dictionary
is possible, but inefficient. RPython provides a way to work
directly at a lower level, if you desire to do so.

Things that require improvements in RPython:

Lack of mutable strings on the RPython level ended up being a problem.
I ended up using lists of characters; which are efficient, but inconvenient,
since they don't support any string methods.

Frame handling is too conservative and too Python-specific, especially around
the calls. It's possible to implement less general, but simpler and faster
frame handling implementation in RPython.

Status of the implementation

Don't use it! It's a research prototype intended to assess the feasibility
of using RPython to create dynamic language VMs. The most notable
feature that's missing is reasonable error reporting. That said, I'm
confident it implements enough of the PHP language to prove that the full
implementation will present the same performance characteristics.

Benchmarks

The benchmarks are a selection of computer language shootout benchmarks, as well
as cache_get_scb, which is a part of old Facebook code. All benchmarks other
than this one (which is not open source, but definitely the most interesting :( ) are
available in the bench directory. The Python program to run them is called
runner.py and is in the same directory. It runs them 10 times, cutting off the first
3 runs (to ignore the JIT warm-up time) and averaging the rest. As you can see
the standard deviation is fairly minimal for all interpreters and runs; if
it's omitted it means it's below 0.5%.

The benchmarks were not selected for their ease of optimization – the optimizations
in the interpreter were written specifically for this set of benchmarks. No special JIT
optimizations were added, and barring what's mentioned below a vanilla PyPy 1.9 checkout
was used for compilation.

So, how fast will my website run if this is completed?

The truth is that I lack the benchmarks to be able to answer that right now. The core
of the PHP language is implemented up to the point where I'm confident
that the performance will not change as we get more of the PHP going.

How do I run it?

Get a PyPy checkout, apply the diff if you want to squeeze out the last
bits of performance and run pypy-checkout/pypy/bin/rpython targethippy.py to
get an executable that resembles a PHP interpreter. You can also directly run
python targethippy.py file.php, but this will be about 2000x slower.

RPython modifications

There was a modification that I did to the PyPy source code; the diff
is available. It's trivial, and should simply be made optional in the
RPython JIT generator, but it was easier just to do it, given the very constrained time
frame.

gen_store_back_in_virtualizable was disabled. This feature is
necessary for Python frames but not for PHP frames. PHP frames
do not have to be kept alive after we exit a function.

Future

Hippy is a cool prototype that presents a very interesting path towards a fast
PHP VM. However, at the moment I have too many other open source commitments
to take on the task of completing it in my spare time. I do think that this project
has a lot of potential, but I will not commit to any further development at
this time. If you send pull requests I'll try to review them. I'm also open
to having further development on this project funded, so if you're interested
in this project and the potential of a fast PHP interpreter, please get in
touch.

Cheers,
fijal

EDIT: Fixed the path to the rpython binary

Hello everyone.

I'm proud to release the result of a Facebook-sponsored study on the feasibility of
using the RPython toolchain to produce a PHP interpreter. The rules were
simple: two months; one person; get as close to PHP as possible, implementing
enough warts and corner cases to be reasonably sure that it answers hard
problems in the PHP language. The outcome is called Hippy VM and implements
most of the PHP 1.0 language (functions, arrays, ints, floats and strings).
This should be considered an alpha release.

The resulting interpreter is obviously incomplete – it does not support all
modern PHP constructs (classes are completely unimplemented), builtin functions,
grammar productions, web server integration, builtin libraries
etc., etc.. It's just complete enough for me to reasonably be able to
say that – given some engineering effort – it's possible to provide a rock-solid
and fast PHP VM using PyPy technologies.

The result is available in a Bitbucket repo and is released under the MIT
license.

Performance

The table below shows a few benchmarks comparing Hippy VM to Zend (a standard
PHP interpreter available in Linux distributions) and HipHop VM (a PHP-to-C++
optimizing compiler developed by Facebook). The versions used were Zend 5.3.2
(Zend Engine v2.3.0) and HipHop VM heads/vm-0-ga4fbb08028493df0f5e44f2bf7c042e859e245ab
(note that you need to check out the vm branch to get the newest version).

The run was performed on 64-bit Linux running on a Xeon W3580 with 8M of
L2 cache, which was otherwise unoccupied.

Unfortunately, I was not able to run it on the JITted version of HHVM, the new effort by Facebook,
but people involved with the project told me it's usually slower or comparable with the compiled HipHop.
Their JITted VM is still alpha software, so I'll update it as soon as I have the info.

benchmark

Zend

HipHop VM

Hippy VM

Hippy / Zend

Hippy / HipHop

arr

2.771

0.508+-0%

0.274+-0%

10.1x

1.8x

fannkuch

21.239

7.248+-0%

1.377+-0%

15.4x

5.3x

heapsort

1.739

0.507+-0%

0.192+-0%

9.1x

2.6x

binary_trees

3.223

0.641+-0%

0.460+-0%

7.0x

1.4x

cache_get_scb

3.350

0.614+-0%

0.267+-2%

12.6x

2.3x

fib

2.357

0.497+-0%

0.021+-0%

111.6x

23.5x

fasta

1.499

0.233+-4%

0.177+-0%

8.5x

1.3x

The PyPy compiler toolchain provides a way to implement a dynamic
language interpreter in a high-level language called RPython. This is
a language which is lower-level than Python, but still higher-level than
C or C++: for example, RPython is a garbage-collected language. The killer
feature is that the toolchain will generate a JIT for your interpreter which
will be able to leverage most of the work that has been done on speeding up Python
in the PyPy project. The resulting JIT is generated for your interpreter, and is not Python-specific.
This was one of the toolchain's original design decisions – in contrast to e.g. the JVM,
which was initially only used to interpret Java and later adjusted to serve as a platform for
dynamic languages.

Another important difference is that there is no common bytecode to which you compile both your
language and Python, so you don't inherit problems presented when implementing language X on top of,
say, Parrot VM or the JVM. The PyPy toolchain does not impose constraints on the semantics of
your language, whereas the benefits of the JVM only apply to languages that map well onto Java concepts.

PHP deviations

The project's biggest deviation from the PHP specification is probably
that GC is no longer reference counting. That means that the object finalizer, when
implemented, will not be called directly at the moment of object death, but
at some later point. There are possible future developments to alleviate that
problem, by providing "refcounted" objects when leaving the current scope.
Research has to be done in order to achieve that.

Assessment

The RPython toolchain seems to be a cost-effective choice for writing
dynamic language VMs. It both provides a fast JIT and gives you
access to low-level primitives when you need them. A good example is
in the directory hippy/rpython which contains the implementation
of an ordered dictionary. An ordered dictionary is not a primitive
that RPython provides – it's not necessary for the goal of
implementing Python. Now, implementing it on top of a normal dictionary
is possible, but inefficient. RPython provides a way to work
directly at a lower level, if you desire to do so.

Things that require improvements in RPython:

Lack of mutable strings on the RPython level ended up being a problem.
I ended up using lists of characters; which are efficient, but inconvenient,
since they don't support any string methods.

Frame handling is too conservative and too Python-specific, especially around
the calls. It's possible to implement less general, but simpler and faster
frame handling implementation in RPython.

Status of the implementation

Don't use it! It's a research prototype intended to assess the feasibility
of using RPython to create dynamic language VMs. The most notable
feature that's missing is reasonable error reporting. That said, I'm
confident it implements enough of the PHP language to prove that the full
implementation will present the same performance characteristics.

Benchmarks

The benchmarks are a selection of computer language shootout benchmarks, as well
as cache_get_scb, which is a part of old Facebook code. All benchmarks other
than this one (which is not open source, but definitely the most interesting :( ) are
available in the bench directory. The Python program to run them is called
runner.py and is in the same directory. It runs them 10 times, cutting off the first
3 runs (to ignore the JIT warm-up time) and averaging the rest. As you can see
the standard deviation is fairly minimal for all interpreters and runs; if
it's omitted it means it's below 0.5%.

The benchmarks were not selected for their ease of optimization – the optimizations
in the interpreter were written specifically for this set of benchmarks. No special JIT
optimizations were added, and barring what's mentioned below a vanilla PyPy 1.9 checkout
was used for compilation.

So, how fast will my website run if this is completed?

The truth is that I lack the benchmarks to be able to answer that right now. The core
of the PHP language is implemented up to the point where I'm confident
that the performance will not change as we get more of the PHP going.

How do I run it?

Get a PyPy checkout, apply the diff if you want to squeeze out the last
bits of performance and run pypy-checkout/pypy/bin/rpython targethippy.py to
get an executable that resembles a PHP interpreter. You can also directly run
python targethippy.py file.php, but this will be about 2000x slower.

RPython modifications

There was a modification that I did to the PyPy source code; the diff
is available. It's trivial, and should simply be made optional in the
RPython JIT generator, but it was easier just to do it, given the very constrained time
frame.

gen_store_back_in_virtualizable was disabled. This feature is
necessary for Python frames but not for PHP frames. PHP frames
do not have to be kept alive after we exit a function.

Future

Hippy is a cool prototype that presents a very interesting path towards a fast
PHP VM. However, at the moment I have too many other open source commitments
to take on the task of completing it in my spare time. I do think that this project
has a lot of potential, but I will not commit to any further development at
this time. If you send pull requests I'll try to review them. I'm also open
to having further development on this project funded, so if you're interested
in this project and the potential of a fast PHP interpreter, please get in
touch.

Tuesday, July 10, 2012

Apart from the usual "fix shallow py3k-related bugs" part, most of my work in
this iteration has been to fix the bootstrap logic of the interpreter, in
particular to setup the initial sys.path.

Until few weeks ago, the logic to determine sys.path was written entirely
at app-level in pypy/translator/goal/app_main.py, which is automatically
included inside the executable during translation. The algorithm is more or
less like this:

find the absolute path of the executable by looking at sys.argv[0]
and cycling through all the directories in PATH

starting from there, go up in the directory hierarchy until we find a
directory which contains lib-python and lib_pypy

This works fine for Python 2 where the paths and filenames are represented as
8-bit strings, but it is a problem for Python 3 where we want to use unicode
instead. In particular, whenever we try to encode a 8-bit string into an
unicode, PyPy asks the _codecs built-in module to find the suitable
codec. Then, _codecs tries to import the encodings package, to list
all the available encodings. encodings is a package of the standard
library written in pure Python, so it is located insidelib-python/3.2. But at this point in time we yet have to addlib-python/3.2 to sys.path, so the import fails. Bootstrap problem!

The hard part was to find the problem: since it is an error which happens so
early, the interpreter is not even able to display a traceback, because it
cannot yet import traceback.py. The only way to debug it was through some
carefully placed print statement and the help of gdb. Once found the
problem, the solution was as easy as moving part of the logic to RPython,
where we don't have bootstrap problems.

Once the problem was fixed, I was able to finally run all the CPython test
against the compiled PyPy. As expected there are lots of failures, and fixing
them will be the topic of my next months.

Apart from the usual "fix shallow py3k-related bugs" part, most of my work in
this iteration has been to fix the bootstrap logic of the interpreter, in
particular to setup the initial sys.path.

Until few weeks ago, the logic to determine sys.path was written entirely
at app-level in pypy/translator/goal/app_main.py, which is automatically
included inside the executable during translation. The algorithm is more or
less like this:

find the absolute path of the executable by looking at sys.argv[0]
and cycling through all the directories in PATH

starting from there, go up in the directory hierarchy until we find a
directory which contains lib-python and lib_pypy

This works fine for Python 2 where the paths and filenames are represented as
8-bit strings, but it is a problem for Python 3 where we want to use unicode
instead. In particular, whenever we try to encode a 8-bit string into an
unicode, PyPy asks the _codecs built-in module to find the suitable
codec. Then, _codecs tries to import the encodings package, to list
all the available encodings. encodings is a package of the standard
library written in pure Python, so it is located insidelib-python/3.2. But at this point in time we yet have to addlib-python/3.2 to sys.path, so the import fails. Bootstrap problem!

The hard part was to find the problem: since it is an error which happens so
early, the interpreter is not even able to display a traceback, because it
cannot yet import traceback.py. The only way to debug it was through some
carefully placed print statement and the help of gdb. Once found the
problem, the solution was as easy as moving part of the logic to RPython,
where we don't have bootstrap problems.

Once the problem was fixed, I was able to finally run all the CPython test
against the compiled PyPy. As expected there are lots of failures, and fixing
them will be the topic of my next months.