This cleanup also removes multimethods from these types. PyPy has
historically used a clever implementation of multimethod dispatch for declaring
methods of the __builtin__ types in RPython.

This multimethod scheme provides some convenient features for doing this,
however we've come to the conclusion that it may be more trouble than it's
worth. A major problem of multimethods is that they generate a large amount of
stub methods which burden the already lengthy and memory hungry RPython
translation process. Also, their implementation and behavior can be somewhat
complicated/obscure.

The alternative to multimethods involves doing the work of the type checking
and dispatching rules in a more verbose, manual way. It's a little more work in
the end but less magical.

Recently, Manuel Jacob finished a large cleanup effort of the
unicode/string/bytearray types that also removed their multimethods. This work
also benefits the py3k branch: it'll help with future PEP 393 (or PEP 393
alternative) work. This effort was partly sponsored by Google's Summer of
Code: thanks Manuel and Google!

Now there's only a couple major pieces left in the multimethod removal (the
float/complex types and special marshaling code) and a few minor pieces that
should be relatively easy.

In conclusion, there's been some good progress made on py3k and multimethod
removal this winter, albeit a bit slower than we would have liked.

This cleanup also removes multimethods from these types. PyPy has
historically used a clever implementation of multimethod dispatch for declaring
methods of the __builtin__ types in RPython.

This multimethod scheme provides some convenient features for doing this,
however we've come to the conclusion that it may be more trouble than it's
worth. A major problem of multimethods is that they generate a large amount of
stub methods which burden the already lengthy and memory hungry RPython
translation process. Also, their implementation and behavior can be somewhat
complicated/obscure.

The alternative to multimethods involves doing the work of the type checking
and dispatching rules in a more verbose, manual way. It's a little more work in
the end but less magical.

Recently, Manuel Jacob finished a large cleanup effort of the
unicode/string/bytearray types that also removed their multimethods. This work
also benefits the py3k branch: it'll help with future PEP 393 (or PEP 393
alternative) work. This effort was partly sponsored by Google's Summer of
Code: thanks Manuel and Google!

Now there's only a couple major pieces left in the multimethod removal (the
float/complex types and special marshaling code) and a few minor pieces that
should be relatively easy.

In conclusion, there's been some good progress made on py3k and multimethod
removal this winter, albeit a bit slower than we would have liked.

Sunday, February 9, 2014

Since the previous
post, we believe we progressed a lot by discovering an alternative
core model for software transactions. Why do I say "believe"? It's
because it means again that we have to rewrite from scratch the C
library handling STM. This is currently work in progress. Once this is
done, we should be able to adapt the existing pypy-stm to run on top of
it without much rewriting efforts; in fact it should simplify the
difficult issues we ran into for the JIT. So while this is basically
yet another restart similar to last
June's, the difference is that the work that we have already put in the PyPy
part (as opposed to the C library) remains.

You can read about the basic ideas of this new C library here.
It is still STM-only, not HTM, but because it doesn't constantly move
objects around in memory, it would be easier to adapt an HTM version.
There are even potential ideas about a hybrid TM, like using HTM but
only to speed up the commits. It is based on a Linux-only system call, remap_file_pages()
(poll: who heard about it before? :-). As previously, the work is done
by Remi Meier and myself.

Currently, the C library is incomplete, but early experiments show good
results in running duhton,
the interpreter for a minimal language created for the purpose of
testing STM. Good results means we brough down the slow-downs from
60-80% (previous version) to around 15% (current version). This number
measures the slow-down from the non-STM-enabled to the STM-enabled
version, on one CPU core; of course, the idea is that the STM version
scales up when using more than one core.

This means that we are looking forward to a result that is much better
than originally predicted. The pypy-stm has chances to run at a
one-thread speed that is only "n%" slower than the regular pypy-jit, for
a value of "n" that is optimistically 15 --- but more likely some number
around 25 or 50. This is seriously better than the original estimate,
which was "between 2x and 5x". It would mean that using pypy-stm is
quite worthwhile even with just two cores.

More updates later...

Armin

Hi all,

A quick note about the Software Transactional Memory (STM) front.

Since the previous
post, we believe we progressed a lot by discovering an alternative
core model for software transactions. Why do I say "believe"? It's
because it means again that we have to rewrite from scratch the C
library handling STM. This is currently work in progress. Once this is
done, we should be able to adapt the existing pypy-stm to run on top of
it without much rewriting efforts; in fact it should simplify the
difficult issues we ran into for the JIT. So while this is basically
yet another restart similar to last
June's, the difference is that the work that we have already put in the PyPy
part (as opposed to the C library) remains.

You can read about the basic ideas of this new C library here.
It is still STM-only, not HTM, but because it doesn't constantly move
objects around in memory, it would be easier to adapt an HTM version.
There are even potential ideas about a hybrid TM, like using HTM but
only to speed up the commits. It is based on a Linux-only system call, remap_file_pages()
(poll: who heard about it before? :-). As previously, the work is done
by Remi Meier and myself.

Currently, the C library is incomplete, but early experiments show good
results in running duhton,
the interpreter for a minimal language created for the purpose of
testing STM. Good results means we brough down the slow-downs from
60-80% (previous version) to around 15% (current version). This number
measures the slow-down from the non-STM-enabled to the STM-enabled
version, on one CPU core; of course, the idea is that the STM version
scales up when using more than one core.

This means that we are looking forward to a result that is much better
than originally predicted. The pypy-stm has chances to run at a
one-thread speed that is only "n%" slower than the regular pypy-jit, for
a value of "n" that is optimistically 15 --- but more likely some number
around 25 or 50. This is seriously better than the original estimate,
which was "between 2x and 5x". It would mean that using pypy-stm is
quite worthwhile even with just two cores.

Thursday, February 6, 2014

Work continued on the NumPy + PyPy front steadily in December and more lightly in January. The continued focus was compatibility, targeting incorrect or unimplemented features that appeared in multiple NumPy test suite failures. We now pass ~2/3 of the NumPy test suite. The biggest improvements were made in these areas:

- Bugs in conversions of arrays/scalars to/from native types
- Fix cases where we would choose incorrect dtypes when initializing or computing results
- Improve handling of subclasses of ndarray through computations
- Support some optional arguments for array methods that are used in the pure-python part of NumPy
- Support additional attributes in arrays, array.flags, and dtypes
- Fix some indexing corner cases that arise in NumPy testing
- Implemented part of numpy.fft (cffti and cfftf)

Looking forward, we plan to continue improving the correctness of the existing implemented NumPy functionality, while also beginning to look at performance. The initial focus for performance will be to look at areas where we are significantly worse than CPython+NumPy. Those interested in trying these improvements out will need a PyPy nightly, and an install of the PyPy NumPy fork. Thanks again to the NumPy on PyPy donors for funding this work.

Work continued on the NumPy + PyPy front steadily in December and more lightly in January. The continued focus was compatibility, targeting incorrect or unimplemented features that appeared in multiple NumPy test suite failures. We now pass ~2/3 of the NumPy test suite. The biggest improvements were made in these areas:

- Bugs in conversions of arrays/scalars to/from native types
- Fix cases where we would choose incorrect dtypes when initializing or computing results
- Improve handling of subclasses of ndarray through computations
- Support some optional arguments for array methods that are used in the pure-python part of NumPy
- Support additional attributes in arrays, array.flags, and dtypes
- Fix some indexing corner cases that arise in NumPy testing
- Implemented part of numpy.fft (cffti and cfftf)

Looking forward, we plan to continue improving the correctness of the existing implemented NumPy functionality, while also beginning to look at performance. The initial focus for performance will be to look at areas where we are significantly worse than CPython+NumPy. Those interested in trying these improvements out will need a PyPy nightly, and an install of the PyPy NumPy fork. Thanks again to the NumPy on PyPy donors for funding this work.