I created a first toy based on the PEP 445 to test of Python handles MemoryError: in short, Python handles them badly...
Home page of the project:
https://pypi.python.org/pypi/pyfailmalloc
I will use this issue to track crashes found by this tool.

misc_fixes.patch: Misc bugfixes related to MemoryError, this patch should be read carefully. The changes in unicodeobject.c may be written differently. I wrote them to fix a failing assertion ("assert(data != NULL);") in _PyUnicode_CheckConsistency(), called when the creation of a new string failed.

frame_fasttolocals.patch: Add a new PyFrame_FastToLocalsWithError() function, same than PyFrame_FastToLocals(), except that errors are reported.
This function is not atomic: if an error occurs in the middle of map_to_dict(), the frame can enter an inconsistent state. Is it worse than not reporting the failure?

import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure.

listpop.patch:
list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.

Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).

2013/7/17 Serhiy Storchaka <report@bugs.python.org>:
> listpop.patch:
>
> list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.
Resizing a buffer to make it smaller "should" not fail, and if it
fails, the case is very unlikely. But I prefer to handle the case,
rather than hoping that the case cannot occur in the libc
implementation.
For example, _PyObject_Realloc() of pymalloc allocates a *new* buffer
to shrink an existing buffer if the size is reduced by more than 25%:
if (nbytes <= size) {
/* The block is staying the same or shrinking. If
* it's shrinking, there's a tradeoff: it costs
* cycles to copy the block to a smaller size class,
* but it wastes memory not to copy it. The
* compromise here is to copy on shrink only if at
* least 25% of size can be shaved off.
*/
if (4 * nbytes > 3 * size) {
/* It's the same,
* or shrinking and new/old > 3/4.
*/
return p;
}
size = nbytes;
}
bp = _PyObject_Malloc(ctx, nbytes);
I plan also to test Python with PyMem_Malloc using pymalloc to check
if it's faster or not. So PyMem_Realloc() may fail when shrinking a
buffer ;-)
By the way, I already fixed similar bugs (handle PyMem_Realloc failure
when shrinking the buffer) in this issue. Examples:
http://hg.python.org/cpython/rev/68887e177dd4http://hg.python.org/cpython/rev/549d8d3297f2
listpop.patch only changes the error handling to make it more
resilient when the "very unlikely" case occurs ;-)

2013/7/17 Charles-François Natali <report@bugs.python.org>:
> Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
> This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).
I plan to write a new module and propose it for inclusion in Python
3.4. It will have the features of pytracemalloc *and* pyfailmalloc,
and also statistics (number of calls to each allocator function, with
a function to reset counters).
The blocker pointer is to get the size of a buffer on free(). In
pytracemalloc, I'm using a glib hash lib: pointer => structure {size
of the buffer, filename, line number}. The dependency to glib should
be removed to integrate the module in Python :-)
The simpler solution is to allocate a wider buffer and store the
metadata directly in the buffer before the data. The second problem is
how this feature would be enabled. Enabling the feature at runtime
would require a new environment variable to set the right memory
allocator at startup. The choice of the memory allocator must be done
before the first memory allocation, so in main().
Integrating pytracemalloc into CPython was my main motivation to write
the PEP 445.
If you would like to use pyfailmalloc right now, you have to:
* build pyfailmalloc with Python 3.4
* apply debug_cpython.patch on Python which is the pyfailmalloc repository
* run gdb -x debug_cpython.gdb (which can also be found in the
pyfailmalloc repository)
* https://bitbucket.org/haypo/pyfailmalloc

Open issues (TODO list):
- review and apply listpop.patch and frame_fasttolocals.patch
- fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION
- fix Modules/pyexpat.c: handlers do no detect failures
- fix sqlite: Modules/_sqlite/connection.c calls the finalize method on the aggregate whereas calling step() failed
- fix tests hang when an exception occurs in a thread
- fix new bugs found by pyfailmalloc :-)

> This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree: (...)
Thanks for the quick report! Are you testing automatically lxml with the Python HEAD?
The issue should be fixed by the changeset 6ec0e9347dd4. I also opened the issue #18501 to discuss how ElementTree should handle Python exceptions.

"import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure."
I created the issue #18507 for that.

"fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION"
I created the issue #18509 for that.

TODO:
- review and integrate frame_fasttolocals.patch
- #18507: import_init() should not use Py_FatalError() but return an error
- #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error
- fix tests hang when an exception occurs in a thread
Most bugs have been fixed.