In Alex Gaynor's post
there is just about everything right. I still want to add my take on it.

Python3 is a fork of the Python community, that has left the user base largely
behind. After developing features very conservatively for a long time (from
__future__ import division and stuff), where features were first available but
not active by default, that whole "Python 3000" thing came up.

Misguided, after having maintained CPython on a level of excellence for such a
long time, there was a sense of "finally we get to make free choices". I can
understand that. And having witnessed Gnome 2.0.0, and KDE 4.0.0, I even start
to believe that there is some kind of underlying law. Something that makes
people want to start over.

That "5 years plan" Alex mentioned has failed, and everybody knows that without
solving the GIL within the next 5 years, i.e. another incompatible Python
change, it will likely become obsolete.

In terms of Python2.8, call it that way. Or have a 3.5 that has print
statement, and then all kinds of incompatible changes to Python3 to make it
behave sane bytes(7) needs to do what str(7) once it. That would be
about from __past__ import .... I suppose.

I also had another idea, having a "python2" built-in module that carries a
CPython2 interpreter and proxy objects for instances, where they talk to another
with incompatible types (new bytes/unicode to old str).

Due to my work on that Python compiler that should have existed 20 years
ago, I am familiar enough, to actually be able and create that kind of
CPython2/CPython3 hybrid.

But the main problems with all of that are:

CPython is developed by volunteers, and as such, they are neither obliged to
do what we need, nor will they bend to any amount of posting. We need to get
our acts together or it's not happening, and we are people who care about
Python relevancy.

There may not be nearly enough overlap of people capable and caring. And that
would be a tremendous amount of work. Highly speculative, in the sense, that
it may not see fruits.

And then, when you got that sorted out, expect a hostile reaction from the
core developers and/or PSF. Whatever it is, that you develop, you won't be
able to call it Python 2.8, as they are going to hate it.

And finally, as a course of action, that may even bring the complete downfall
of Python as a community. The controversy associated is not for everybody,
and we don't need a libreoffice/openoffice situation in Python, do we.

It's a pity, because clearly, for Nuitka there would be a bunch of patches, that
I would like to make, that would make my life far easier, and even interpreted
Python faster.

Many of you who turn to my easy to use, highly compatible Python compiler
Nuitka, do this mostly because they seek to solve the
deployment problem that Python suffers from.

Be this, because you want to use a newer Python2.7 on RHEL5 without installing
anything at all there. Or because Windows is difficult to tackle otherwise.

For the longest time, Nuitka had not offered anything in this domain, focusing
solely on being an accelerator. Lately, I have taken up the challenge and
polished initial solutions submitted by contributors.

This first showed up in the 0.4.7 release, but turned out relatively weak. While
first examples were working on Linux, it was not working at all on Windows
(anymore). And basically there was a huge lack of tests.

Actually I didn't mean for it to be released with that feature, but as parts of
it seemed to work, I did so. But truth to be sad, that feature is not nearly as
polished in that release as you would like it to.

In current development releases, of
what is going to become 0.5.0 really soon now, it's much better already. More
things actually work. But it appears, there will be more ground to cover, and
this is a lot of stuff to sort out.

So, this is mostly about asking you two things. Give that development release a
try and report issues you have with it. And help me.

And have patience. I am developing Nuitka as an accelerator on a "no known bugs"
basis. That means, once I know of a bug, I will fix it. OK, some issues in fact
take longer, but then it really is not important at all, but difficult at the
time. For standalone mode, I can't do it that way, or I would have to neglect
the acceleration parts, which I totally don't want to do.

Because while you maybe are only interested in a packaging solution, many others
would like to have that orders of magnitude speedup that I have been aiming for
and that feels near now. This is about making Python a viable language for more
uses than it currently is.

So why do it in the first place. For one, I am hoping that it helps people to
not turn away from Python. And second, and more important, I am hoping that by
making it more useful, more people will join me. (Oh, and thirdly, it's also a
nice puzzle to solve. I seem to enjoy that.)

Ultimately both modes will be needed, standalone, and acceleration. And it seems
like I am working to provide both. For standalone, more often, than seeking to
avoid bugs as far as possible, I am going to rely on your participation.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes important new features, lots of polishing cleanups, and
some important performance improvements as well.

Bug Fixes

The RPM packages didn't build due to missing in-line copy of Scons. Fixed in
0.4.6.1 already.

The recursion into modules and unfreezing them was not working for packages
and modules anymore. Fixed in 0.4.6.2 already.

The Windows installer was not including Scons. Fixed in 0.4.6.3 already.

Windows: The immediate execution as performed by nuitka --execute was not
preserving the exit code. Issue#26.

Python3.3: Packages without __init.py__ were not properly embedding the
name-space package as well.

Python3: Fix, modules and packages didn't add themselves to sys.modules
which they should, happened only for programs.

Python3.3: Packages should set __package to their own name, not the one of
their parents.

Python3.3: The __qualname__ of nested classes was corrected.

For modules that recursed to other modules, an infinite loop could be
triggered when comparing types with rich comparisons. Issue#115.

New Features

The "standalone" mode allows to compile standalone binaries for programs and
run them without Python installation. The DLLs loaded by extension modules on
Windows need to be added manually, on Linux these are determined
automatically already.

To achieve running without Python installation, Nuitka learned to freeze
bytecode as an alternative to compiling modules, as some modules need to be
present when the CPython library is initialized.

New option --python-flag allows to specify flags to the compiler that the
"python" binary normally would. So far -S and -v are supported, with
sane aliases no_site and trace_imports.

The recommended use of --python-flag=-S is to avoid dependency creep in
standalone mode compilations, because the site module often imports many
useless things that often don't apply to target systems.

New Optimization

Faster frame stack handling for functions without try/except (or
try/finally in Python3). This gives a speed boost to "PyStone" of
ca. 2.5% overall.

Python2: Much quicker calls of __getattr__ and __setattr__ as this is
now using the quicker call method avoiding temporary tuples.

Don't treat variables usages used in functions called directly by their owner
as shared. This leads to more efficient code generation for contractions and
class bodies.

Create unicode constants directly from their UTF-8 string representation
for Python2 as well instead of un-streaming. So far this was only done for
Python3. Affects only program start-up.

Directly create int and long constants outside of 2**31 and
2**32-1, but only limited according to actual platform values. Affects
only program start-up.

When creating set values, no longer use a temporary tuple value, but
use a properly generated helper functions instead. This makes creating sets
much faster.

Directly create set constants instead of un-streaming them. Affects only
program start-up.

For correct line numbers in traceback, the current frame line number must be
updated during execution. This was done more often than necessary, e.g. loops
set the line number before loop entry, and at first statement.

Module variables are now accessed even faster, the gain for "PyStone" is only
0.1% and mostly the result of leaner code.

Organizational

The "standalone mode" code (formerly known as "portable mode" has been redone
and activated. This is a feature that a lot of people expect from a compiler
naturally. And although the overall goal for Nuitka is of course acceleration,
this kind of packaging is one of the areas where CPython needs improvement.

Added package for Ubuntu 13.10 for download, removed packages for Ubuntu 11.04
and 11.10, no more supported.

Added package for openSUSE 13.1 for download.

Nuitka is now part of Arch and can be installed with pacman -S nuitka.

Using dedicated Buildbot for continuous integration
testing. Not yet public.

Windows: In order to speed up repeated compilation on a platform without
ccache, added Scons level caching in the build directory.

Disabled hash randomization for inside Nuitka (but not in ultimately created
binaries) for a more stable output, because dictionary constants will not
change around. This makes the build results possible to cache for ccache
and Scons as well.

Tests

The programs tests cases now fail if module or directory recursion is not
working, being executed in another directory.

Added test runner for packages, with initial test case for package with
recursion and sub-packages.

Made some test cases more strict by reducing PYTHONPATH provision.

Detect use of extra flags in tests that don't get consumed avoiding
ineffective flags.

Use --execute on Windows as well, the issue that prevented it has been
solved after all.

Cleanups

The generated code uses const_, var_, par_ prefixes in the
generated code and centralized the decision about these into single place.

Module variables no longer use C++ classes for their access, but instead
accessor functions, leading to much less code generated per module variable
and removing the need to trace their usage during code generation.

The test runners now share common code in a dedicated module, previously they
replicated it all, but that turned out to be too tedious.

Massive general cleanups, many of which came from new contributor Juan Carlos
Paco.

Moved standalone and freezer related codes to dedicated package
nuitka.freezer to not pollute the nuitka package name space.

The code generation use variable identifiers and their accesses was cleaned
up.

Removed several not-so-special case identifier classes because they now behave
more identical and all work the same way, so a parameters can be used to
distinguish them.

Moved main program, function object, set related code generation to dedicated
modules.

Summary

This release marks major technological progress with the introduction of the
much sought standalone mode and performance improvements from improved code
generation.

The major break through for SSA optimization was not yet achieved, but this is
again making progress in the direction of it. Harmonizing variables of different
kinds was an important step ahead.

Also very nice is the packaging progress, Nuitka was accepted into Arch after
being in Debian Testing for a while already. Hope is to see more of this kind of
integration in the future.

Please see the page "What is Nuitka?" for
clarification of what it is now and what it wants to be.

Nuitka is a typical compiler, just for Python, and aims to be integrating with
build tools, and as such is so much a command line tool, it hurts.

Yet, there are GUIs. Yes, plural, there are even two.

Many of you are maybe using Ninja IDE already. I have developed most of early
Nuitka in vim, and later, mostly because of Nuitka, switched back to Emacs. I
had switched from Emacs at one point 12 years ago, but then went back a couple
of years ago.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release includes progress on all fronts. The primary focus was to advance
SSA optimization over older optimization code that was already in place. In this
domain, there are mostly cleanups.

Another focus has been to enhance Scons with MSVC on Windows. Nuitka now finds
an installed MSVC compiler automatically, properly handles architecture of
Python and Windows. This improves usability a lot.

Then this is also very much about bug fixes. There have been several hot fixes
for the last release, but a complicated and major issue forced a new release,
and many other small issues.

And then there is performance. As can be seen in the performance graph, this release is the fastest so
far. This came mainly from examining the need for comparison slots for compiled
types.

And last, but not least, this also expands the base of supported platforms,
adding Gentoo, and self compiled Python to the mix.

Bug Fixes

Support Nuitka being installed to a path that contains spaces and handle main
programs with spaces in their paths. Issue#106. Fixed in 0.4.5.1 already.

Support Python being installed to a path that contains spaces. Issue#106. Fixed in 0.4.5.2 already.

Windows: The option --windows-disable-console was not effective with
MSVC. Issue#107. Fixed in 0.4.5.3
already.

Windows: For some users, Scons was detecting their MSVC installation properly
already from registry, but it didn't honor the target architecture. Issue#99. Fixed in 0.4.5.3 already.

When creating Python modules, these were marked as executable ("x" bit), which
they are of course not. Fixed in 0.4.5.3 already.

Python3.3: On architectures where Py_ssize_t is not the same as long
this could lead to errors. Fixed in 0.4.5.3 already.

Code that was using nested mutable constants and changed the nested ones was
not executing correctly. Issue#112.

Python2: Due to list contractions being re-formulated as functions, del
was rejected for the variables assigned in the contraction. Issue#111.

[expr(x)forxiniterable()]delx# Should work, was gave an unjustified SyntaxError.

New Features

Compiled types when used in Python comparison now work. Code like this will
work:

deff():passasserttype(f)==types.FunctionType

This of course also works for in operator, and is another step ahead in
compatibility, and surprising too. And best of all, this works even if the
checking code is not compiled with Nuitka.

Windows: Detecting MSVC installation from registry, if no compiler is already
present in PATH.

Windows: Now options --mingw to force compilation with MinGW.

New Optimization

Rich comparisons (==, <, and the like) are now faster than ever before
due to a full implementation of its own in Nuitka that eliminates a bit of the
overhead. In the future, we will aim at giving it type hints to make it even
faster. This gives a minor speed boost to PyStone of ca. 0.7% overall.

Integer comparisons are now treated preferably, as they are in CPython, which
gives 1.3% speed boost to CPython.

The SSA based analysis is now used to provide variable scopes for temporary
variables as well as reference count needs.

Cleanups

Replaced "value friend" based optimization code with SSA based optimization,
which allowed to remove complicated and old code that was still used mainly in
optimization of or and and expressions.

Delayed declaration of temp variables and their reference type is now
performed based on information from SSA, which may given more accurate
results. Not using "variable usage" profiles for this anymore.

The Scons interface and related code got a massive overhaul, making it more
consistent and better documented. Also updated the internal copy to 2.3.0 for
the platforms that use it, mostly Windows.

Stop using os.system and subprocess.call(..., shell = True) as it is
not really portable at all, use subprocess.call(..., shell = False)
instead.

As usual lots of cleanups related to line length issues and PyLint.

Organizational

Added support for Gentoo Linux.

Added support for self compiled Python versions with and without debug
enabled. Issue#110

Added use of Nuitka fonts for headers in manuals.

Does not install in-line copy of Scons only on systems where it is not going to
be used, that is mostly non-Windows, and Linux where it is not already
present. This makes for cleaner RPM packages.

Summary

While the SSA stuff is not yet bearing performance fruits, it starts to carry
weight. Taking over the temporary variable handling now also means we can apply
the same stuff to local variables later.

To make up for the delay in SSA driven performance improvements, there is more
traditional code acceleration for rich comparisons, making it significant, and
the bug fixes make Nuitka more compatible than ever.

Once upon a time ago physics had multiple forces. Each with very distinct
traits. These were the early days. Then as time went on people started to
realize that weak nuclear force and electromagnetic force were the same
sort. That was after discovering that electricity and magnetism were the same,
and that is the pattern. It's called symmetry, equivalence, and so on.

The Nuitka project is effectively a Python research
that goes through the same steps. There are e.g. currently local variables,
closure variables (shared from containing function), module variables, temp
holder, temp keeper, and temp variables.

Often when I added one more this was driven by code generation needs or
deficiencies at the time. These are of course way too many, and all are good at
one thing only.

So the main task Nuitka is faced with now to e.g. generalize that "temp keeper"
variables are temporary variables local to one statement only, surely to be
released afterwards, and make that particular feature e.g. work for all
variables across the board. Right now, code generation uses "cheats" where it
puts a C++ block and local declarations for "temp keeper" variables.

It would be good, if it could be a try with del statement on the "temp
keeper variable" in its finally part and yet, get the same code
generated. Difficult, yes, but not impossible, and definitely what I am aiming
at.

So that is what Nuitka will be dealing with in the next releases. Once done,
maybe there will be a "gravity", i.e. the one thing not yet harmonized, but for
good reasons. Potentially limited due to lack of understanding, potentially
because there is a good reason. And so not all we be unified or maybe
e.g. module variables will be a bit more special than local variables, although
modules are just functions with variables writable everywhere.

Good stuff is coming, hold on. Join the project, or support it financially if you
wish.

Nuitka still is too much of a one man show. Should I be asked to name one single
weakness of it - that would be it.

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release incorporates very many bug fixes, most of which were already part
of hot fixes, usability improvements, documentation improvements, new logo,
simpler Python3 on Windows, warnings for recursion options, and so on. So it's
mostly a consolidation release.

Bug Fixes

When targeting Python 3.x, Nuitka was using "python" to run Scons to run it
under Python 2.x, which is not good enough on systems, where that is already
Python3. Improved to only do the guessing where necessary (i.e. when using the
in-line copy of Scons) and then to prefer "python2". Issue#95. Fixed in 0.4.4.1 already.

When using Nuitka created binaries inside a "virtualenv", created programs
would instantly crash. The attempt to load and patch inspect module was
not making sure that site module was already imported, but inside the
"virtualenv", it cannot be found unless. Issue#96. Fixed in 0.4.4.1 already.

The option --recurse-directory to include plugin directories was
broken. Issue#97. Fixed in 0.4.4.2
already.

Enhanced documentation to number page numbers starting after table of
contents, removed header/footer from cover pages.

Summary

This release is mostly the result of improvements made based on the surge of
users after Europython 2013. Some people went to extents and reported their
experience very detailed, and so I could aim at making e.g. their misconceptions
about how recursion options work, more obvious through warnings and errors.

This release is not addressing performance improvements. The next release will
be able to focus on that. I am taking my claim of full compatibility very
serious, so any time it's broken, it's the highest priority to restore it.

Back from Europython 2013 in Florence for a while now. I had a presentation
about my Python compiler Nuitka. You can watch the
Video on Youtube. I believe it's absolutely worth your time. I was not doing a
lot of "why" at all now, only "what", and demo times, and answering questions:

The talk went really well. I believe one can clearly see that I felt really good
and at ease. The state presented is very good and progress - there was a lot, so
that was great. The slides of the talk
may also be interesting.

And it definitely raised a lot of interest.

Last years talk is also there. You can (maybe) see that I was 20kg more weight
then, and also much more excited, at least initially. Background: I was
surprised then that Guido van Rossum was attending the talk, plus I was kind of
not in the state of this year, the confidence that people should believe, that
things will work after that, was not yet there. I knew it, but standing in front
of a croud and saying, look I am attemting what you consider impossible, or even
have failed at, is not easy:

There are also the slides of that talk
which may also be interesting, but of course are outdated somewhat now.

Years make differences

So yeah, the progress happened since 2012 made a difference. Nuitka has proven
things. Let's see what this year does to it. I would hope for real type
inference optimization and portable mode to be there next year.

Consider that is the a spare time effort. Lots of people lauded it for the out
of the box experience, and how it just works. That's right. I put a lot of focus
on stability, correctness, and ease of use. In fact, since my return, I have
mostly worked on the feedback I got in that domain.

But for the relatively small amount of time I can invest (outside of holidays),
it's make very fast progress.

Out of the box thinking

The feedback during the conference was great. While Nuitka is not yet seeing a
lot of users, and not as many contributions or donations as I would hope for,
but that's kind of expected with my out of the box thinking there.

Few people at Europython really need a Python compiler. The most important
exception are scientific people, doing number crunching. And I made some
interesting contacts there.

Since my return, I have been receiving a bunch of bug reports. Some of which
were unfortunately regressions of 0.4.4, and these got fixed in two hotfix
releases. Currently 0.4.4.2 is released, and contains a huge amount of fixes,
mostly stuff found by the surge new users.

But I deserved that, having said that I didn't have to do hotfixes for a while
now.

Florence

Such a lovely place, and so lovely people. It's a tourist place, true, but it's
very open minded. The food is extremely good. And going from the conference
hotel straight over to the river to meet up and have a few beers was very
comfortable for after-conference stuff.

Now that I have been there 2 times, and the conference even 3 times, it's moving
on, to Berlin, which I sort of regret. Knowing the place somewhat definitely is
an advantage by itself.

Python Revolution

One thing that came to my mind during the conference, witnessing the large
amount of non-satisfaction, was that the community is fed up with the
leadership. Questioned about lack of Python3 adoption, a PSF guy in his
presentation said they were 3 years into a 5 years plan, and got interrupted by
laughter.

It may be about time, somebody actually gets up, removes the GIL and has a
Python2.8 that is compatible. Obviously that's much more useful. It could be a
Python3.x with compatibility names of modules, print statement, and
dict.iteritems added, probably also making some of the unicode mess more
harmless, and providing unicode and long built-in names.

Considering that I have in fact been playing with the idea of forking CPython
top versions, to add things, that would allow Nuitka to work better with it,
that could be a staging ground, where such changes are nurtured. On the other
hand, Nuitka is too irrelevant yet to drive this, and it would detract me away
from it.

Gains

So Nuitka got a lot more exposure. And one guy watching the talk even created a
really nice logo for it. I will make
it a separate announcement though, and the next release is going to use it.

And in the aftermath, there were much bug useful reports. And new ideas. And
reinforcement that what I am doing is actually useful to some people already.

Donations

My travel to Florence was in part funded by donations. Thanks a lot to all of
you who have given, and who would like to also make a donation. Should I receive sufficient donations, I intend to go
to FOSDEM 2014 as well.

Going to Berlin will be a bit cheaper this time around, as I can travel there
via train.