Overview

After that I concentrated on aligning this with JyNI's usual build process.
It quickly turned out that already JyNI's DemoExtension was not C89 compliant. After fixing this, I got the demo extension working on original CPython 2.7, which asserts that the ABI is correct.

During this work I suffered from frequent mouse freezes on Windows 10. While these always occurred during working with Eclipse I cannot tell whether it was Eclipse related. Also for other reasons I preferred to return to my Linux environment and checked for ways to achieve this.
First idea was to configure gcc to behave more like MSVC, e.g. accept C89 only. This approach proved insufficient for producing MSVC compatible code, so I tried to run cl.exe using Wine. Finally I came up with a handy wine-based makefile that allows me to develop MSVC compatible code on Linux (linking not yet tested).

Official setup for Python 2.7 on Windows

I somehow thought that I should be able to build CPython 2.7 using the original toolchain.
The CPython 2.7 PC-build README mentions that “Another configuration proven to work is Visual Studio 2008, Windows SDK 7.1, and Visual Studio 2013.” So I was curious if it could alternatively work with MSVS 2015 + MSVS 2008 + SDK 7.1 (Nice thing with MSVS 2015 is that it is the official tool for building Python 3.). But after installing this huge setup (for each of them the download alone took 1.5 hours for me, so more or less the whole day passed by with setup stuff) it turned out that MSVS 2015 could not detect the compiler from MSVS 2008, unless also MSVS 2010 would be installed. Indeed, that ultimately solved the problem. So, after many hours of setup and more than 20GB harddrive space filled up, I was finally able to compile CPython 2.7. Next step was to build a custom C extension using this setup and to assert that the result is workable with original CPython 2.7. First issue here was C89 compatibility of JyNI's DemoExtension. With that fixed I ran into various linking issues using MSVS GUI despite following various building tutorials pedantically. (The official documentation does not cover building with MSVS GUI). After some more head-banging-on-table it turned out that it is necessary to reference the pythoncore project in addition to linker- and include-configuration.

With that working I wanted to figure out how to accomplish this from CLI, i.e. by calling cl.exe. Since this involves complex command lines with include paths etc I directly approached this using makefiles. On Windows it takes a bit more than selecting make in the package manager, e.g. select an actual make implementation first. For sake of free software I decided to use GNU make. Rather than selecting a distribution I quickly built it from source, which was luckily straight forward with three entire distributions of MSVS installed.

It turned out that it is not suitable to run vcvarsall.bat from a makefile: The adjusted environment variables are not memorized for subsequent tasks.
So I figured out – partly from the build command issued by MSVS GUI – what libs needed to be linked etc. The only info the user must adjust in the resulting makefile is the location of MSVC, but that would have also been required to run vcvarsall.bat, because it is the location of that file.

Windows toolchain with GNU make and Visual C++ for Python

To sort things out regarding building (from command line), the Cython Wiki was enormously helpful. It pointed me to the Visual C++ for Python distribution, a link that I really miss in the official Python documentation. This ~84MB file contains the entire toolchain required to build extensions for CPython 2.7 (and maybe even CPython 2.7 itself? Let's see...) and is thus extremely lightweight compared to the setup described above.
I am using it as the default C compiler in JyNI's DemoExtension's makefile now.
The next step was to build JyNI itself on Windows. With lessons learned from the DemoExtension it was straight forward to setup a Windows makefile for JyNI itself. However, MSVC 9.0 requires C89 code and converting JyNI's entire codebase to C89 is a different story than converting the DemoExtension.

Return to Linux: Using gcc in C89 mode vs cl.exe with Wine

During work on C89 conversion I suffered from mouse cursor freezes on Windows 10 (luckily the touch screen kept working so I could save the work and cleanly restart). Also the system appears to be constantly slower than Linux on the same machine and finally the continuous annoyance by automatic updates contributed its part to my wish to return to my Linux environment for developing.

So I checked if it would be possible to configure gcc for C89 in order to do at least the C99 to C89 conversion from Linux. Unfortunately as soon as you call gcc with -std=c89 it starts complaining about “// ...” C++ style comments, which are still fine for MSVC 9.0. There is apparently no option to run gcc in C89 mode except for C++ style comments. This puzzles me, given that it looks like a popular use case to assert MSVC compatibility of cross platform code. So I got the idea to perform preprocessing separately, i.e. running the preprocessor in C99 mode to strip away comments, while doing the actual compile task in C89 mode. However, this worked, but still did not yield MSVC compliant C code – there must be more to it. Same result with using -ANSI.

Finally I gave up the ggc-approach (didn't try with clang) and tried to run cl.exe using Wine. This basically worked, but had issues with passing space-containing paths, even when surrounded by quotation marks. Maybe this would have been solvable by fiddling around with makefile escape sequence notation (I tried this in various ways, but without success, so there must be more to it). Instead I now let the makefile create symlinks to the folders in question. This works like a charm and resulted in a wine-based makefile for JyNI (linking not yet tested).

And further – that DemoExtension also was not working when built using setup.py rather than by my makefile. That was a drawback, but remember that DemoExtension was working fine with CPython, even if built with the makefile.
After carefully studying the difference between the build commands I found out that setup.py would add the /MD flag. Beneath several other options, e.g. /nologo /Ox /W3 /GS- /DNDEBUG, /MD was the one that made the difference. It tells MSVC to link msvcrt.dll, precisely speaking msvcr90.dll for Python 2.7.
I guess the DemoExtension build without explicit /MD worked, because it still linked libcmt.lib. However, after adding…

After fixing the remaining symbol errors MSVC successfully created JyNI.dll for the very first time – what a historical moment!
Unfortunately the euphoria didn't last long, because when loading DemoExtension, it directly crashed.
After some investigation I found out that JyNI.dll was successfully loaded by the JVM and I also verified that code in the dll was actually accessed and properly run. Even the initial call into DemoExtension.pyd worked. However, the call to Py_InitModule4 crashed:
It seemed like symbols from JyNI.dll were not exported for access by DemoExtension.pyd. Strangely, the error message wasn't like one would expect for this. It rather looked like DemoExtension.pyd was calling into CPython's python27.dll, i.e. the one installed on the system.

The definite proof of concept
After some reading in MSDN docs I learned that the names of dll dependencies are hard coded at compile time, which means that DemoExtension.pyd will insist on loading python27.dll, no mat…

So, the first two weeks of GSoC are over now and I used the time to finalize JyNI 2.7-alpha.4 release. The project achieved an initial NumPy support in late 2016, but that milestone was not releasable until now, because it requires Jython 2.7.1, which suffered an unfortunate delay due to several release-blocking issues.
Recently, Jython 2.7.1RC2 was released, which I regarded as sufficient base for a new JyNI release. Even more important, fixing up loose ends for a release was the ideal work to get back into project detail after more than half a year.
First Week
Working on the release, I soon recognized some memory flaws on a 32 bit system, which I could luckily solve as of da5fc5c (debugging this stuff on my old 32 bit desktop machine is a pain).
In earlier work on JyNI I had added some control for how specific extensions perceive platform detection via os.name and sys.platform on Jython using techniques one might call "Python black magic" (see follow up report of JyNI GSoC…