This article describes one way to use CPython extensions from IronPython / .NET by embedding CPython itself via Python.NET.

Since this article was written a much more practical approach has been implemented as an Open Source project by Resolver Systems. This project fakes Python25.dll and has a large subset of the Python C API implemented in C#. A large percentage (around a thousand tests at the time of writing) of the Numpy test suite passes when used from IronPython and other whole extension libraries are usable:

The download includes Python.Runtime.dll from Python.NET, for a UCS2 build of Python. This is the right assembly for Python 2.4 on Windows. For Linux you probably want a UCS4 build. The Python.NET distribution contains runtimes for Python 2.4 and 2.5 in both UCS2 and UCS4. This technique does depend on having the appropriate version of Python installed (although you could also ship the relevant Python dll).

This code only works with IronPython 2 because of an annoying bug when passing Arrays as arguments in IronPython 1. This can be worked around, but targeting IronPython 2 may be better as we could look at using the DLR to help with making PyObjects behave like IronPython data types (see below).

The approach in this module, is to embed the CPython interpreter in an assembly and access CPython extensions through the hosted interpreter. This module is far from the final solution and it's not even clear that it is the best approach to take.

At Resolver we have also been experimenting with directly loading CPython assemblies and replacing the CPython API with function pointers that use delegates to call back into managed code. This would also give us binary compatibility and avoid some of the problems caused by hosting a real CPython interpreter. So far we have Python binary extensions loading, calling into our code and then crashing! This is a great first step because it means the basic approach works, and know all we need to do is implement the whole Python C API in managed code...

As you can see, the the pylab module imported from CPython behaves in (apparently) the same way as it does when running directly in CPython.

Caution!

The actual code in test.py imports the sys module from the hosted interpeter, and attempts to adds to sys.path.

Ufortunately this has no effect! (Although the path seems to be set correctly for my computer anyway.) The reason it doesn't work highlights something to be aware of if you want to use this module.

sys=Import('sys')sys.path.append('c:\\Python24\\Lib\\')

In the code above, the sys module is successfully imported as a proxy object. When you access sys.path, this proxy object recognises that you are accessing a Python list (on the CPython side) and copies it across to IronPython for you. This means that the append is executed on the copy, not on the original. d'oh

The solution would be, either to not copy the list and to proxy access to it as well, or to provide functions on the CPython side allowing you to manipulate sys.path.

Installing the import hook allows you import Python binary extensions using normal import statements! Python binary extensions are .pyd files on Windows and .so files on other platforms. To install the import hook, execute the following code:

importcextcext.install()

You can then do things like import cElementTree.

The goal is that eventually this will be build into FePy and enabled by an option, so that you can import CPython modules without having to take any special steps.

Under the hood, the import hook uses the embedding.Import function.

When you use the import hook to import binary modules you may want to do things like setting the import path on the hosted interpreter. Obviously normal import statements (like import sys) will import the IronPython version. To access the builtin modules of CPython you will still need to use embedding.Import.

This provides a very thin wrapper around the CPython embedding API. You have to acquire and release the Global Interpreter Lock (GIL) around every operation and it works with PyObjects which are managed wrappers around CPython types.

The code below imports the PythonEngine and initializes it. It also defines two decorators

GIL acquires and releases the GIL, and wraps all operations with CPython objects.

handle_exception handles exceptions that occur in CPython and reraises them as IronPython exceptions

When you import a module it returns a proxied object. By default all objects you access are proxied objects unless they are a fundamental datatype - which will be converted from a PyObject into the equivalent IronPython type.

Proxied objects let you get and set attributes on them, plus call them. The proxied class is PythonObject:

The ConvertToPy and ConvertToIPy functions do the converting between CPython and IronPython types.

It can convert integers, long integers, strings, booleans and None, lists, tuples and dictionaries. This means that if you call a Python function (or set an attribute) with a reference to an IronPython data structure it will be copied into CPython types. Any return values will be copied from CPython types to IronPython types. See the limitations section below for some of the consequences of this.

It does mean that once you have imported a module you can call functions / methods and instantiate classes from inside CPython. You can also fetch and set attributes and all the right things should happen.

test.py tests the basic functionality of the embedding.py module. It also serves as a usage example.

When run with IronPython 2, it imports echo.py into CPython and passes data back and forth to check that it survives the journey. These are done with asserts, so if any fail then it will bomb out with an error.

As you have probably already gathered, this is an early implementation and suffers from some serious limitations.

Some basic types like complex numbers aren't converted for you. Additionally data structures are copied in and out which is expensive - and you lose changes that CPython makes to mutable data structures you pass in. The technique for copying data structures (in and out), doesn't take into account recursive data structures - and so will probably never terminate if it encounters them.

For more efficient work you can leave data as PyObject structures rather than copy. One avenue of investigation would be to see if we can make these PyObject structures (managed wrappers around the CPython data) behave like their equivalent IronPython objects. This would allow some source code to operate unmodified.

CPython objects (other than the basic datatypes) are proxied. This means that they have the wrong type and magic methods probably won't work.

Oh, and strings from .NET come in as unicode on the CPython side.

Despite these problems, as you can see from the matplotlib demo it works fine. With some CPython helper modules you may be able to solve quite difficult problems with this as a starting point. Feel free to experiment with and extend this code. If you do fix any of the problems then please send the code back to me.