Embedding Python in Your C Programs

C, meet Python. Python, this is C. With surprisingly little effort, the Python interpreter can be integrated into your program to add features quickly that could take months if written entirely in C.

The language of choice for large, high-performance applications
in Linux is almost always C, or somewhat less often C++. Both are
powerful languages that allow you to create high-performance natively
compiled programs. However, they are not languages that lend themselves
to runtime flexibility. Once a C/C++ application is compiled, its code is
pretty much static. At times, that can be a real hindrance. For example,
if you want to allow users of a program to create plugins easily that
extend the application's functionality, you have to deal with complex
dynamic linking issues that can cause no end of headaches. Additionally,
your users will have to know C/C++ in order to extend the application,
which severely limits the number of people capable of writing extensions.

A much better solution is to provide your users with a scripting language
they can use to extend your application. With a scripting language,
you will tend to have much more runtime flexibility, as well as shorter
development times and a lower learning curve that will extend the base
of users capable of creating extensions.

Unfortunately, creating a scripting language is very much a nontrivial
task that easily could become a major portion of your program.
Fortunately, you don't need to create a scripting language. With Python,
you can embed the interpreter directly into your application and expose
the full power and flexibility of Python without adding very much code
at all to your application.

Including Python in an Application

Including the Python interpreter in your program is extremely simple.
Python provides a single header file for including all of the definitions
you need when embedding the interpreter into your application,
aptly named Python.h. This contains a lot of stuff, including several of
the standard headers. For compiling efficiency, it might be nice if you
could include only those parts of the interface that you actually intend
to use, but unfortunately Python doesn't really give you that option.
If you take a look at the Python.h file, you'll see that it defines
several important macros and includes a number of common headers that
are required by the individual components included later in the file.

To link your application to the Python interpreter at compile time,
you should run the python-config program to get a list of the linking
options that should be passed to the compiler. On my system, those are:

-lpython2.3 -lm -L/usr/lib/python2.3/config

A Very Simple Embedded App

So, how much code does it take to run the Python interpreter from a
C app? As it turns out, very little. In fact, if you look at Listing
1, you'll see that it can be done in as little as three lines of code,
which initialize the interpreter, send it a string of Python code to
execute and then shut the interpreter back down.

Or, you could embed an interactive Python terminal in your program by
calling Py_Main() instead, as in Listing 2. This brings up the
interpreter just as if you'd run Python directly from the command line.
Control is returned to your application after the user exits from
the interpreter shell.

Embedding the interpreter in three lines of code is easy enough,
but let's face it, just executing arbitrary strings of Python code inside
a program is neither interesting nor all that useful. Fortunately,
it's also far from the extent of what Python allows. Before I get too
deep into what it can do though, let's take a look at initializing the
environment that Python executes within.

When you run the Python interpreter, the main environment context is
stored in the __main__ module's namespace dictionary. All functions,
classes and variables that are defined globally can be found in this
dictionary. When running Python interactively or on a script file, you
rarely need to care about this global namespace. However, when running
the embedded interpreter, you'll often need to access this dictionary to
get references to functions or classes in order to call or construct them.
You also may find that you occasionally want to copy the global dictionary
so that different bits of code can be run in distinct environments.
For instance, you might want to create a new environment for each plugin
that you load.

To get at the __main__ module's dictionary, you first need to
get a reference to the module. You can do this by calling the
PyImport_AddModule() function, which looks up the module name you
supply and returns a PyObject pointer to that object. Why a PyObject?
All Python data types derive from PyObject, which makes it a handy
lowest-common denominator. Therefore, almost all of the functions that you'll
deal with when interacting with the Python interpreter will take or return
pointers to PyObjects rather than another more specific Python data type.

Once you have the __main__ module referenced by a PyObject, you can
use the PyModule_GetDict() function to get a reference to the main
module's dictionary, which again is returned as a PyObject pointer.
You can then pass the dictionary reference when you execute other Python
commands. For example, Listing 3 shows how you could duplicate the
global environment and execute two different Python files in separate
environments.

The Python/C API is very low level, verbose, painful to work with, and highly error prone. With C++ code in particular it just sucks - you'll spend half your time writing const_cast("blah") to work around "interesting" APIs or writing piles of "extern C" wrapper functions, and the rest of your time writing verbose argument encoding/decoding or reference counting code. It's immensely frustraing to use plain C to write a Python object to wrap a C++ object.

Do yourself a favour, and once you've got embedding working, expose the Python interface to your program using a higher level tool. I hear SWIG is pretty good, especially for plain C code, but haven't used it myself (I work with heavily templated C++ with Qt). SIP (used to make PyQt) from Riverbank computing has its advantages also, and is good if you want your Qt app's API to integrate cleanly into PyQt. Otherwise, I'd suggest the amazing Boost::Python for C++ users, as it's ability to almost transparently wrap your C++ interfaces, mapping them to quite sensible Python semantics, is pretty impressive.

Boost::Python has the added advantage that you can write some very nice C++ code that integrates cleanly with Python. For example, you can iterate over a Python list much like a C++ list, throw exceptions between C++ and Python, build Python lists as easily as (oversimplified example):

The equivalent Python/C API code is longer, filled with dangerous reference count juggling, contains a lot of manual error checking that's often ignored, and is a lot uglier.

With regards to the article above, it's good to see things like this written. I had real trouble getting started with embedding Python, and I think this is a pretty well written intro. I do take issue with one point, though, and that's duplicating the environment. Cloning the main dict does not provide separate program environments - very far from it. It only gives them different global namespaces. Interpreter-wide state changes still affect both programs. For example, if one program imports a module, the other one can see it in sys.modules ; if one program changes a setting in a module, the other one is affected. Locale settings come to mind. Most well designed modules will be fine, but you'll run into the odd one that thinks that module-wide globals are a good idea and consequently chokes.

Unfortunately, the alternative is to use sub-interpreters. Sub-interpreters are a less than well documented part of Python's API, and as they rely on thread local storage they're hopeless in single threaded programs. They can be made to work (see Scribus, for example) but it's not overly safe, and will abort if you use a Python debug build.

When you combine this with a GUI toolkit like Qt3 that only permits GUI operations from the main thread (thankfully, the limitiation is relieved by Qt4), this becomes very frustrating. If you're not stuck with this limitation, you can just spawn off a thread for your users' scripts, and should consider designing your interface that way right from the start.