June 2002 Progress Report

Contents

July was mostly focused on allowing expressive manipulation of
individual Python objects, or what Ralf Grosse-Kunstleve calls
"Writing Python in C++". The work began with this posting,
which outlines the issues and intention.

The most basic element needed was a replacement for the
reference<> class template and the
ref typedef from Boost.Python v1, a simple smart
pointer to a Python object. The old v1 typedef
"ref" (for
reference<PyObject>) had to be retired because I
thought it would be too confusing given the importance of boost::ref() to this
library. I began a discussionof
possible names, and it was eventually decided
to rename reference to handle and supply a
default argument so that ref could be spelled
handle<> without an additional typedef. There
were also some interface changes to make it safer and more-efficient
to interface with the raw
PyObject*s forced on us by Python's 'C' API. A
discussion of those protocols can be found here.

It is intended that users will seldom need or want to work with
handle<>; its major distinguishing features are
that it gives direct access to the underlying object representation
through operator* and operator->, and
that can be NULL, both sources of danger. Instead the
library provides a class called object, which
encapsulates a valid Python object and provides a similar interface to
Python's.

The first challenge was to provide support for object manipulations
using a Python-like syntax, mostly in the form of operator overloads:

Python

C++

y = x.foo

y = x.attr("foo");

x.foo = 1

x.attr("foo") = 1;

y = x[z]

y = x[z];

x[z] = 1

x[z] = 1;

y = x[3:-1]

y = x.slice(3,-1);

y = x[3:]

y = x.slice(3,_);

y = x[:-2]

y = x.slice(_,-2);

z = x(1, y)

z = x(1, y);

z = x.f(1, y)

z = x.attr("f")(1, y);

not x

!x

x and y

x and y

I'm still a unsatisfied with the interface for attribute access. There
original proposal used a syntax like this one:

y = x._("foo");
x._("foo") = 1;

which was only marginally better than what we've got. Niki Spahiev
then pointed
out a potential conflict with the macro which GNU Gettext suggests
people define. This unfortunate state of affairs forced us into using
attr instead. I'd still like to find a better interface,
but the lack of overloadable C++ operators which aren't already used
in Python is an obstacle. The comma operator is still a possibility,
but it has the wrong precedence:

The object class also provided an opportunity to replace
Boost.Python v1's to_python() as a user-level
interface. Instead, object has a templated constructor
which can be used to convert any C++ object to Python using the same
underlying mechanisms used for the arguments to call<>.

Incidentally, the implementation of operator and conversion support
for object uncovered an inordinate number of compiler bugs in our
targeted platforms. It was a lot more "interesting" than it
should have been.

With object implemented, it was time to begin replacing
the ad-hoc implementations of list, string,
and dictionary supplied by Boost.Python v1 with something
more robust. I started with list as an example. Because
object already provides all of the requisite operators,
publicly deriving list from object seemed like a good
choice. The remaining issues were what do do about the one-argument
list constructor (which in Python attempts to convert its argument to
a list), and how to deal converting with list arguments
to wrapped functions. Some of the issues are laid out in this
thread. Ultimately, it was decided that list(x)
should do the same thing in C++ as in Python (conversion), while
list arguments should only match Python
lists (and list subclasses). The
implementation worked well, and provided a roadmap
for the protocol to be used for implementation of the other built-in
types.

Support for C++ long long and unsigned long
long
(and __int64 on MSVC) to/from python conversions was
added this month. We also improved handling of numeric overflows when
converting, e.g., a Python int to a type with a more limited range of
representation.

As I write this we are already well into the month of July, so I
suggest you consult the Mailing
List Archive if you want to know what's been happening. Otherwise
you'll just have to wait till next month (hopefully the beginning).