Boost C++ Libraries

In C++, we often deal with arguments and return types such as pointers and references. Such primitive types are rather, ummmm, low level and they really don't tell us much. At the very least, we don't know the owner of the pointer or the referenced object. No wonder languages such as Java and Python never deal with such low level entities. In C++, it's usually considered a good practice to use smart pointers which exactly describe ownership semantics. Still, even good C++ interfaces use raw references and pointers sometimes, so Boost.Python must deal with them. To do this, it may need your help. Consider the following C++ function:

X& f(Y& y, Z* z);

How should the library wrap this function? A naive approach builds a Python X object around result reference. This strategy might or might not work out. Here's an example where it didn't

>>> x = f(y, z) # x refers to some C++ X
>>> del y
>>> x.some_method() # CRASH!

What's the problem?

Well, what if f() was implemented as shown below:

X& f(Y& y, Z* z)
{
    y.z = z;
    return y.x;
}

The problem is that the lifetime of result X& is tied to the lifetime of y, because the f() returns a reference to a member of the y object. This idiom is is not uncommon and perfectly acceptable in the context of C++. However, Python users should not be able to crash the system just by using our C++ interface. In this case deleting y will invalidate the reference to X. We have a dangling reference.

Here's what's happening:

We could copy result into a new object:

>>> f(y, z).set(42) # Result disappears
>>> y.x.get()       # No crash, but still bad
3.14

This is not really our intent of our C++ interface. We've broken our promise that the Python interface should reflect the C++ interface as closely as possible.

Our problems do not end there. Suppose Y is implemented as follows:

struct Y
{
    X x; Z* z;
    int z_value() { return z->value(); }
};

Notice that the data member z is held by class Y using a raw pointer. Now we have a potential dangling pointer problem inside Y:

>>> x = f(y, z) # y refers to z
>>> del z       # Kill the z object
>>> y.z_value() # CRASH!

For reference, here's the implementation of f again:

X& f(Y& y, Z* z)
{
    y.z = z;
    return y.x;
}

Here's what's happening:

Call Policies

Call Policies may be used in situations such as the example detailed above. In our example, return_internal_reference and with_custodian_and_ward are our friends:

def("f", f,
    return_internal_reference<1,
        with_custodian_and_ward<1, 2> >());

What are the 1 and 2 parameters, you ask?

return_internal_reference<1

Informs Boost.Python that the first argument, in our case Y& y, is the owner of the returned reference: X&. The "1" simply specifies the first argument. In short: "return an internal reference X& owned by the 1st argument Y& y".

with_custodian_and_ward<1, 2>

Informs Boost.Python that the lifetime of the argument indicated by ward (i.e. the 2nd argument: Z* z) is dependent on the lifetime of the argument indicated by custodian (i.e. the 1st argument: Y& y).

It is also important to note that we have defined two policies above. Two or more policies can be composed by chaining. Here's the general syntax:

policy1<args...,
    policy2<args...,
        policy3<args...> > >

Here is the list of predefined call policies. A complete reference detailing these can be found here.

The following illustrates a scheme for manually wrapping an overloaded member functions. Of course, the same technique can be applied to wrapping overloaded non-member functions.

We have here our C++ class:

struct X
{
    bool f(int a)
    {
        return true;
    }

    bool f(int a, double b)
    {
        return true;
    }

    bool f(int a, double b, char c)
    {
        return true;
    }

    int f(int a, int b, int c)
    {
        return a + b + c;
    };
};

Class X has 4 overloaded functions. We will start by introducing some member function pointer variables:

bool    (X::*fx1)(int)              = &X::f;
bool    (X::*fx2)(int, double)      = &X::f;
bool    (X::*fx3)(int, double, char)= &X::f;
int     (X::*fx4)(int, int, int)    = &X::f;

With these in hand, we can proceed to define and wrap this for Python:

.def("f", fx1)
.def("f", fx2)
.def("f", fx3)
.def("f", fx4)

Boost.Python wraps (member) function pointers. Unfortunately, C++ function pointers carry no default argument info. Take a function f with default arguments:

int f(int, double = 3.14, char const* = "hello");

But the type of a pointer to the function f has no information about its default arguments:

int(*g)(int,double,char const*) = f;    // defaults lost!

When we pass this function pointer to the def function, there is no way to retrieve the default arguments:

def("f", f);                            // defaults lost!

Because of this, when wrapping C++ code, we had to resort to manual wrapping as outlined in the previous section, or writing thin wrappers:

// write "thin wrappers"
int f1(int x) { return f(x); }
int f2(int x, double y) { return f(x,y); }

/*...*/

    // in module init
    def("f", f);  // all arguments
    def("f", f2); // two arguments
    def("f", f1); // one argument

When you want to wrap functions (or member functions) that either:

BOOST_PYTHON_FUNCTION_OVERLOADS

Boost.Python now has a way to make it easier. For instance, given a function:

int foo(int a, char b = 1, unsigned c = 2, double d = 3)
{
    /*...*/
}

The macro invocation:

BOOST_PYTHON_FUNCTION_OVERLOADS(foo_overloads, foo, 1, 4)

will automatically create the thin wrappers for us. This macro will create a class foo_overloads that can be passed on to def(...). The third and fourth macro argument are the minimum arguments and maximum arguments, respectively. In our foo function the minimum number of arguments is 1 and the maximum number of arguments is 4. The def(...) function will automatically add all the foo variants for us:

def("foo", foo, foo_overloads());

BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS

Objects here, objects there, objects here there everywhere. More frequently than anything else, we need to expose member functions of our classes to Python. Then again, we have the same inconveniences as before when default arguments or overloads with a common sequence of initial arguments come into play. Another macro is provided to make this a breeze.

Like BOOST_PYTHON_FUNCTION_OVERLOADS, BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS may be used to automatically create the thin wrappers for wrapping member functions. Let's have an example:

struct george
{
    void
    wack_em(int a, int b = 0, char c = 'x')
    {
        /*...*/
    }
};

The macro invocation:

BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS(george_overloads, wack_em, 1, 3)

will generate a set of thin wrappers for george's wack_em member function accepting a minimum of 1 and a maximum of 3 arguments (i.e. the third and fourth macro argument). The thin wrappers are all enclosed in a class named george_overloads that can then be used as an argument to def(...):

.def("wack_em", &george::wack_em, george_overloads());

See the overloads reference for details.

init and optional

A similar facility is provided for class constructors, again, with default arguments or a sequence of overloads. Remember init<...>? For example, given a class X with a constructor:

struct X
{
    X(int a, char b = 'D', std::string c = "constructor", double d = 0.0);
    /*...*/
}

You can easily add this constructor to Boost.Python in one shot:

.def(init<int, optional<char, std::string, double> >())

Notice the use of init<...> and optional<...> to signify the default (optional arguments).

It was mentioned in passing in the previous section that BOOST_PYTHON_FUNCTION_OVERLOADS and BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS can also be used for overloaded functions and member functions with a common sequence of initial arguments. Here is an example:

void foo()
{
   /*...*/
}

void foo(bool a)
{
   /*...*/
}

void foo(bool a, int b)
{
   /*...*/
}

void foo(bool a, int b, char c)
{
   /*...*/
}

Like in the previous section, we can generate thin wrappers for these overloaded functions in one-shot:

BOOST_PYTHON_FUNCTION_OVERLOADS(foo_overloads, foo, 0, 3)

Then...

.def("foo", (void(*)(bool, int, char))0, foo_overloads());

Notice though that we have a situation now where we have a minimum of zero (0) arguments and a maximum of 3 arguments.

Manual Wrapping

It is important to emphasize however that the overloaded functions must have a common sequence of initial arguments. Otherwise, our scheme above will not work. If this is not the case, we have to wrap our functions manually.

Actually, we can mix and match manual wrapping of overloaded functions and automatic wrapping through BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS and its sister, BOOST_PYTHON_FUNCTION_OVERLOADS. Following up on our example presented in the section on overloading, since the first 4 overload functins have a common sequence of initial arguments, we can use BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS to automatically wrap the first three of the defs and manually wrap just the last. Here's how we'll do this:

BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS(xf_overloads, f, 1, 4)

Create a member function pointers as above for both X::f overloads:

bool    (X::*fx1)(int, double, char)    = &X::f;
int     (X::*fx2)(int, int, int)        = &X::f;

Then...

.def("f", fx1, xf_overloads());
.def("f", fx2)