Boost C++ Libraries

...one of the most highly regarded and expertly designed C++ library projects in the world. Herb Sutter and Andrei Alexandrescu, C++ Coding Standards

Boost.Python

March 2002 Progress Report

Contents

Accomplishments
Calling Python from C++
Virtual Functions
Abstract Classes
C++ Implicit Conversions
C++ Data Members
Miscellaneous
The Near future
Notes

Accomplishments

March was mostly devoted to the reimplementation of features from Boost.Python v1, and some new features. Re-examination of the features from Boost.Python v1 allowed me to make significant improvements.

Calling Python from C++

The ability to call Python from C++ is crucial for virtual function support. Implementing this feature well for V2 proved to be more interesting than I expected. You can review most of the relevant design decisions here.

One point which isn't emphasized in that document is that there are subtle differences in the way from_python conversions work when used for C++ function arguments and Python function return values. In particular, while T const& arguments may invoke rvalue converters, a reference-to-const return value requires an lvalue converter, since a temporary conversion result would leave the returned reference dangling.

I'm not particularly pleased with the current callback interface, since it usually results in constructs like:

return returning<X&>::call(f, obj);
However, I think the following may be possible and I plan to investigate:
return apply<X&>(f, obj);
I'm open to suggestion for better names (and syntaxes)!

Virtual Functions

Once Python callbacks were implemented, it was just a short step to implementing virtual functions. Python extension class exposing a C++ class whose virtual functions are overridable in Python must actually hold a C++ instance of a class derived from the one exposed to Python. Needing some way for users to specify that class, I added an optional template argument to value_holder_generator and pointer_holder_generator<> to specify the class actually held. This move began to put pressure on the class_<> interface, since the need for the user to produce complicated instantations of class_<> was increased:
class<Foo, bases<>, value_holder_generator<Foo_callback> >("Foo")
.def("hello", &Foo::hello)
...

Abstract Classes

Normally when a C++ class is exposed to Python, the library registers a conversion function which allows users to wrap functions returning values of that type. Naturally, these return values are temporaries, so the conversion function must make a copy in some dynamically-allocated storage (a "holder") which is managed by the corresponding Python object.

Unfortunately, in the case of abstract classes (and other types without a publicly-accessible copy constructor), instantiating this conversion function causes a compilation error. In order to support non-copyable classes, there had to be some way to prevent the library from trying to instantiate the conversion function. The only practical approach I could think of was to add an additional template parameter to the class_<> interface. When the number of template parameters with useful defaults begins to grow, it is often hard to choose an order which allows users to take advantage of the defaults.

This was the straw that broke the class_<> interface's back and caused the redesign whose outcome is detailed here. The approach allows the user to supply the optional parameters in an arbitrary order. It was inspired by the use of named template parameters in the Boost Iterator Adaptor Library, though in this case it is possible to deduce the meaning of the template parameters entirely from their type properties, resulting in a simpler interface. Although the move from a policy-based design to what resembles a configuration DSL usually implies a loss of flexibility, in this case I think any costs are far outweighed by the advantages.

Note: working around the limitations of the various compilers I'm supporting was non-trivial, and resulted in a few messy implementation details. It might be a good idea to switch to a more-straightforward approach once Metrowerks CodeWarrior Pro8 is released.

C++ Implicit Conversions

Support for C++ implicit conversion involves creating from_python converters for a type U which in turn use from_python converters registered for a type T where there exists a implicit conversion from T to U. The current implementation is subject to two inefficiencies:
  1. Because an rvalue from_python converter produces two pieces of data (a function and a void*) from its convertible() function, we end up calling the function for T twice: once when the converter is looked up in the registry, and again when the conversion is actually performed.
  2. A vector is used to mark the "visited" converters, preventing infinite recursion as T to U and U to T converters continually search through one-another.
I consider the former to be a minor issue. The second may or may not prove to be computationally significant, but I believe that architecturally, it points toward a need for more sophisticated overload resolution. It may be that we want CLOS-style multimethod dispatching along with C++ style rules that prevent more than one implicit conversion per argument.

C++ Data Members

To supply the ability to directly access data members, I was able to hijack the new Python property type. I had hoped that I would also be able to re-use the work of make_function to create callable python objects from C++ functions which access a data member of a given class. C++ facilities for specifying data member pointer non-type template arguments require the user to explicitly specify the type of the data member and this under-utilized feature is also not well-implemented on all compilers, so passing the member pointer as a runtime value is the only practical approach. The upshot is that any such entity would actually have to be a function object, and I haven't implemented automatic wrapping of C++ callable function objects yet, so there is less re-use in the implementation than I'd like. I hope to implement callable object wrapping and refactor this code one day. I also hope to implement static data member support, for which Python's property will not be an appropriate descriptor.

Miscellaneous


The Near Future

Before April 15th I plan to
  1. Document all implemented features
  2. Implement a CallPolicy interface for constructors of wrapped classes
  3. Implement conversions for char types.
  4. Implement automated code generation for all headers containing families of overloaded functions to handle arbitrary arity.
I also hope to implement a mechanism for generating conversions between arbitrary Python sequences and C++ containers, if time permits (and others haven't already done it)!

Notes

The older version of KCC used by Kull is generating lots of warnings about a construct I use to instantiate static members of various class templates. I'm thinking of moving to an idiom which uses a function template to suppress it, but worry about bloating the size of debug builds. Since KCC users may be moving to GCC, I'm not sure that it's worth doing anything about it.

Revised 13 November, 2002

© Copyright Dave Abrahams 2002.