Recent progress on bento – build numpy !

I have spent the last few days on a relatively big feature for bento: recursive package description. The idea is to be able to simply describe packages in a deeply nested hierarchy without having to write long paths, and to split complicated packages descriptions into several files.

At the bento.info level, the addition is easy:

Subento: numpy/core, numpy/lib ...

It took me more time to figure out a way to do it in the hook file. I ended up with a recurse decorator:

@recurse(["numpy/core/bscript", "numpy/lib/bscript"])
@post_configure
def some_func(ctx):
    ....

I am not sure it is the right solution yet, but it works for now. My first idea was to simply use a recurse function attached to hook contexts (the ctx argument), but I did not find a good way to guarantee an execution order (declaration order == execution order), and it was a bit unintuitive to integrate both hook decorator and the recurse together.

The reason why I tackle this now is that bento is at a stage where it need to be used on “real” builds to get a feeling of what works and what does not. The target is numpy and hopefully later scipy. Although I still hope to integrate waf or scons in bento as the canonical way of building numpy/scipy with bento, this also gives a good test for yaku (my simple build system).

It took me less than half a day to port the scons scripts to bento/yaku. A full build, unnoptimized build of numpy with clang is less than 10 seconds. A no-op build is ~ 150 ms, but as yaku does not have all the infrastructure for header dependency tracking yet, the number for no-op build is rather meaningless.

Advertisements

4 thoughts on “Recent progress on bento – build numpy !

  1. This looks great. I would like to start using Bento for a simple module that will include f2py extensions. I’m not sure if bento/yaku support f2py and cython yet?

    It would be really helpful to have a simple example package (it’s not quite clear to me how to link bento and yaku together). The only slightly tricky thing is that I need to preprocess the fortran files to subsitute the correct integer size to match the native numpy integer type. I would also like the build to complete if any of the extensions fail (so the compiled stuff is optional and I conditionally import it if available in __init__.py). I made a simple example package here http://github.com/robince/testpackage – perhaps it could form the basis of an example.

    Another thing that would be nice is an example of how to build an extension against the GSL (or any other large library that uses the lib-config –prefix idiom for finding compilation arguments). Here is my current setup.py for something that uses this (Cython wrapper to some code that uses GSL):
    http://code.google.com/p/pyentropy/source/browse/trunk/pyentropy/setup.py
    Are bento/yaku at the stage where they could replace something like this? (I guess if they can build numpy they should be).

    • yaku supports cython, but not f2py yet. There is no example on linking yaku/bento yet on purpose, because I am still flushing out the API. If you want to try it, you can look at the bento build of numpy: http://github.com/cournape/numpy/tree/_bento_build. But I cannot stress things enough, it still far from being flushed out – if you are using hook files (the bscript files in numpy bento build), a lot of magic is happening at the moment, and many things are too low level.

      Using the bento.info is pretty safe: the syntax is much less likely to change (I have not changed a single thing for months now, just added new keywords).

      • Thanks very much – I guess I’m jumping the gun a bit then and will wait until things settle down.

        Thanks a lot for doing this btw – I think it is going to be incredibly useful for the scientific python community.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s