Numscons: current state, future, alternative build tools for numpy

Several people in numpy/scipy community have raised build issues recently. Brian Granger has wondered whether numscons is still maintained, and Ondrej recently asked why numscons was not the default build tool for numpy. I thought I owe some explanations and how I see the future for numscons.

First, I am still interested in working on numscons, but lately, time has become a sparser ressource: I am at the end of my PhD, and as such cannot spend too much time on it. Also, numscons is more or less done, in the sense that it does what it was supposed to do. Of course, many problems remain. Most of them are either implementation details or platform specific bugs, which can only be dealt if numscons is integrated to numpy — which raises the question of integrating numscons into numpy.

Currently, I see one big limitation in numscons: the current architecture is based on launching a scons subprocess for every subpackage, sequentially. As stupid as this decisions may sound, there are relatively strong rationales for this. First, scipy is designed as a set of almost independent packages, and that’s true for the build process as well. Every subpackage declares its requirements (blas, lapack, etc…), and can be build independently of the others: if you launch the build process at the top of the source tree, the whole scipy is built; if you launch it in scipy/sparse/sparsetools, only sparsetools is built. This is a strong requirement: this is impossible to do with autotools, for example, unless each subpackage has its own configure (like gcc for example).
It is possible in theory with scons, but it is practically almost impossible to do so while staying compatible with distutils, because of build directory issues (the build directory cannot be a subdirectory of the source tree). When I started numscons, I could see only two solutions: launching independent scons builds for each subpackage, or having a whole source tree with the configuration at the top, but scons is too slow for the later solution (although it is certainly the best one from a design POV).

The second problem is that scons cannot be used as library. You cannot do something like “from scons import *; build(‘package1′); build(‘package2′)”. Which means the only simple solution to have independent builds for each package is to launch independent scons processes. Having to use subprocesses to launch scons is the single fundamental numscons issue.

1 Because scons is slow to start (it needs to check for tools, etc…), it means no-op builds are slow (it takes 20 seconds to complete a no-op full scipy built on a more than decent machine, which is why numscons has an option–package-list to list the packages to rescan, but that’s nothing more than an ugly hack).

2 Error handling is hard to do: if scons fails, it is hard to pass useful information back to the calling process

3 Since distutils still handle installation and tarballs generation, it needs to know about the source files. But since only scons knows about it, it is hard to pass this information back to distutils from scon. Currently, it only works because it knows the sources from the conventional setup.py files.

Another limitation I see with scons is the code quality: scons is a relatively old project, and focused a lot on backward compatibility, with a lot of cruft (scons still support python 1.5). There is still a lot of development happening, and is still supported; scons is used in several high profile projects (some vmware products are built with scons, Intel acknowledges its use internally, Google uses it – Steve Knight, the first author of scons works at Google on the Chrome project, and chromes sources have scons scripts). But there is a lot of tight coupling, and changing core implementations issues is extremely challenging. It is definitely much better than distutils (in the sense that in distutils, everything is wrong: the implementation, the documentation, the UI, the concepts). But fixing scons tight coupling is a huge task, to the point where rewriting from scratch some core parts may be easier (see here). There are also some design decisions in scons which are not great, like options handling (everything is passed through Environments instances, which is nothing more than a big global variable).

A potential solution would be to use waf insteaf of scons. Waf started as a scons fork, and dropped backward compatibility. Waf has several advantages:

  • it is much smaller and nicer implementation-wise than scons (core waf is ~ 4000 LOC, scons is ten times more). There are some things I can do today in waf I still have no idea how to do in scons, although I am much more familiar with the latter.
  • waf is much faster than scons (see here for some benchmarks)
  • it seems like waf can be used as a library

But:

  • waf is not stable (the API kept changing; the main waf developer said he would focus on stability from now on)
  • waf does not support Fortran – this one should be relatively easy to solve (I have some code working already for most fortran requirements in scipy)
  • I am still not satisfied with its tool handling – that’s a hard problem though, I have not seen a single build tool which handle this well. That’s not a fundamental issue to prevent the use of waf.
  • support on windows looks flaky

There may also be hope to see more cross-pollination between scons and waf – but that’s a long term goal, unless someone can work on it fulltime for at least several weeks IMHO. Porting numscons to waf should be relatively easy once fortran is handled – I think basic porting could be done in one day or two.

Ondrej also mentioned cmake: I think it would be very hard to build numpy with cmake, because it means you have to give up distutils entirely. How to make it work with easy_install ? How to generate tarballs, windows installers, etc… ? If anyone wants to try something else here is my suggestion: ignoring tarball/install issues, try to build numpy.core alone on windows, mac os X and linux, with either Atlas, Accelerate or nothing. Use scons scripts as informations for the checks to do (it is much more readable than distutils setup.py files). If you can do this, you will have solved most build details necessary to build the whole scipy. It will certainly give you a good overview of the difficulty of the task.

8 responses to “Numscons: current state, future, alternative build tools for numpy

  1. Richard

    I feel your pain. Waf’s instability is very irritating sometimes, though it is much better since 1.5. I’m overly familiar with autotools, SCons, CMake and Waf. Waf is far from perfect, but it is the best of the bunch IMO. When you have interdependent libraries, for example, the export_incdir and uselib_local shortcuts are brilliant. In the other tools you end up with “../../../path/to/someplace” and it gets messy quickly.

    Waf handles sub-builds quite well.

    def configure(conf):
    conf.sub_config(‘subproject’)

    def build(bld):
    bld.add_subdirs(‘subproject’)

    (the API here is not symmetric – Build’s add_subdirs accepts multiple directories, Configure’s sub_config does not) There can be some stepping-on-each-other’s-toes problems if both define different ways to build the same type of thing. Building software is harder than it sounds :-/

  2. Marcelo

    I never understood why we need distutils in numpy/scipy.
    It has a spaghetti like design, and this autodetection of flags and libraries is a nightmare for maintainers and only brings bugs.

    scipy/numpy have only few dependencies: blas, cblas, lapack, umfpack. So why not just using python distutils, and let the user specify the libaries with simple variable WITHOUT autodetection, which never worked nicely in numpy anyway. Let the user select its own flags, compilers, lapack libraries, etc.
    It is quite industrious to do at the moment (just one example: try to set both FFLAGS and LDFLAGS on x86_64 or try to use GOTO BLAS+netlib cblas/lapack instead of atlas).

    Why not just using environment variables, with an unset default?
    In *nix, it could be like:
    LAPACK_LIBS=”-L/usr/lib -llapack”
    CBLAS_LIBS=”-L/usr/lib/atlas -lptcblas -latlas -lpthread”
    CBLAS_CFLAGS=”-I/usr/include/atlas”
    and use the standard CC, CXX, CFLAGS, CXXFLAGS, FFLAGS, FCFLAGS, LDFLAGS, etc… compiler flags instead of trying to detect machine, compiler, compilers, etc…
    cblas, blas, lapack implementations are standard, so there is no need to force atlas on the code.

    The current site.cfg file doesn’t have as much flexibility as this, and the current environment variables assume too atlas is built some way. Note pretty much all Linux and BSD distributions have to patch heavily the site.cfg, the setup.py and sometimes the distutils routines to work out numpy/scipy installation.

    The only advantage I see in scons/cmake is the -jN option.

  3. cournape

    I agree with you that python distutils is horrible code; I don’t think anybody likes it. I also think you severely underestimate the complexity of what is needed to build numpy :)

    First, the problem with flags is difficult: we need to be able to overwrite the flags, and people usually want to add some flags. The only way to make this reliably is to have something like autoconf, with configuration tests which depend on the compilation flags (to detect errors early). Concerning CPU detection, I agree with you that it is too fragile: I have disabled most of them for upcoming version of numpy, I don’t think it brings anything but problems when detection fails.

    Concerning libraries: blas/lapack are very far from being consistent. Every single implementation is different: different flags, different names, etc… If every library was like blas/lapack, I think people would have give up on shared libraries a long time ago. The current autodetection scheme is very fragile, I don’t disagree with this – but asking users to use their own flags is even worse IMHO.

    You’re right that the solution is easier customization – but with good default. We need both.

    Concerning scons, the -jN option is a very minor feature compared to other benefits IMHO. First, scons, for all its flaws, is a decent build system, with some good concepts, some documentation and some design. We benefit from automatic dependencies, the configuration tests are like autoconf, that is much more robust, you can modify flags (CFLAGS, CXXFLAGS, etc… should work with numscons). Numscons is ~ 3000 LOC (counting some code I needed to copy from distutils), compared to 10000 LOC for numpy.distutils.

  4. Marcelo

    I thought python distutils is horrible, but numpy.distutils is along the line. So my point was not to base the next numpy/scipy installer on numpy.distutils.

    Yes, good default will help, but they’ll have to be easy to overwrite, and unset. numpy has internal flags, but should not overwrite the user flags, which are CFLAGS, CXXFLAGS, FC, FFLAGS, etc…
    Probably scons will help, and it also supports windows stuff, which is more complex with autoconf.

    blas,cblas and lapack libraries do have a consistent api, defined in netlib reference libraries. Different implementations (mkl, acml, goto, atlas, essl, sunstudio, etc…) use different flags and libraries names and complement the original routines with extra blobs, but they can be called the same way within numpy/scipy. So I don’t think this is numpy’s job to try to detect a specific library. Look right now at the ATLAS detection in numpy.distutils, it needs ATLAS to build the dotblas, but does not use anything else than standard cblas routines. So why not gotoblas+netlib-cblas? My point is: don’t check every single existing implementation; set a default one (e.g. the netlib ones), and make tests only on the api routines.

    With autoconf, there is a blas.m4 and lapack.m4 macros which are fairly simple to overwrite, e.g.:
    –with-blas=”-L/home/mandrier/atlas -lptblas -latlas -lpthread”
    and have default too. So why not something like that?

    I don’t think numpy/scipy should not be that complex to install-
    Check for system libraries, test compilers, enable/disable blas/cblas/lapack, and if enabled, check 1) user set libs, 2) default libs with a trivial test of the standard api.

    Looking at numscons code, it seems there is a lot of code to detect every single fortran compiler, perf libraries, flag setting, fftw (which is not even used anymore by scipy), there exist on the planet. This is too ambitious. Why not starting something much more simple, and more robust? The problem with numpy.distutils was it started too complex and ended up spaghetti.

    People installing numpy/scipy by hand should know what they are doing, and if not, they should use the packages from their distributions. The current state is: it’s not easy either for them to set up by hand, nor it’s easy for distribution maintainers.

    It used to be difficult to set fortran compilers with scons, I don’t know how hard it is at the moment.

  5. cournape

    I think you don’t realize all that is done by numpy.distutils. The blas/lapack detection is only like 20 % of the code. A lot of code is needed for fortran (distutils does not support fortran at all, obviously), for mingw support, etc… There are 10000 LOC in numpy.distutils, so any change would be difficult – and many people use numpy.distutils outside numpy, so we have to keep the code anyway. Also, numpy.distutils has more code for checking libraries, headers, etc… (I for example added a few days ago something to check sizeof types in a cross-compilation setting).

    Concerning options –with-blas, etc..: I agree this is desirable, but it is not easily possible with distutils: you need to pass the options to each command, which quickly becomes impractical. It is one big hindrance of distutils, this design by commands.

    Saying that it is as easy as “checking config 1, checking config 2″ is not very useful for build issues: the whole difficult is to make it work on a vast range of configurations. How do you make this work for Visual Studio, etc… Note that both numscons and numpy.distutils can do things that autoconf cannot (fortran/C linking on windows, with MS compilers)

    The code to detect blas/lapack is numscons is horrible, that’s the part which needs the most work. But it is difficult to do much better from a user POV, because of this per command options handing in distutils. Again, I agree something like configure/build/install is much better. But that would mean getting away entirely from distutils, with all the problems related to external tools expecting setup.py (easy_install, etc…).

  6. cournape

    Concerning the blas/lapack library: they are not as consistent as you think they are. First, not only the library names are different depending on the library, but also on the platform (Atlas is different on FreeBSD and Linux for example), on the API coverage, etc… Many disttributions used to have incomplete Blas or Lapack. The numscons code for blas/lapack detection is not much worse than blas/lapack m4 code, from an implementation detail POV.

  7. Marcelo

    Well I do realize completely what is done by numpy.distutils and that’s my complaint: it does too much to the point it leads to user dissatisfaction and bugs.

    blas, cblas and lapack api are consistent enough, otherwise you would not be able to use a lot of software.
    They are different in the extra routines they need, in atlas, they are the ATL_*, in mkl they are the many many others.
    ATLAS is a fragile build and tunes the library to the machine it has been compiled, so yes, it is different on FreeBSD and Linux, but still builds the same interface routines as netlib cblas for numpy to call.

    What counts for numpy is that the libraries contain the needed symbols, and I don’t think numpy should rely on extra symbols of a specific blas/cblas/lapack implementation.

    In most FreeBSD packages depending on lapack, you can either use the reference one or atlas. On Gentoo Linux, you can interchange at least 4 different lapack with their eselect mechanism. And las t I checked on Debian, you can choose either refblas, or atlas for many packages. Many distributions implement the “virtual” package. And which distributions have incomplete blas or lapack?

    Right now in numpy.distutils, it is painful to select any other lapack library than ATLAS built by the user and placed in a specific directory.

    I agree even if I don’t care, numpy.distutils and numscons should have support for MS. But how much code is needed for this?

    acx_blas.m4 and acx_lapack.m4 are not great either, it has a lot of autodetection crap, but with one simple command, you can overwrite all the settings.
    Yes, the difficult thing is to make it work on all possible configurations. There will be always one configuration numpy won’t be ready for. Trying to autodetect every single configuration does not lead anywhere but bug report.

    I would say keep only the common part to all: the APIs, the flags (FC, FFLAGS, etc…), test on what you can test only. Then let the community test and complement the other configuration.

    Dump numpy.distutils and your life will be easier. Packages depending on it are very high end specialty packages and will be happy to not rely on it anyway. Seeing tickets reports, ML and forums topics relating problems with numpy.distutils, it seems a lot of other users will be happy.

  8. cournape

    Concerning blas/lapack: no, they are not consistent, even if you restrict to the netlib API (which we do; AFAIK, we don’t use any ATLAS specific API). Incomplete blas/lapack: FC up to 5 has the problem, as well as Open suse 9 and 10 (because they built BLAS from LAPACK source code, and LAPACK 3.0 had some missing functions). A same symbol has also different meaning on different platform (because it is sometimes the f2c API, with everything passed by reference, or sometimes the C API, with int/float and co passed by values).

    The ONLY way to solve this problem is to do like R: have our own copy, and use only this. Personally, I would be very inclined to do exactly that, but I am pretty sure many people would be pissed (including distributions because we would duplicate things).

    Using something else than ATLAS is not difficult (but not really documented): just set ATLAS=None when building numpy, and set up library/header directories in site.cfg. If the libraries are installed system-wide, you don’t even need site.cfg.

    About automatic detection: again, I agree the current way is suboptimal. The way to detect libraries by files is fundamentally flawed IMO – the ways numscons does it (like autoconf) is way better. But you can’t have an easy way to set things up because of a fundamental design flaw of distutils (which we cannot fix); if it were, I would have implemented something like –enable-blas= ala autoconf a long time ago.

    Concerning the flags: we cannot change them easily, because we get them from distutils, and distutils does not make the difference between necessary flags (-fPIC, etc…), warn flags and optimizations flags. So it is an all or nothing situation.

    Finally, about windows: yes, that’s a lot of work, at least for numscons.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

February 2009
M T W T F S S
« Dec   Mar »
 1
2345678
9101112131415
16171819202122
232425262728  

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 10 other followers

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: