Numscons: current state, future, alternative build tools for numpy

Several people in numpy/scipy community have raised build issues recently. Brian Granger has wondered whether numscons is still maintained, and Ondrej recently asked why numscons was not the default build tool for numpy. I thought I owe some explanations and how I see the future for numscons.

First, I am still interested in working on numscons, but lately, time has become a sparser ressource: I am at the end of my PhD, and as such cannot spend too much time on it. Also, numscons is more or less done, in the sense that it does what it was supposed to do. Of course, many problems remain. Most of them are either implementation details or platform specific bugs, which can only be dealt if numscons is integrated to numpy — which raises the question of integrating numscons into numpy.

Currently, I see one big limitation in numscons: the current architecture is based on launching a scons subprocess for every subpackage, sequentially. As stupid as this decisions may sound, there are relatively strong rationales for this. First, scipy is designed as a set of almost independent packages, and that’s true for the build process as well. Every subpackage declares its requirements (blas, lapack, etc…), and can be build independently of the others: if you launch the build process at the top of the source tree, the whole scipy is built; if you launch it in scipy/sparse/sparsetools, only sparsetools is built. This is a strong requirement: this is impossible to do with autotools, for example, unless each subpackage has its own configure (like gcc for example).
It is possible in theory with scons, but it is practically almost impossible to do so while staying compatible with distutils, because of build directory issues (the build directory cannot be a subdirectory of the source tree). When I started numscons, I could see only two solutions: launching independent scons builds for each subpackage, or having a whole source tree with the configuration at the top, but scons is too slow for the later solution (although it is certainly the best one from a design POV).

The second problem is that scons cannot be used as library. You cannot do something like “from scons import *; build(‘package1’); build(‘package2’)”. Which means the only simple solution to have independent builds for each package is to launch independent scons processes. Having to use subprocesses to launch scons is the single fundamental numscons issue.

1 Because scons is slow to start (it needs to check for tools, etc…), it means no-op builds are slow (it takes 20 seconds to complete a no-op full scipy built on a more than decent machine, which is why numscons has an option–package-list to list the packages to rescan, but that’s nothing more than an ugly hack).

2 Error handling is hard to do: if scons fails, it is hard to pass useful information back to the calling process

3 Since distutils still handle installation and tarballs generation, it needs to know about the source files. But since only scons knows about it, it is hard to pass this information back to distutils from scon. Currently, it only works because it knows the sources from the conventional setup.py files.

Another limitation I see with scons is the code quality: scons is a relatively old project, and focused a lot on backward compatibility, with a lot of cruft (scons still support python 1.5). There is still a lot of development happening, and is still supported; scons is used in several high profile projects (some vmware products are built with scons, Intel acknowledges its use internally, Google uses it – Steve Knight, the first author of scons works at Google on the Chrome project, and chromes sources have scons scripts). But there is a lot of tight coupling, and changing core implementations issues is extremely challenging. It is definitely much better than distutils (in the sense that in distutils, everything is wrong: the implementation, the documentation, the UI, the concepts). But fixing scons tight coupling is a huge task, to the point where rewriting from scratch some core parts may be easier (see here). There are also some design decisions in scons which are not great, like options handling (everything is passed through Environments instances, which is nothing more than a big global variable).

A potential solution would be to use waf insteaf of scons. Waf started as a scons fork, and dropped backward compatibility. Waf has several advantages:

  • it is much smaller and nicer implementation-wise than scons (core waf is ~ 4000 LOC, scons is ten times more). There are some things I can do today in waf I still have no idea how to do in scons, although I am much more familiar with the latter.
  • waf is much faster than scons (see here for some benchmarks)
  • it seems like waf can be used as a library

But:

  • waf is not stable (the API kept changing; the main waf developer said he would focus on stability from now on)
  • waf does not support Fortran – this one should be relatively easy to solve (I have some code working already for most fortran requirements in scipy)
  • I am still not satisfied with its tool handling – that’s a hard problem though, I have not seen a single build tool which handle this well. That’s not a fundamental issue to prevent the use of waf.
  • support on windows looks flaky

There may also be hope to see more cross-pollination between scons and waf – but that’s a long term goal, unless someone can work on it fulltime for at least several weeks IMHO. Porting numscons to waf should be relatively easy once fortran is handled – I think basic porting could be done in one day or two.

Ondrej also mentioned cmake: I think it would be very hard to build numpy with cmake, because it means you have to give up distutils entirely. How to make it work with easy_install ? How to generate tarballs, windows installers, etc… ? If anyone wants to try something else here is my suggestion: ignoring tarball/install issues, try to build numpy.core alone on windows, mac os X and linux, with either Atlas, Accelerate or nothing. Use scons scripts as informations for the checks to do (it is much more readable than distutils setup.py files). If you can do this, you will have solved most build details necessary to build the whole scipy. It will certainly give you a good overview of the difficulty of the task.