Progress for numpy on windows 64 bits

The numpy 1.3.0 installer for windows 64 does not work very well. On some configurations, it does not even import without crashing. The crashes are mostly likely due to some bad interactions between the 64 bits mingw compilers and python (built with Visual Studio 2008). Although I know it is working, I had no interest in building numpy with MS compiler, because gfortran does not work with VS 2008. There are some incompatibilities because the fortran runtime from gfortran is incompatible with the VS 2008 C runtime (I get some scary linking errors).

So the situation is either building numpy with MS compiler, but with no hope of getting scipy afterwards, or building a numpy with crashes which are very difficult to track down. Today, I realized that I may go somewhere if somehow, I could use gfortran without using the gfortran runtime (e.g. libgfortran.a). I first tried calling a gfortran-built blas/lapack from a C program built with VS 2008, and after a couple of hours, I managed to get it working. Building numpy itself with full blas/lapack was a no-brainer then.

Now, there is the problem of scipy. Since scipy has some fortran code, which itself depends on the gfortran runtime when built with gfortran, I am trying to ‘fake’ a minimal gfortran runtime built with the C compiler. Since this mini runtime is built with the MS compiler and with the same  C runtime as used by python, it should work if the runtime is ABI compatible with the gfortran one. As gfortran is open source, this may not be intractable :)

With this technique, I could go relatively far in a short time. Among the packages which build and pass most of the test suite:
 - scipy.fftpack
 - scipy.lapack
 - some scipy.sparse

Some packages like cluster or spatial are not ANSI C compatible, so they fail to build. This should not be too hard to fix. The main problem is scipy.special: the C code is horrible, and there needs many hacks to build the C code. The Fortran code needs quite a few functions from the fortran runtime, so this needs some work. But ~ 300 unit tests of scipy pass, so this is encouraging.

About these ads

5 responses to “Progress for numpy on windows 64 bits

  1. Python Beginner

    Could you please explain in steps (for a beginner) how did you build only NumPy with Visual Studio 2008 on 64bit Windows?

    • cournape

      Sorry for the late reply. Building numpy is not difficult, you just need to make sure you have a compiler targetting 64 bits. The expression version of VS 2008 does not include 64 bits targetting compiler, but if you install the MS SDK 6.0a or above, it includes compilers targetting amd64. Make sure to install amd64 and not ia64 (IA64 stands for Itanium, and is a totally different architecture).

      Then, once the SDK installed, go into the start menu, SDK, and find a x64 retail command line. From there, go into numpy sources, and build with python setup.py install – it should work.

  2. Les

    will this latest SDK be compatible with the python executable? i have seen other web pages say that numpy (like all extensions???) must be built with the same compiler as the executable. i am using python2.6 for 64-bit. here is what my python says (running from 32-bit cygwin):

    $ python
    Python 2.6.2 (r262:71605, Apr 14 2009, 22:46:50) [MSC v.1500 64 bit (AMD64)] on win32

    also, if what you say works, why isn’t this distributed, rather than the numpy-1.3.0.win-amd64-py2.6.msi file which is broken. for one thing, it puts the modules into the Program Files (x86) folder instead of the normal Program Files. when those files/folders are moved over by hand, numpy crashes while loading lapack_lite.

    since i just got a new Vista 64-bit machine and use Windows (more than i would like to), i am game to do a proper build, if the instructions make sense and are consistent with other web lore. so if your scheme works, I’d like to try it and then have a scheme for getting it packaged properly for distribution, so others don’t have to go thru this.

  3. cournape

    The exact restriction for the compiler concerns the runtime. The Python C API was not designed to be robust against multiple C runtimes, so if your compiler uses a different C runtime than the one used by python, you will have problems. This can be checked with the dependency walker software.

    The program files (x86) issue is very weird: where did you install python ?

    As to why not building with VS 2008 – as long as the issue is not fixed for both numpy and scipy, there is no point in publishing new builds, because later ones may be incompatible if something changes again in the toolchain. Once the choice on which toolchain will be used for 64 bits binaries will be made, new binaries will be published. Numpy is easy enough that if you need numpy, you may just do it by yourself.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

June 2009
M T W T F S S
« May   Jan »
1234567
891011121314
15161718192021
22232425262728
2930  

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 10 other followers

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: