Updated: building 64-bit Numpy with Intel compilers (icc)

I had to re-build Numpy because our cluster was upgraded and the Intel compilers and libraries were moved to a different directory.  This turned out to be a half-day affair of trial-and-error.  I learned a few important things, which I will try to list here:

*Delete the numpy-1.0.4/build directory after every build attempt.  Doing “python setup.py clean” isnoteffective.  I kept getting errors about undefined symbols when I tried to “import numpy” on the Python command line.  It was looking for symbols in the old locations, even though I had just rebuilt the code using the new library locations.  It turned out that I needed to delete the build directory in order to force a complete bottom-up rebuild.*The use of “setup.py” from distutils is not well documented online.  The best thing to do is run “python setup.py –help-commands” to get a list of available commands.  Then run “python setup.py <cmd> –help” to get help for that specific command.  You can string commands together on the command line, as I will show in the example below.When you test the new numpy, make sure you arenotin the numpy-1.0.4 directory!  If you are in the numpy source directory, when you import numpy, you will get the message “Running from numpy source directory.” and you will not be able to load any symbols from numpy. On 64-bit architectures, you need to compile position-independent library code.  For some reason, distutils does not do this automatically, and the compilation will fail with an error similar to the following:

```text
relocation R_X86_64_PC32 against `_various_library_symbols'
can not be used when making a shared object; recompile with -fPIC
```

Edit the file numpy-1.0.4/numpy/distutils/intelccompiler.py.  Change the line **cc\_exe="icc"**to**cc\_exe="icc -fPIC"**(line 11 in my version of numpy).* Make sure your LD\_LIBRARY\_PATH is pointing at the right location, or the compiled code won't be able to find shared libraries.

Here is the command line I used to build numpy: python setup.py config –library-dirs=/apps/intel/mkl/10.0.1.014/lib/em64t/ –library-dirs=/apps/intel/cce/10.1.008/lib/ –compiler=intel –fcompiler=intel build –verbose install –home=~ > install_output.txt Explanation: “config” is a command that accepts arguments –library-dirs to set the path where icc searches for shared libraries, –compiler to set the C compiler, and –fcompiler to set the Fortran compiler.  Technically, I think you can omit the “build” command.  The “install” command installs numpy, and the –home=~ installs it in the user directory (I don’t have admin privileges on this machine).  Finally, > install_output.txt redirects the lengthy output to a text file for debugging purposes. I hope this saves somebody some time!

Comments

Comment #1 by Update 2: building 64-bit Numpy with Intel compilers and MKL | as through a mirror dimly

Update 2: building 64-bit Numpy with Intel compilers and MKL | as through a mirror dimly - Dec 2, 2008

[…] a previous post I described how I built Numpy with Intel compilers and the Math Kernel Library on a 64-bit cluster.  Today I upgraded to Numpy-1.2.1 and I made a few improvements to my install process.  Please […]

• Linux
• Python