Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

This post will take you through the installation and configuration of an Infiniband card on a server running Red Hat Enterprise Linux 5.4.  These steps are applicable to any version of Red Hat 5, and will probably work with version 6 as well.  It has been surprisingly hard to find all of these steps in one document.

Required packages

openib-1.4.1-6.el5.noarch libibverbs-1.1.3-2.el5.x86_64 libnes-0.9.0-2.el5.x86_64 libibumad-1.3.3-1.el5.x86_64 opensm-libs-3.3.3-2.el5.x86_64 swig-1.3.29-2.el5.x86_64 ibutils-libs-1.2-11.1.el5.x86_64 ibutils-1.2-11.1.el5.x86_64 (provides ibdiagnet and others) opensm-3.3.3-2.el5.x86_64 libibmad-1.3.3-1.el5.x86_64 infiniband-diags-1.5.3-1.el5.x86_64 (provides handy tools like ibstat and ibstatus) libibverbs-utils-1.1.3-2.el5.x86_64 (provides handy tools ibv_devinfo and ibv_devices) libibverbs-devel-1.1.3-2.el5.x86_64

xCAT is the eXtreme Cloud Administration Toolkit from IBM.  It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC).  I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5).  Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster.  In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good.  There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru).  If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems.  We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach.  For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense.  The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools.  I’ll list some of the key differences between the IBM and Aspen systems:

OpenFOAM is a notoriously difficult piece of software to compile, install and run. OpenCFD (the authors of OpenFOAM) have chosen to require the use of recent versions of gcc that are not available on most stable enterprise-class systems (ie Red Hat Enterprise Linux). To make things worse, until recently, OpenCFD also bundled a large number of libraries and helper applications (like VTK and ParaView) with the OpenFOAM source instead of using libraries and tools that are already on the system. Fortunately, OpenCFD has now moved the extra tools to a separate tarball, and the wizards at Gentoo have managed to create an ebuild for OpenFOAM. This is why I run Gentoo on my desktop workstation!

CFD-ACE+ is a multiphysics and computational fluid dynamics (CFD) simulation tool that was originally developed by CFD Research Corp. and is now distributed by ESI Software. The only platforms officially supported by CFD-ACE+ are Red Hat Enterprise Linux, SUSE Linux and Windows. Fortunately, it seems that ACE+ runs on other Linux distributions with only a little hacking. I just installed and tested CFD-ACE+ successfully (albeit not very thoroughly) on an up-to-date Gentoo Linux system.  The process will require hacking some config files to build an external library from source. I started by downloading the files intended for Red Hat 6. Each package is a strange self-extracting executable file that unpacks itself when run. They also provide a C-shell script called install.com that automates the install process. The install script is very well behaved in that it doesn’t try to spew libraries and binaries throughout your filesystem, and you can run it as an ordinary user and install the file in your home directory.  The tricky part about the installation is that the unpacking software (which is bundled into the self-extracting executable) needs the library selinux.so.1 in order to run. That library isn’t normally installed on a Gentoo system unless you’ve specifically set up SELinux.  In order to avoid the potential problems of installing this library in my system paths,  I  created srcandincludedirectories in my home directory.  I downloaded source tarballs for libsepol and libselinux from the download page of the selinux project.  First, build libsepol usingmake.  Then, copy the libsepol.so.1 binary from the src directory to the lib directory you just created in your home directory. Copy the include/sepol directory with all of its contents to the include directory in your home directory.  When you are done, you should have:

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

I recently installed ATLAS on my new workstation and I’ve been wondering what kind of performance gain I would get compared to the reference BLAS. After some searching, I finally found a BLAS benchmarking tool called BLASbench, which is part of the LLCbench suite. This tool has a rather strange build process, so follow the instructions on the web page.  You have to create a file called sys.def which contains build instructions for your system.  Here’s the sys.def file I used for my Gentoo system:

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL).  This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

Prerequisites: boost-devel, openmpi-devel, valgrind-devel I installed PETSc and libMesh in my user directory, since I have a single-user workstation.

Installing PETSc for use with libMesh

First, ensure that OpenMPI is installed and the system paths have been configured correctly.  You will need to add the directory containing OpenMPI binaries to your $PATH, and the directory containing OpenMPI libraries to $LD_LIBRARY_PATH.  CentOS does not do this automatically!  Type mpicc on the command line to make sure the shell can find the binary.  If mpicc and mpif77 run, then you should not need to use the comnand-line flags to the configure script for PETSc. If you are going to use libMesh with PETSc, you need to enable C++ with a command-line option to the configure script.  Here is the process I used (it might be a good idea to set PETSC_ARCH and PETSC_DIR in your .bashrc so they are consistent for all builds).