xCAT is the eXtreme Cloud Administration Toolkit from IBM.  It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC).  I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5).  Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster.  In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good.  There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru).  If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems.  We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach.  For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense.  The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools.  I’ll list some of the key differences between the IBM and Aspen systems:

Updated 2017-08-01 to fix typo and use new demo URL. Also note that the Homebrew package manager has matured since this post was first published in 2012. Once you’ve configured Homebrew, installing a tool like wget is as simple as running brew install wget wget is an incredibly useful GNU tool on Linux. Unfortunately, it doesn’t come with OS X (as of Mountain Lion). OS X includes curl, which is a very handy tool but lacks at least one important feature of wget: the ability to use wildcards to get multiple files at the same time. For example, let’s say you want to download a subset of files from an FTP server. With wget, you could type:

OpenFOAM is a notoriously difficult piece of software to compile, install and run. OpenCFD (the authors of OpenFOAM) have chosen to require the use of recent versions of gcc that are not available on most stable enterprise-class systems (ie Red Hat Enterprise Linux). To make things worse, until recently, OpenCFD also bundled a large number of libraries and helper applications (like VTK and ParaView) with the OpenFOAM source instead of using libraries and tools that are already on the system. Fortunately, OpenCFD has now moved the extra tools to a separate tarball, and the wizards at Gentoo have managed to create an ebuild for OpenFOAM. This is why I run Gentoo on my desktop workstation!

The Problem

I ran into what was probably the strangest error I have ever encountered as a Linux admin.  I created a fresh CentoOS 6 virtual machine (VM) using VirtualBox on my OS X Mountain Lion system.  To my great surprise, I was only able to access Google and a handful of other web sites from Firefox in the VM.  I assumed that there was an issue with the networking of the VM, but after several frustrating hours of checking my setup I realized that the network in the VM was working fine.  I was able to ping any web site by name on the command line, so clearly DNS was working.  I could navigate to <<www.google.com&gt>; and perform searches, but clicking on most links resulted in the browser waiting forever.  I could navigate to any site by IP, although the sites didn’t always load correctly (such as centos.org).

CFD-ACE+ is a multiphysics and computational fluid dynamics (CFD) simulation tool that was originally developed by CFD Research Corp. and is now distributed by ESI Software. The only platforms officially supported by CFD-ACE+ are Red Hat Enterprise Linux, SUSE Linux and Windows. Fortunately, it seems that ACE+ runs on other Linux distributions with only a little hacking. I just installed and tested CFD-ACE+ successfully (albeit not very thoroughly) on an up-to-date Gentoo Linux system.  The process will require hacking some config files to build an external library from source. I started by downloading the files intended for Red Hat 6. Each package is a strange self-extracting executable file that unpacks itself when run. They also provide a C-shell script called install.com that automates the install process. The install script is very well behaved in that it doesn’t try to spew libraries and binaries throughout your filesystem, and you can run it as an ordinary user and install the file in your home directory.  The tricky part about the installation is that the unpacking software (which is bundled into the self-extracting executable) needs the library selinux.so.1 in order to run. That library isn’t normally installed on a Gentoo system unless you’ve specifically set up SELinux.  In order to avoid the potential problems of installing this library in my system paths,  I  created srcandincludedirectories in my home directory.  I downloaded source tarballs for libsepol and libselinux from the download page of the selinux project.  First, build libsepol usingmake.  Then, copy the libsepol.so.1 binary from the src directory to the lib directory you just created in your home directory. Copy the include/sepol directory with all of its contents to the include directory in your home directory.  When you are done, you should have:

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

Newer enterprise-class hardware is starting to use UEFI instead of BIOS. UEFI is a big improvement, but there isn’t much documentation on the subject. Today I had to configure the UEFI for a new node in our cluster. I wanted to configure it exactly like an existing node that works. It was challenging just to find out how to display the configuration of the old node without rebooting the node and paging through dozens of configuration screens with the keyboard. Then I discovered IBM’s Advanced Settings Utility software, which  fortunately was already installed on our cluster.  It appears to be free for download, but with a restricted license.  To display the complete UEFI configuration for a server, ssh to the server and run

I recently installed ATLAS on my new workstation and I’ve been wondering what kind of performance gain I would get compared to the reference BLAS. After some searching, I finally found a BLAS benchmarking tool called BLASbench, which is part of the LLCbench suite. This tool has a rather strange build process, so follow the instructions on the web page.  You have to create a file called sys.def which contains build instructions for your system.  Here’s the sys.def file I used for my Gentoo system:

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL).  This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines: