Most of the time, RPM (especially in conjunction with yum) is a decent package management solution. However, I can think of two common circumstances when you don’t want to let RPM install a package:

You don’t have root permissions on a system such as a shared cluster You are an administrator on a shared cluster and you can’t risk having a package over-write system-critical files

One approach is to extract the files from the RPM package and install them manually. I used this method to install FDTD Solutions from Lumerical on the STOKES Linux cluster. The application is distributed as an RPM package. I never install a third-party RPM as root, because a badly constructed package might over-write a system-critical file that some user is depending on. I downloaded the appropriate TAR file from Lumerical, uncompressed it, and looked at the contents. There is a simple script called install.sh which checks to make sure the user is root and then tries to install a hardware key driver and an RPM that contains the FDTD software. Here are my recommendations for installing Lumerical products on a cluster:

I found an interesting quirk when trying to build an OpenMPI application on a visualization node with a “stock” version of Red Hat Enterprise Linux 5.8.  I used mpicc to compile the application and got the following error:

$ mpicc hello_world_mpi.c -o hello_world
/usr/bin/ld: cannot find -lnuma

This was rather surprising, since this node mounts a directory via NFS that contains OpenMPI and Intel Composer 2013, and these applications are known to work on other nodes. To find the source of the problem, I used the showme option to see the command that is actually run by mpicc:

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. The hickle developers have made a good start, and they have a long way to go before hickle will be useful to a wider audience. Right now, hickle can only store NumPy ndarrays and Python list objects. If you only need to store lists and arrays, you might as well use HDF5 bindings for Python such as PyTables or h5py. The power of the pickle module is that you can immediately serialize almost any Python object of arbitrary complexity, store it on disk, and retrieve it. hickle will only achieve its full potential once it replicates this functionality, and I’m not sure how difficult this will be. Ideally, you might be able to derive a class from Pickler that uses Picker’s methods to serialize an object, and then add your own method to write the serialized object to an HDF5 file. In a future post, I’ll describe some of the practical problems with using pickle files to store data, and try to organize some thoughts about how they might be solved.

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

This article from Admin Magazine has a handy list of profiling tools.

3D printers have been getting more attention in the mainstream media in the past few months. 3D printers were mentioned in a story about tech trends on National Public Radio today, and my father showed me a newsletter from a mutual fund company with a short article about 3D printing. You know that something is hitting the mainstream when retired people learn about it. Of course, 3D printing is old news to those of us in the Maker community. We know that just because consumers can buy 3D printers for their homes doesn’t mean that they should. Today’s 3D printers aren’t exactly consumer products, and someone who believes the hype and spends a couple thousand dollars without doing a lot of research is probably going to be disappointed. The technology is not at the point where you can print objects that look as good or work as well as similar items you can buy at the store. Likewise, designing your own custom objects sounds good until you discover the price (and learning curve) of 3D design software. Focusing on 3D printers risks missing a broader trend in the democratization of manufacturing. From the late 1970’s to the early 1990’s, computing power became affordable to the masses. From the late 1990’s to the present, the Internet has given the common man access to an amount of information that would have been inconceivable two decades ago. These trends democratized the means of production for media, music, video, and information technology. Over the past few years, the barriers to entry for manufacturing are becoming lower at a rapid pace. Now you can print goods in your home, or send your design out for rapid prototyping, at an affordable price. For example, I recently spent $25 to buy a mount for my smartphone through Kickstarter. The business that is developing this item consists of one man with a 3D CAD program. The prototypes were fabricated using the Ponoko 3D printing service and the production tooling will only require an up-front investment of $18,000. Karl Marx developed communism because he perceived that the common man was oppressed by the few wealthy people who owned the means of production. Today, the common man doesn’t need to own the means of production because he can make use of them at an affordable price. Our concept of work is being transformed in a post-industrial revolution, and no one describes it better than Seth Godin. If you haven’t read any of his books, Linchpin is a good place to start. We’re living in an exciting time, and I’m looking forward to seeing what 2013 will bring!

This post will take you through the installation and configuration of an Infiniband card on a server running Red Hat Enterprise Linux 5.4.  These steps are applicable to any version of Red Hat 5, and will probably work with version 6 as well.  It has been surprisingly hard to find all of these steps in one document.

Required packages

openib-1.4.1-6.el5.noarch libibverbs-1.1.3-2.el5.x86_64 libnes-0.9.0-2.el5.x86_64 libibumad-1.3.3-1.el5.x86_64 opensm-libs-3.3.3-2.el5.x86_64 swig-1.3.29-2.el5.x86_64 ibutils-libs-1.2-11.1.el5.x86_64 ibutils-1.2-11.1.el5.x86_64 (provides ibdiagnet and others) opensm-3.3.3-2.el5.x86_64 libibmad-1.3.3-1.el5.x86_64 infiniband-diags-1.5.3-1.el5.x86_64 (provides handy tools like ibstat and ibstatus) libibverbs-utils-1.1.3-2.el5.x86_64 (provides handy tools ibv_devinfo and ibv_devices) libibverbs-devel-1.1.3-2.el5.x86_64

Glossary of Infiniband Terminology

GID: Global Identifier GUID: Global Unique Identifiers (also known as Direct Address) HCA: Host Channel Adapter LID: Local Identifier TCA: Target Channel Adapter SM: Subnet Manager

Infiniband: The Host Perspective

Every host on an Infiniband fabric has three identifiers: GUID, GID, and LID. A GUID is similar in concept to a MAC address because it consists of a 24-bit manufacturer’s prefix and a 40-bit device identifier (64 bits total). The Global Identifier (GID) is a 128-bit identifier similar to an IPv6 address (technically, a GID is a valid IPv6 identifier with restrictions). The GID consists of the 64-bit GUID plus an additional 64-bit EUI-64 identifier, for a total of 128 bits. The GID is used for routing between subnets. The default GID prefix is 0xfe80::0. Finally, there is the local identifier (LID), which is assigned by the subnet manager. The LID is a 16-bit identifier that is unique within a subnet. Hosts have an LID between 0 and 48,000, usually expressed in hexadecimal notation (such as 0xb1). Routing within a subnet is managed by LID. The GUID, GID, and LID for a Linux server are stored in text files. The exact path to this text file will depend upon which Infiniband driver is used on the system. For a system with the MLX4 driver (such as RedHat/CentOS 5.x), the commands are:

xCAT is the eXtreme Cloud Administration Toolkit from IBM.  It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC).  I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5).  Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster.  In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good.  There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru).  If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems.  We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach.  For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense.  The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools.  I’ll list some of the key differences between the IBM and Aspen systems: