A previous post documented that a Linux server running a pre-2.6.24 kernel can fail to allocate large chunks of memory after its memory has been fragmented by a “thrashing” incident. In this post, I will point out some ways to prevent this problem.

Use a Newer Kernel

We have some servers running RHEL 5.9 with the kernel updated to 2.6.34.14. After a thrashing incident, these servers do not experience the same problem with allocating large blocks of memory. I think the fix is documented in the release notes for kernel 2.6.24. Section 2.4 talks about “anti-fragmentation patches” and includes a link to this article about Linux memory management, which links to this thorough documentation of the anti-fragmentation patches.(BTW, here is the full list of 2.6 kernel changelogs) My plan is to deploy RHEL 5.9 with the updated kernel to all the compute nodes in our cluster. However, this still doesn’t solve the problem of a user who requests some portion of the RAM on a node and then proceeds to consume more memory than requested. This is unfair to another user whose job is running on the same node.

I have data from the STOKES High Performance Compute Cluster which definitively shows that kernel versions prior to 2.6.24 can suffer significant performance degradation due to memory fragmentation. I noticed the problem on servers running Red Hat Enterprise Linux (RHEL) 5.4 with kernel version 2.6.18-164.el5. This post will document my findings. The graphs were taken from our Ganglia monitoring system.

This node has 24GB of RAM. As long as processes do not request more than 23GB of RAM, the node operates normally. Processes can use 23GB of RAM all day long:

It’s surprisingly difficult to find out whether it is possible to include an image from an Encapsulated PostScript (EPS) file into an Asymptote vector graphics document. It turns out that it is easy, but difficult to find in the Asymptote docs (I finally found the answer, via Google, in the FAQ). It turns out that you use the label function to insert an image into an Asymptote document. Here is a snippet of code that I used to assemble a multi-part figure from several EPS documents:

This post shows how I set up MATLAB to perform parallel calculations on a multi-core server which is part of an existing cluster that is managed using Torque and Moab. I did this without making any changes to the cluster submission system. The installation documentation for the MATLAB Parallel Computing Toolbox and Distributed Computing Server is poor. The instructions make a lot of assumptions that don’t fit into an existing “production” cluster environment. For example, they assume that a GUI is available to configure the cluster (this step is not required for my method!) Further, the instructions give the impression that MATLAB components must be “installed” on both the head node and compute nodes by an administrator. THIS IS FALSE! Ordinary users can run concurrent (multicore) MATLAB jobs without administrative privileges!

GROMACS produces graphical output in the form of .xvg files. These are designed to be viewed with a classic UNIX/Linux plotting program called Grace. If you happen to be using Linux and you have Grace installed, it is very easy to plot the data with the command

xmgrace my_file.xvg

If you aren’t using Linux, plotting .xvg files is quite a bit more difficult. Gnuplot is the only free plotting program that I have found that can handle .xvg files. Gnuplot is available for Windows (there is a direct download link near the top of the download page), but I will caution you that Gnuplot is not easy to learn. It has a command-line interface, and there are no point-and-click options to do basic operations such as labeling the axes of the plot. If you are using Gnuplot, you can plot the contents of an .xvg file with the command:

My current employer, the STOKES Advanced Research Computing Center (STOKES ARCC), is hiring a postdoctoral research associate to conduct research in high performance computing with an emphasis on next-generation networking technologies. The ARCC has internal funding that will be used to upgrade our research network to the Internet2 Innovation Platform standard. We are also seeking external funding to extend the research network across the UCF campus. We are looking for a candidate with an interest in topics such as defining a “Science DMZ,” Internet2, GENI, perfSONAR, software-defined networks, etc. Please use the link above to apply for the position. Feel free to contact me if you have questions-my contact information is on the about page.

I have published up-to-date versions of two classic GROMACS tutorials on GitHub. The Getting Started section of the GROMACS online documentation contains some helpful tutorials.  Unfortunately, these tutorials have not been updated in a while. They also don’t explain how to set up an efficient workflow to run large molecular dynamics simulations on a shared cluster using a resource manager such at Torque. I have created a set of files that implement the speptide tutorial from the GROMACS documentation.You can use my files and follow along with the explanations in the GROMACS manual. The speptide directory has two subdirectories: one for MDP files (which control the simulations) and one for simulation files. I have found this to be an efficient setup because many MD experiments involve simulating multiple variants of a molecule (such as mutants of a protein) and comparing the results. All variants should be run with the same parameters, so the same MDP files are used for each simulation. Keeping the MDP files in their own directory and creating symbolic links from the run directory to the MDP files ensures that all simulations are run with identical parameters. In the run directory, there is a Bash script called setup_GROMACS_job.sh. At the top of the script, the user sets variables to set simulation parameters such as the box size. The script can be run with the command

Most of the time, RPM (especially in conjunction with yum) is a decent package management solution. However, I can think of two common circumstances when you don’t want to let RPM install a package:

You don’t have root permissions on a system such as a shared cluster You are an administrator on a shared cluster and you can’t risk having a package over-write system-critical files

One approach is to extract the files from the RPM package and install them manually. I used this method to install FDTD Solutions from Lumerical on the STOKES Linux cluster. The application is distributed as an RPM package. I never install a third-party RPM as root, because a badly constructed package might over-write a system-critical file that some user is depending on. I downloaded the appropriate TAR file from Lumerical, uncompressed it, and looked at the contents. There is a simple script called install.sh which checks to make sure the user is root and then tries to install a hardware key driver and an RPM that contains the FDTD software. Here are my recommendations for installing Lumerical products on a cluster:

I found an interesting quirk when trying to build an OpenMPI application on a visualization node with a “stock” version of Red Hat Enterprise Linux 5.8.  I used mpicc to compile the application and got the following error:

$ mpicc hello_world_mpi.c -o hello_world
/usr/bin/ld: cannot find -lnuma

This was rather surprising, since this node mounts a directory via NFS that contains OpenMPI and Intel Composer 2013, and these applications are known to work on other nodes. To find the source of the problem, I used the showme option to see the command that is actually run by mpicc: