Scientific Computing on shocksolution.com

Preventing "soft" failures due to memory fragmentation in Linux

Wed, 20 Nov 2013 13:27:41 +0000

A previous post documented that a Linux server running a pre-2.6.24 kernel can fail to allocate large chunks of memory after its memory has been fragmented by a “thrashing” incident. In this post, I will point out some ways to prevent this problem.

Use a Newer Kernel

We have some servers running RHEL 5.9 with the kernel updated to 2.6.34.14. After a thrashing incident, these servers do not experience the same problem with allocating large blocks of memory. I think the fix is documented in the release notes for kernel 2.6.24. Section 2.4 talks about “anti-fragmentation patches” and includes a link to this article about Linux memory management, which links to this thorough documentation of the anti-fragmentation patches.(BTW, here is the full list of 2.6 kernel changelogs) My plan is to deploy RHEL 5.9 with the updated kernel to all the compute nodes in our cluster. However, this still doesn’t solve the problem of a user who requests some portion of the RAM on a node and then proceeds to consume more memory than requested. This is unfair to another user whose job is running on the same node.

Memory fragmentation degrades performance in Linux kernels < 2.6.24 (RHEL 5.4)

Fri, 15 Nov 2013 21:30:38 +0000

I have data from the STOKES High Performance Compute Cluster which definitively shows that kernel versions prior to 2.6.24 can suffer significant performance degradation due to memory fragmentation. I noticed the problem on servers running Red Hat Enterprise Linux (RHEL) 5.4 with kernel version 2.6.18-164.el5. This post will document my findings. The graphs were taken from our Ganglia monitoring system.

This node has 24GB of RAM. As long as processes do not request more than 23GB of RAM, the node operates normally. Processes can use 23GB of RAM all day long:

Insert an EPS file into an Asymptote Vector Graphics document

Wed, 23 Oct 2013 13:08:06 +0000

It’s surprisingly difficult to find out whether it is possible to include an image from an Encapsulated PostScript (EPS) file into an Asymptote vector graphics document. It turns out that it is easy, but difficult to find in the Asymptote docs (I finally found the answer, via Google, in the FAQ). It turns out that you use the label function to insert an image into an Asymptote document. Here is a snippet of code that I used to assemble a multi-part figure from several EPS documents:

Using the MATLAB Distributed/Parallel Computing Toolbox without modifying your cluster

Wed, 02 Oct 2013 19:44:53 +0000

This post shows how I set up MATLAB to perform parallel calculations on a multi-core server which is part of an existing cluster that is managed using Torque and Moab. I did this without making any changes to the cluster submission system. The installation documentation for the MATLAB Parallel Computing Toolbox and Distributed Computing Server is poor. The instructions make a lot of assumptions that don’t fit into an existing “production” cluster environment. For example, they assume that a GUI is available to configure the cluster (this step is not required for my method!) Further, the instructions give the impression that MATLAB components must be “installed” on both the head node and compute nodes by an administrator. THIS IS FALSE! Ordinary users can run concurrent (multicore) MATLAB jobs without administrative privileges!

How to plot .xvg files from GROMACS on Windows

Tue, 12 Mar 2013 18:00:51 +0000

GROMACS produces graphical output in the form of .xvg files. These are designed to be viewed with a classic UNIX/Linux plotting program called Grace. If you happen to be using Linux and you have Grace installed, it is very easy to plot the data with the command

xmgrace my_file.xvg

If you aren’t using Linux, plotting .xvg files is quite a bit more difficult. Gnuplot is the only free plotting program that I have found that can handle .xvg files. Gnuplot is available for Windows (there is a direct download link near the top of the download page), but I will caution you that Gnuplot is not easy to learn. It has a command-line interface, and there are no point-and-click options to do basic operations such as labeling the axes of the plot. If you are using Gnuplot, you can plot the contents of an .xvg file with the command:

Opportunity for postdoctoral research associate in high performance computing

Mon, 04 Mar 2013 22:51:12 +0000

My current employer, the STOKES Advanced Research Computing Center (STOKES ARCC), is hiring a postdoctoral research associate to conduct research in high performance computing with an emphasis on next-generation networking technologies. The ARCC has internal funding that will be used to upgrade our research network to the Internet2 Innovation Platform standard. We are also seeking external funding to extend the research network across the UCF campus. We are looking for a candidate with an interest in topics such as defining a “Science DMZ,” Internet2, GENI, perfSONAR, software-defined networks, etc. Please use the link above to apply for the position. Feel free to contact me if you have questions-my contact information is on the about page.

Updated GROMACS tutorials

Thu, 28 Feb 2013 21:40:54 +0000

I have published up-to-date versions of two classic GROMACS tutorials on GitHub. The Getting Started section of the GROMACS online documentation contains some helpful tutorials. Unfortunately, these tutorials have not been updated in a while. They also don’t explain how to set up an efficient workflow to run large molecular dynamics simulations on a shared cluster using a resource manager such at Torque. I have created a set of files that implement the speptide tutorial from the GROMACS documentation.You can use my files and follow along with the explanations in the GROMACS manual. The speptide directory has two subdirectories: one for MDP files (which control the simulations) and one for simulation files. I have found this to be an efficient setup because many MD experiments involve simulating multiple variants of a molecule (such as mutants of a protein) and comparing the results. All variants should be run with the same parameters, so the same MDP files are used for each simulation. Keeping the MDP files in their own directory and creating symbolic links from the run directory to the MDP files ensures that all simulations are run with identical parameters. In the run directory, there is a Bash script called setup_GROMACS_job.sh. At the top of the script, the user sets variables to set simulation parameters such as the box size. The script can be run with the command

Installing Lumerical FDTD on a linux cluster

Mon, 25 Feb 2013 20:25:43 +0000

Most of the time, RPM (especially in conjunction with yum) is a decent package management solution. However, I can think of two common circumstances when you don’t want to let RPM install a package:

You don’t have root permissions on a system such as a shared cluster You are an administrator on a shared cluster and you can’t risk having a package over-write system-critical files

One approach is to extract the files from the RPM package and install them manually. I used this method to install FDTD Solutions from Lumerical on the STOKES Linux cluster. The application is distributed as an RPM package. I never install a third-party RPM as root, because a badly constructed package might over-write a system-critical file that some user is depending on. I downloaded the appropriate TAR file from Lumerical, uncompressed it, and looked at the contents. There is a simple script called install.sh which checks to make sure the user is root and then tries to install a hardware key driver and an RPM that contains the FDTD software. Here are my recommendations for installing Lumerical products on a cluster:

OpenMPI, Intel Compilers and RedHat 5: cannot find -lnuma

Wed, 20 Feb 2013 23:21:42 +0000

I found an interesting quirk when trying to build an OpenMPI application on a visualization node with a “stock” version of Red Hat Enterprise Linux 5.8. I used mpicc to compile the application and got the following error:

$ mpicc hello_world_mpi.c -o hello_world
/usr/bin/ld: cannot find -lnuma

This was rather surprising, since this node mounts a directory via NFS that contains OpenMPI and Intel Composer 2013, and these applications are known to work on other nodes. To find the source of the problem, I used the showme option to see the command that is actually run by mpicc:

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Wed, 16 Jan 2013 22:29:19 +0000

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

Installing and configuring Infiniband on a Red Hat system

Wed, 12 Dec 2012 23:15:54 +0000

This post will take you through the installation and configuration of an Infiniband card on a server running Red Hat Enterprise Linux 5.4. These steps are applicable to any version of Red Hat 5, and will probably work with version 6 as well. It has been surprisingly hard to find all of these steps in one document.

Required packages

openib-1.4.1-6.el5.noarch libibverbs-1.1.3-2.el5.x86_64 libnes-0.9.0-2.el5.x86_64 libibumad-1.3.3-1.el5.x86_64 opensm-libs-3.3.3-2.el5.x86_64 swig-1.3.29-2.el5.x86_64 ibutils-libs-1.2-11.1.el5.x86_64 ibutils-1.2-11.1.el5.x86_64 (provides ibdiagnet and others) opensm-3.3.3-2.el5.x86_64 libibmad-1.3.3-1.el5.x86_64 infiniband-diags-1.5.3-1.el5.x86_64 (provides handy tools like ibstat and ibstatus) libibverbs-utils-1.1.3-2.el5.x86_64 (provides handy tools ibv_devinfo and ibv_devices) libibverbs-devel-1.1.3-2.el5.x86_64

Managing an HPC cluster or cloud infrastructure: alternatives to xCAT

Sun, 04 Nov 2012 19:01:15 +0000

xCAT is the eXtreme Cloud Administration Toolkit from IBM. It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC). I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5). Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster. In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good. There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru). If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems. We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach. For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense. The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools. I’ll list some of the key differences between the IBM and Aspen systems:

An Improved Ebuild for OpenFOAM on Gentoo Linux

Mon, 22 Oct 2012 09:38:00 +0000

OpenFOAM is a notoriously difficult piece of software to compile, install and run. OpenCFD (the authors of OpenFOAM) have chosen to require the use of recent versions of gcc that are not available on most stable enterprise-class systems (ie Red Hat Enterprise Linux). To make things worse, until recently, OpenCFD also bundled a large number of libraries and helper applications (like VTK and ParaView) with the OpenFOAM source instead of using libraries and tools that are already on the system. Fortunately, OpenCFD has now moved the extra tools to a separate tarball, and the wizards at Gentoo have managed to create an ebuild for OpenFOAM. This is why I run Gentoo on my desktop workstation!

Hack of the Day: running CFD-ACE+ on Gentoo Linux

Tue, 24 Jul 2012 18:50:49 +0000

CFD-ACE+ is a multiphysics and computational fluid dynamics (CFD) simulation tool that was originally developed by CFD Research Corp. and is now distributed by ESI Software. The only platforms officially supported by CFD-ACE+ are Red Hat Enterprise Linux, SUSE Linux and Windows. Fortunately, it seems that ACE+ runs on other Linux distributions with only a little hacking. I just installed and tested CFD-ACE+ successfully (albeit not very thoroughly) on an up-to-date Gentoo Linux system. The process will require hacking some config files to build an external library from source. I started by downloading the files intended for Red Hat 6. Each package is a strange self-extracting executable file that unpacks itself when run. They also provide a C-shell script called install.com that automates the install process. The install script is very well behaved in that it doesn’t try to spew libraries and binaries throughout your filesystem, and you can run it as an ordinary user and install the file in your home directory. The tricky part about the installation is that the unpacking software (which is bundled into the self-extracting executable) needs the library selinux.so.1 in order to run. That library isn’t normally installed on a Gentoo system unless you’ve specifically set up SELinux. In order to avoid the potential problems of installing this library in my system paths, I created srcandincludedirectories in my home directory. I downloaded source tarballs for libsepol and libselinux from the download page of the selinux project. First, build libsepol usingmake. Then, copy the libsepol.so.1 binary from the src directory to the lib directory you just created in your home directory. Copy the include/sepol directory with all of its contents to the include directory in your home directory. When you are done, you should have:

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

Mon, 02 Jul 2012 17:14:54 +0000

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

Benchmarking BLAS Part 1: Building BLASbench with gcc

Fri, 25 May 2012 22:37:22 +0000

I recently installed ATLAS on my new workstation and I’ve been wondering what kind of performance gain I would get compared to the reference BLAS. After some searching, I finally found a BLAS benchmarking tool called BLASbench, which is part of the LLCbench suite. This tool has a rather strange build process, so follow the instructions on the web page. You have to create a file called sys.def which contains build instructions for your system. Here’s the sys.def file I used for my Gentoo system:

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

Tue, 24 Apr 2012 16:40:40 +0000

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL). This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

Installing PETSc and libMesh on CentOS 5

Fri, 20 Jan 2012 20:06:18 +0000

Prerequisites: boost-devel, openmpi-devel, valgrind-devel I installed PETSc and libMesh in my user directory, since I have a single-user workstation.

Installing PETSc for use with libMesh

First, ensure that OpenMPI is installed and the system paths have been configured correctly. You will need to add the directory containing OpenMPI binaries to your $PATH, and the directory containing OpenMPI libraries to $LD_LIBRARY_PATH. CentOS does not do this automatically! Type mpicc on the command line to make sure the shell can find the binary. If mpicc and mpif77 run, then you should not need to use the comnand-line flags to the configure script for PETSc. If you are going to use libMesh with PETSc, you need to enable C++ with a command-line option to the configure script. Here is the process I used (it might be a good idea to set PETSC_ARCH and PETSC_DIR in your .bashrc so they are consistent for all builds).

Paraview 3.12 on 64-bit CentOS 5.7

Thu, 19 Jan 2012 19:23:15 +0000

I finally got around to installing Paraview on my CentOS 5 box. There are no official RPMS for CentOS 5, so I was expecting a painful build process. To my great surprise, I was able to download the Paraview binary for Linux (64-bit) from Kitware, unpack the archive, and run it! As root, I placed the entire ParaView-3.12.0 directory under /opt/Paraview. You could also keep it in your home directory if you are running a single-user system. I added /opt/Paraview/ParaView-3.12.0/bin to the $PATH in the .bashrc file in my home directory. Now I can type paraview at the command line and it runs! What if you want Paraview to appear in the Applications menu in your desktop environment? Most modern desktops (I use XFCE4) construct the Applications menu “on the fly” based upon the files in a standard directory (/usr/share/applications on CentOS). The Free Desktop Project has created a standard for desktop entry files. You may also find this summary of the standard to be helpful. To add Paraview to the menu, you simply need to create a new file in the standard location. If you installed Paraview in your user directory, you may want to place the desktop file in $HOME/.local/applications. Here are the contents of a file I called paraview.desktop:

Installing Sage 4.7 on CentOS 5

Mon, 07 Nov 2011 23:07:59 +0000

I recently upgraded my desktop workstation from an old 32-bit version of Gentoo to 64-bit CentOS 5. I downloaded and installed the latest version of Sage, and the process went smoothly. If you find this post helpful, please check out the Sage Beginner’s Guide at Packt Publishing. Since I don’t use Sage every day, I actually refer to my own book on a regular basis!

Download

Since CentOS is designed to be binary compatible with Red Hat Enterprise Linux, the correct binary to download is sage-4.7.2-linux-64bit-red_hat_enterprise_linux_server_release_5.6_tikanga-x86_64-Linux.tar.gz

Python string format examples

Thu, 03 Nov 2011 22:38:10 +0000

The format method for Python strings (introduced in 2.6) is very flexible and powerful. It’s also easy to use, but the documentation is not very clear. It all makes sense with a few examples. I’ll add more as I have time:

Formatting Numbers in Python Strings

Formatting a floating-point number

[code language=“python”] “{0:.4f}".format(0.1234567890) “{0:.4f}".format(10.1234567890) [/code] The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field” 0 indicates the first argument to method format : indicates the start of the format specifier .4 indicates four decimal places f indicates a floating-point number

How to build ScipPy with Python 2.7.2 on CentOS5

Thu, 25 Aug 2011 00:17:57 +0000

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path. This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib). EDIT 2: you should use the –prefix=/home/yourusername instead of –user. The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages. The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up! I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5. I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system. Here is how I did it. First, with superuser privileges, use yum to install the following packages. You may need to set up epel as an alternate repository:

Removing an axis or both axes from a matplotlib plot

Wed, 17 Aug 2011 22:40:14 +0000

Sometimes, the frame around a matplotlib plot can detract from the information you are trying to convey. How do you remove the frame, ticks, or axes from a matplotlib plot?

matplotlib plot without a y axis

Some books you may find useful when working with matplotlib:

The full example is available on github.

First, we construct a figure and an axes object:

fig1 = plt.figure(facecolor='white')
ax1 = plt.axes(frameon=False)

The Axes object is a container that holds the axes, the ticks, the labels, the plot, the legend, etc. You will always have an Axes object, even if the axes are not visible! The keyword argument frameon=False turns the frame off. An alternative method is:

Finding dimensions of a bounding box in Jmol

Mon, 13 Jun 2011 20:22:35 +0000

The Jmol applet (or Java application) is widely used to visualize the structure of molecules. It is very powerful, but not that well documented. If you don’t have Jmol running, open the page for Glucose Oxidase (1CF3) from the Protein Databank so you can follow the instructions for finding the size of the bounding box. Right-click on the applet to get a pop-up menu with lots of options. Under the Stylesubmenu, click onBoundboxto show a bounding box around the molecule:

Show bounding box

A self-contained Fortran linear equation solver

Wed, 16 Mar 2011 21:33:27 +0000

I’ve just released a self-contained Fortran module that solves a system of linear equations using the LU decomposition. Download the Fortran linear solver from github This module is based on code that was implemented and released on the Web by Jean-Pierre Moreau. His implementation was based on one of the Numerical Recipes books. I updated his code to a more strict Fortran 90 standard and added the necessary comments so that it can be built as a Python module using f2py. I replaced Jean-Pierre’s Fortran test program with a simpler, self-contained program. I also included a Python script that implements the same test case. I created this module because sometimes a self-contained routine is more appropriate than a full library. I am compiling a library that implements a custom boundary condition for a proprietary computational fluid dynamics solver (CFD-ACE+). The library has to be written in Fortran, and it has to be built using a proprietary set of build scripts. I could either try to reverse-engineer the build process and to modify it to link to a shared library like LAPACK, or I could implement a self-contained solver. Since Pierre had already implemented the solver, I was able to slightly modify his code and get it working relatively quickly.

Finding a value in an unordered Fortran array

Fri, 11 Mar 2011 16:27:51 +0000

I have been optimizing some Fortran code that involved searching for an integer value in an unordered array (we know the value occurs only once). Since there is no intrinsic procedure to accomplish this, I thought I’d try a couple of approaches to see which was fastest. The simple answer is that, in this case, brute force beats elegance, even when the target value is near the end of the array. Download the full example from GitHub

Sage: open-source mathematical software

Mon, 07 Mar 2011 14:07:19 +0000

I’ve recently gained a lot of experience with Sage, an open-source alternative to MATLAB, Mathematica, Maple, MuPAD, and Magma. Here are a couple of links to check out:

Public notebook servers--try it online right now! Interactive examples with Sage

Mathematical publications citing Sage

Sage vs. Mathematica

My experience with mathematical software started my freshman year at the University of Illinois when I signed up for a calculus class that was taught almost entirely with Mathematica. I grew to love Mathematica’s symbolic computation and plotting capabilities, although I found its programming language to be cumbersome. Once I was no longer a student, Mathematica was no longer an option due to the hefty licensing fees. With the caveat that I haven’t used Mathematica in many years, I will say that Sage compares very favorably to my experience with Mathematica. Sage has a notebook interface that allows you to integrate code, results, text, typeset equations, and graphics in an interactive document that can be viewed with any standards-compliant web browser. I don’t know how Sage stacks up against Mathematica in areas of advanced mathematics. I have heard that Sage is far ahead of Mathematica in number theory, because the lead developer of Sage is a number theorist.

Linear system simulation with Python

Fri, 17 Dec 2010 21:33:16 +0000

Linear time-invariant (LTI) systems are widely used in the field of signal processing. Scipy contains powerful tools for simulating LTI systems in the scipy.signal package, but they are not well documented. I will provide a simple example that demonstrates how to use a few of the core classes and functions in scipy.signal for simulating LTI systems with Python.

Define an LTI system

You will need to have Scipy installed, and you will need to have Matplotlib as well to make plots. We will start with an example of a first-order LTI system, which is characterized by a single parameter known as the time constant. scipy.signal defines the class lti to represent a linear system in Python.

Using Microsoft Word for Technical Documents

Wed, 18 Aug 2010 23:30:34 +0000

Microsoft Word is not the best tool for doing technical writing. However, sometimes we are required to use Word because we need to collaborate with others who want to use Word. Right now, I am using Word 2007 on Windows XP to write several mathematical papers. In general, it is a big improvement from previous versions. The new integrated equation editor is outstanding–except for the major bug I’ll discuss below. Here is a brief “FAQ” you will want to read if you are using Word for technical writing. **Q:**How do I enter multi-line equations in Word 2007?**A:Press Shift-Enter where you want a line break to appearQ:**How do I prevent a page break from splitting a table into two parts?**A:**Select the table. On the Home tab, click the little box in the lower-right corner of the Paragraph box. On the paragraph dialog, choose the “Lines and Page Breaks” tab. Check the box for “Keep Lines Together.”**Q:**How do I automatically number headings in Word 2007? For example: Section 1, Subsection 1.1, 1.2, 1.3, Section 2, etc.**A:**It’s not obvious. See this page from dummies.com about number headings.**Q:**In Word 2007, why do equations sometimes appear as blank spaces or question marks when I print or save my document as a PDF file?**A.**This occurs when Word is installed on Windows XP Pro. See the following Microsoft tech support item to find out how to install missing scripts: The characters in an equation are not printed… You may also have an outdated printer driver: Microsoft Support Item 920228**Q:**Why do equations created in Word 2007 disappear when I open the document in Word 2008 for Mac?A. Word for Mac does not support equations written in the new Word 2007 equation editor. Unfortunately, neither does PowerPoint 2007 on the PC. You can work around this by inserting equations into Word the old-fashioned way: go to the Insert tab, click on Object (found in the “Text” box towards the right side of the tab), and choose “Microsoft Equation 3.0” from the list in the dialog box.

Constrained least-squares fitting with Python

Fri, 05 Mar 2010 19:27:25 +0000

Scipy contains a good least-squares fitting routine, leastsq(), which implements a modified Levenberg-Marquardt algorithm. I just learned that it also has a constrained least-squared routine called fmin_slsqp(). I am using simple upper and lower bound constraints, but it’s also possible to specify more complex functional constraints. What I did not realize, at first, is that fmin_slsqprequires a different type of objective function thanleastsq. leastsqrequires you to write a function that returns a vector of residuals, andleastsqautomatically squares and sums the residuals.fmin_slsqp is actually more flexible, in that it can use any objective function that returns a single scalar value. To implement least-squares curve fitting, your objective function will need to find the residual at each data point, square the values, and sum them up. Hopefully this tip will save you some time. Check out the scipy optimization tutorial for more examples. Here is the original paper by Dieter Kraft which introduces the algorithm used by fmin_slsqp.

Storing large Numpy arrays on disk: Python Pickle vs. HDF5

Sun, 10 Jan 2010 23:00:20 +0000

In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: [code language=“Python”]#!/usr/bin/env python from numpy import array

Building and linking to a shared Fortran library

Mon, 26 Oct 2009 22:34:02 +0000

I’m using GNU Fortran (gfortran) to build several shared libraries, and then dynamically linking to them from a Fortran program. The process is a little different than what I’m used to for C libraries, so I thought I’d explain it. Unlike C, there is no need to #include header files when compiling code that relies on functions defined in an external library. Likewise, there is no need to use -l or -L linking flags to tell the linker about s hared libraries (at least when they’re in the same directory). In fact, the whole process requires a lot less command-line options than I had expected.

Profiling memory usage of Python code

Fri, 17 Apr 2009 22:08:52 +0000

In a previous post, I explained how to use the Python profiler. The profile is great for finding out which parts of the code run the slowest, or are called most often. However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed. If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude. On the positive side, your code may run much faster if it fits entirely in the processor cache. In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

Lookup tables and spline fitting in Python

Wed, 08 Apr 2009 23:36:09 +0000

Lookup tables and spline fitting are widely used by scientific programmers. A particular function may not have an analytic solution–in other words, it can’t be expressed as an equation of elementary functions. This might happen if the function were empirically determined from experimental data, or if the equation can’t be algebraically solved for one variable. If the analytical function is available, but takes a long time to evaluate, a lookup table or spline approximation can be considerably faster. In a previous post, I showed how to use the function interp1d from scipy.interpolate as a lookup table. In a later post, I showed that interp1d is actually rather slow, and scipy.interpolate.UnivariateSpline is much faster. Now, I will show some benchmark results, and explain a potential pitfall when using UnivariateSpline. The two functions I will use for this demonstration are not very complicated. Each function is a piecewise approximation of a more complex function. Here is one of the functions. For small z, the function uses one approximation, and for large z, it uses a far-field approximation. For intermediate values, a polynomial is used to smoothly match the two solutions.

3D Plotting Software for Python::Part 1::PyX

Fri, 20 Mar 2009 23:08:55 +0000

There are lots of good open-source tools that you can use to make high-resolution, publication-quality 2D plots. Personally, I like to use Python, numpy, and matplotlib. Unfortunately, it is much harder to find a good tools to make 3D plots. Older versions of matplotlib had rudimentary 3D support, but this was removed in version 0.98. In this post, I will review a Python 3D plotting library called PyX.

Preparing the data

Figuring out how to store the data to be plotted was actually the hardest part of learning to use PyX. The data format for 3D plots is not well documented. PyX requires a list of (x,y,z) lists like this:

Updated Python class for writing Paraview (VTK) (.vtu) files

Fri, 23 Jan 2009 23:28:05 +0000

I have released a new version of my Python class that generates VTK data files in the .vtu format, which is compatible with Paraview and other VTK applications. If you have downloaded the old one, please get the latest version, which incorporates some bug fixes and has been more thoroughly tested.

Scipy.integrate ODEPACK import error solved!

Wed, 14 Jan 2009 16:46:16 +0000

I recently found a solution to a problem that had been vexing me for about a year. In order to successfully import anything from scipy.integrate, I had edit the file scipy/integrate/__init__.py and comment out the line

from odepack import *

If not, I would get various import errors such as

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python2.5/site-packages/scipy/integrate/__init__.py", line 10,
in <module>
 from odepack import *
 File "/usr/lib/python2.5/site-packages/scipy/integrate/odepack.py", line 7,
in <module>
 import _odepack
ImportError: /usr/lib/python2.5/site-packages/scipy/integrate/_odepack.so:
undefined symbol: daxpy_

I found the solution in Gentoo bug 251165. The problem only occurs when Scipy is built with non-reference versions of BLAS and CBLAS. The solution is to install the reference implementation of BLAS and CBLAS, rebuild Scipy, and then use whichever implementation of BLAS or CBLAS you want. I don’t know why this happens, and I don’t know if it affects distributions other than Gentoo. By the way, Gentoo has a really useful system tool called eselect, which has various modules that are used to choose between different versions or implementations of tools on your system (BLAS, Java virtual machine, kernel sources, OpenGL, etc.)

Optimizing Python code for fast math

Fri, 09 Jan 2009 21:14:38 +0000

I spent some time today profiling a Brownian dynamics simulation written in Python to see how I could make it faster before starting some long runs on a Linux cluster. In the sections below, I have attempted to quantify the speed improvements due to various changes. Keep in mind that the actual speed improvement in your code will vary, depending on where the actual bottlenecks occur. See my post about profiling Python code. Another caveat: I am running Python 2.4.4 because it’s installed on our cluster.

A lookup table for fast Python math

Thu, 11 Dec 2008 23:17:06 +0000

Numerical programming frequently requires the use of look-up tables. A look-up table is a collection of pre-computed values. When given an “x” value, the table returns a pre-computed “y” value. Look-up tables can be used to speed up numerical codes, when it is faster to look up a value in the table than it is to compute the value. They are also used when the data in the table cannot be computed–for example, experimental data or averaged results from an ensemble of Monte Carlo simulations. Another application is to compute a value when a function cannot be solved algebraically. Assume that you have a formula for a function q(h). You need the value of h for a given value of q, but the formula cannot be algebraically solved to get h(q). Instead, choose a range of h values, compute the function q(h), and store each value in a look-up table. Now you can get h(q) for any value stored in the table. The major limitation of a look-up table is that it cannot return valid results for any value of q which is outside the range of those stored in the table. Depending on its implementation, the table may be able to interpolate to return values between known points.

Update 2: building 64-bit Numpy with Intel compilers and MKL

Tue, 09 Dec 2008 19:33:12 +0000

NOTE: these instructions are obsolete. Please see Building NumPy on a 64-bit Red Hat Cluster with Intel MKL. In a previous post I described how I built Numpy with Intel compilers and the Math Kernel Library on a 64-bit cluster. Today I upgraded to Numpy-1.2.1 and I made a few improvements to my install process. Please read the previous post, since I will not duplicate some important information, and then read on. This time, I made use of a site.cfg file. Copy the file site.cfg.exampletosite.cfg and edit. At the end of the file, uncomment the [mkl] section and set the path to your library. Mine looks like:

Even faster collision detection in Python using Numpy

Thu, 16 Oct 2008 19:44:15 +0000

Last night, in the shower, I realized that my collision detection routine could be even faster. Here is a representative snippet of code from my previous post:

 d2 = (x-self.x[0:i])*(x-self.x[0:i]) + (y-self.y[0:i])*(y-self.y[0:i]) + (z-self.z[0:i])*(z-self.z[0:i])

For some reason, I used the code (x-self.x)*(x-self.x) instead of (x-self.x)**2. Upon further reflection, I realized that (x-self.x)*(x-self.x) computes the difference between array elements twice, and then multiplies the results. Using a “power function” should enable the interpreter to compute the difference only once, and then multiply each element times itself. Here is the updated code, using Python’s power operator:

Installing numpy with the Intel Math Kernel Library (mkl)

Fri, 19 Sep 2008 21:24:58 +0000

Today I installed numpy on a cluster. Normally, as a Gentoo admin, I just install things with emerge, and all the details are taken care of automagically. However, this cluster runs Red Hat Enterprise, and I don’t have admin privileges, so I had to install numpy in my home directory. I installed 1.0.4, to match the version used on another system. You may not need to do this for more recent versions of numpy, which may have an improved setup script. The overall process is:

Reading Labview binary files with Python

Wed, 25 Jun 2008 16:04:46 +0000

My research group uses Labview 7.1 to write custom data acquisition (DAQ) software. I code everything else in Python, so I need to get data from Labview into Python for processing. Our DAQ program produces Labview binary files, so I had to find a way to read them with Python. Binary files are nice because they are a compact way to store numerical data as compared to ASCI or (heaven forbid) XML, but they are much harder to read. The binary format used by Labview is documented only indirectly, so I had to hack a little. The first thing to realize is that the Labview binary file is a direct dump of the data that was stored in RAM. How Labview stores data in memory is documented here. Indirectly, this documents how binary files are stored on disk. Our DAQ program writes a rather complex “cluster” (Labview’s version of a C structure) to disk. The elements of the cluster are stored contiguously as a sequence of bytes, and there’s no way to know which byte goes with which element, unless you know the size of each element and the order in which they are stored in the cluster. So, the first step is to document the cluster that’s being written to disk. You can use the context help in Labview to view the data type of the wire that leads to the VI that writes the file. With this in hand, you are ready to write Python code. First, make sure you open the file in binary mode: