In a previous post, I explained how to use the Python profiler.  The profile is great for finding out which parts of the code run the slowest, or are called most often.  However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed.  If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude.  On the positive side, your code may run much faster if it fits entirely in the processor cache.  In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

Lookup tables and spline fitting are widely used by scientific programmers.  A particular function may not have an analytic solution–in other words, it can’t be expressed as an equation of elementary functions.  This might happen if the function were empirically determined from experimental data, or if the equation can’t be algebraically solved for one variable.  If the analytical function is available, but takes a long time to evaluate, a lookup table or spline approximation can be considerably faster.  In a previous post, I showed how to use the function interp1d from scipy.interpolate as a lookup table.  In a later post, I showed that interp1d is actually rather slow, and scipy.interpolate.UnivariateSpline is much faster.  Now, I will show some benchmark results, and explain a potential pitfall when using UnivariateSpline. The two functions I will use for this demonstration are not very complicated.  Each function is a piecewise approximation of a more complex function.  Here is one of the functions.  For small z, the function uses one approximation, and for large z, it uses a far-field approximation.  For intermediate values, a polynomial is used to smoothly match the two solutions.

There are lots of good open-source tools that you can use to make high-resolution, publication-quality 2D plots.  Personally, I like to use Python, numpy, and matplotlib.  Unfortunately, it is much harder to find a good tools to make 3D plots.  Older versions of matplotlib had rudimentary 3D support, but this was removed in version 0.98.  In this post, I will review a Python 3D plotting library called PyX.

Preparing the data

Figuring out how to store the data to be plotted was actually the hardest part of learning to use PyX.  The data format for 3D plots is not well documented.  PyX requires a list of (x,y,z) lists like this:

My simulation library, which is written mostly in Python, needs a lot of data and parameters in order to run. In some cases, I just hard-code the values in the script that calls the library, and in other cases I load a pickle file containing a Python objext. What if I want to read in data or configuration parameters from a human-editable text file? If the information is extensive or complex, XML might be a good choice, but XML is overkill for simple configuration or data files. Fortunately, a standard Python library called ConfigParser has already defined a configuration file format, and provides methods to interact with such files. Here is a sample of the format used in a config file:

I found two complementary plugins that enable me to put highlighted formatted code in a Wordpress page or post. Here’s an example of what they do:

[sourcecode language="python"]
# Plot flux at continuum boundary
pylab.figure()
pylab.hold(True)
pylab.plot(nd_times, Jl_BD, 'b-', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD+Jl_BD_std, 'b.', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD-Jl_BD_std, 'b.', label="Flux from BD simulation")
[/sourcecode]

Test successful! Note that Javascript must be enabled to see the highlighting. Get the plugins here:

Writing applications in Python on a Linux system is almost too easy.  Deploying Python apps on other Linux systems is not hard, because most Linux systems already have Python, with its core libraries and tools, installed.  Most Linux systems also have package managers that make it easy to find and install required components.  But, what happens when your co-workers who use Windows need to use your app?  When you tell them to “go to the command line and…” you’ve pretty much lost them at “command line.” How do you package Python in a way that’s easy for a Windows user to install? Here is a process that worked for me:

I spent some time today profiling a Brownian dynamics simulation written in Python to see how I could make it faster before starting some long runs on a Linux cluster. In the sections below, I have attempted to quantify the speed improvements due to various changes. Keep in mind that the actual speed improvement in your code will vary, depending on where the actual bottlenecks occur. See my post about profiling Python code. Another caveat: I am running Python 2.4.4 because it’s installed on our cluster.

I wrote a previous post about how to redirect standard output from a Python script to a GUI window.  In this post, I will give an even simpler example of to redirect standard output to a log file.  During the early development and debugging of Python programs, I use print statements to keep me informed of what’s happening.  However, printing to the terminal is not always practical–for example, when I run numerical code on a parallel cluster, there is no way to determine which output came from which instance of the program.  Here is a class that you can use to redirect standard output to a log file:

I have found a few tools over the years that I find extremely useful for developing software. Python is my language of choice at the moment, but I’m sure these tools will be handy for any language.

• Subversion is an open-source version-control system. Version control was designed to allow multiple programmers to work on the same project at the same time without stepping all over one another. However, even though I am a solo developer, I find version control to be extremely helpful.
  1. When I commit changes to the repository, I can document what I’ve changed and why. This is a great help when I introduce a bug and have to go back and find it.
  2. The repository is stored on a remote server that is backed up nightly.
  3. It’s easy to make an “unstable” branch for implementing new features. When I make changes that don’t work, it’s easy to revert to a previous version that works.
  4. It is easy to deploy my code to the Linux cluster and make sure that the cluster is running the latest version of my software.

Go read the documentation on the Subversion web site to find out what it can do for you.RapidSVN is a GUI client for a Subversion server. By default, Subversion comes with a command-line client that does everything you need. However, sometimes it’s easier to stay organized when everything is presented visually. Here is a screenshot of RapidSVN: