I had to re-build Numpy because our cluster was upgraded and the Intel compilers and libraries were moved to a different directory.  This turned out to be a half-day affair of trial-and-error.  I learned a few important things, which I will try to list here:

*Delete the numpy-1.0.4/build directory after every build attempt.  Doing “python setup.py clean” isnoteffective.  I kept getting errors about undefined symbols when I tried to “import numpy” on the Python command line.  It was looking for symbols in the old locations, even though I had just rebuilt the code using the new library locations.  It turned out that I needed to delete the build directory in order to force a complete bottom-up rebuild.*The use of “setup.py” from distutils is not well documented online.  The best thing to do is run “python setup.py –help-commands” to get a list of available commands.  Then run “python setup.py <cmd> –help” to get help for that specific command.  You can string commands together on the command line, as I will show in the example below.When you test the new numpy, make sure you arenotin the numpy-1.0.4 directory!  If you are in the numpy source directory, when you import numpy, you will get the message “Running from numpy source directory.” and you will not be able to load any symbols from numpy. On 64-bit architectures, you need to compile position-independent library code.  For some reason, distutils does not do this automatically, and the compilation will fail with an error similar to the following:

Last night, in the shower, I realized that my collision detection routine could be even faster. Here is a representative snippet of code from my previous post:

    d2 = (x-self.x[0:i])*(x-self.x[0:i]) + (y-self.y[0:i])*(y-self.y[0:i]) + (z-self.z[0:i])*(z-self.z[0:i])

For some reason, I used the code (x-self.x)*(x-self.x) instead of (x-self.x)**2. Upon further reflection, I realized that (x-self.x)*(x-self.x) computes the difference between array elements twice, and then multiplies the results. Using a “power function” should enable the interpreter to compute the difference only once, and then multiply each element times itself. Here is the updated code, using Python’s power operator:

We recently learned the hard way that the use of haze can lead to reliability problems for some LCD projectors. We have two rather large LCD projectors (don’t know the specs offhand) permanently mounted to the ceiling, projecting onto the front of screens located on both sides of the stage.  A third LCD projector hits a large rear-projection screen that basically forms the back wall of the stage.  We also have an oil-based hazer mounted above the stage.  We had some old, tired projectors, so we assumed that they were just old and unreliable.  However, after replacing them with new, more powerful models, the reliability issues continued.  After sending the new projectors out for service for the second time in less than a year, the service center informed us that a film of oil had been deposited on the LCD.  You can buy expensive sealed-optics projectors, which should work reliably in dirty environments, but we saved money and bought standard models.  We don’t have the budget to replace the “new” projectors, so we’ve had to stop using haze altogether.  The stage doesn’t look nearly as good without it, and our moving lights are much less useful. On a side note: you should be aware that haze can also damage moving light fixtures that use a fan to cool the power supply or motors.  The fan sucks in haze and coats the internal components with a film of oil.  Eventually, something overheats and the fixture can actually catch fire.  Fortunately, we have older High End Studiospots and Trackspots that apparently don’t use forced-air cooling, so apparently the haze doesn’t really get inside.  Be warned!

Python is a very-high-level language.  That makes it easy to write code quickly, but the program may not be as fast as a program compiled from a lower-level language.  For this reason, many scientific programs are written in Fortran or C++.  However, it has always been my experience that the majority of time on a project is spent in writing, modifiying, and debugging code, rather than executing.  Fortunately, if written correctly, the time-critical parts of Python code can execute almost as fast as compiled software.  Here is an example of a collision-detection algorithm which achieved almost a ten-fold increase in speed when written to use Numpy.

In my previous post about cheap LED fixtures, I didn’t mention our Chauvet Q-Wash intelligent moving light LED fixtures. We bought four at a very discounted price to supplement our conventional movers. They weren’t bright enough to stand out in the stage wash, but they made nice little beams in the haze, and they could be used to “fill in” small dark spots on the stage or set pieces. As wash lights, they lack gobos or beam control, but they do have smooth color RGB mixing (most “real” wash lights do have some kind of beam width control). So far, you might be thinking that you should pick up a couple, so I should tell you now that they all stopped working within a year or so of installation. One died within a month or two, and after about six months of operation, we lost one every couple of months until they were all gone. The problem wasn’t the LEDs–it was the motors or control circuitry. We were using haze, which is known to shorten the lifespan of moving lights, but that’s the standard operating environment for concert lights! The fixtures were also permanently mounted above our stage, so they weren’t damaged during handling. They just quietly died on their own–well, except for the one that went insane and just started moving around on its own until we got up in the lift and unplugged it! Fortunately this didn’t happen during a performance, as the motors got very hot and might have started smoking or something. The moral of the story is that a low initial price doesn’t always save you money in the long run.

Today I installed numpy on a cluster. Normally, as a Gentoo admin, I just install things with emerge, and all the details are taken care of automagically. However, this cluster runs Red Hat Enterprise, and I don’t have admin privileges, so I had to install numpy in my home directory. I installed 1.0.4, to match the version used on another system. You may not need to do this for more recent versions of numpy, which may have an improved setup script. The overall process is:

The pickle module provided with Python is so useful that I’m surprised I haven’t used it before. Pickle allows you to save an entire data structure (such as an object) to disk as a binary file in a effortless (and fairly efficient) manner. For example, in my latest project I have created a Monte Carlo simulation that can take quite a bit of time to run. I also need to make multiple runs to get statistics on the results. At the end of each run, I need to dump the resulting data to disk so that it can be read in later by an analysis program. If I had to write data in a format that could be interchanged with other scientific software, I’d use the hdf5 format with the pytables package. However, right now I just need to get something working, and the pickle module is perfect. Here is how I save an object called box:

http://www.spore.com/flash/dsDemo/sporeds_widget.swf

It’s remarkably easy to spawn a Python thread.  However, before doing so, I caution you that a Python thread is not the same thing as an OS thread.  Python threads run within the Python interpreter, but the Python interpreter always executes in a single process.  The reasons why have already been explained elsewhere, so I refer you to the thread module documentation to learn about the Global Interpreter Lock.  You probably have objections to this state of affairs, and I assure you they have already been voiced by Juergen Brendel and responded to by Guido van Rossum (creator of Python).  Anyway, the upshot is that Python can only utilize one core of a multi-core CPU.  This isn’t such a big deal for me because I’m a scientific programmer, and if I really need to write parallel code it’s going to have to run on a cluster or a grid.  Threads don’t help with that. Having said all that, threads in Python are still useful.  I will detail one example in which I spawn a thread to load a large binary file.  While this doesn’t spread the work across multiple CPU cores, it does enable the GUI to remain interactive while the file loads. All you have to do to create a Python thread is create a class that is derived from Thread. In the example below, I derived a class called Loader, which “wraps” the function that actually reads the binary files. The __init__ method accepts the filename and other options as arguments. The run() method is required. Don’t call run() directly–instead, call the start() method (inherited from the base class) to start the thread.