Python on shocksolution.com

Service account credentials with the Python client for the Google Drive API (v3)

Mon, 23 Apr 2018 12:23:29 +0000

EDIT: Get the full code for this post on Github. This article only contains important snippets of code that require explanation.

There are numerous ways to authenticate against the Google Drive API. If you have an application running on Google Compute Engine that needs to access Drive, a Service Account is probably the easiest way to do it. One use case is for an application to write reports or log files to Drive so that users can see them without logging into a server. Before you try this example, go through all of the steps in Google’s Using OAuth 2.0 for Server to Server Applications guide and save your service account’s private key locally in JSON format. Getting credentials from a service account file is easy:

pickle, hickle and HDF5

Sun, 03 Feb 2013 19:33:57 +0000

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. The hickle developers have made a good start, and they have a long way to go before hickle will be useful to a wider audience. Right now, hickle can only store NumPy ndarrays and Python list objects. If you only need to store lists and arrays, you might as well use HDF5 bindings for Python such as PyTables or h5py. The power of the pickle module is that you can immediately serialize almost any Python object of arbitrary complexity, store it on disk, and retrieve it. hickle will only achieve its full potential once it replicates this functionality, and I’m not sure how difficult this will be. Ideally, you might be able to derive a class from Pickler that uses Picker’s methods to serialize an object, and then add your own method to write the serialized object to an HDF5 file. In a future post, I’ll describe some of the practical problems with using pickle files to store data, and try to organize some thoughts about how they might be solved.

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Wed, 16 Jan 2013 22:29:19 +0000

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

Tricks for Writing XML with Python 3

Thu, 27 Sep 2012 16:35:49 +0000

I’ve added a Python 3 XML example to my Shocksolution_Examples repo on GitHub. The new example shows how to generate an XML file which functions as a template for building a GUI with wxGlade. However, this example should be helpful for anyone who needs to create XML files with Python. The full example is on GitHub, so I’m just going to highlight a few interesting snippets. Use the SubElement factory function to create a new Element instance and add it to an existing element. Here, I create an element called templatedata and add it to the root element. I then create another element called author and add it to templatedata.

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

Mon, 02 Jul 2012 17:14:54 +0000

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

Tue, 24 Apr 2012 16:40:40 +0000

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL). This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

Python string format examples

Thu, 03 Nov 2011 22:38:10 +0000

The format method for Python strings (introduced in 2.6) is very flexible and powerful. It’s also easy to use, but the documentation is not very clear. It all makes sense with a few examples. I’ll add more as I have time:

Formatting Numbers in Python Strings

Formatting a floating-point number

[code language=“python”] “{0:.4f}".format(0.1234567890) “{0:.4f}".format(10.1234567890) [/code] The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field” 0 indicates the first argument to method format : indicates the start of the format specifier .4 indicates four decimal places f indicates a floating-point number

How to build ScipPy with Python 2.7.2 on CentOS5

Thu, 25 Aug 2011 00:17:57 +0000

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path. This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib). EDIT 2: you should use the –prefix=/home/yourusername instead of –user. The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages. The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up! I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5. I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system. Here is how I did it. First, with superuser privileges, use yum to install the following packages. You may need to set up epel as an alternate repository:

Removing an axis or both axes from a matplotlib plot

Wed, 17 Aug 2011 22:40:14 +0000

Sometimes, the frame around a matplotlib plot can detract from the information you are trying to convey. How do you remove the frame, ticks, or axes from a matplotlib plot?

matplotlib plot without a y axis

Some books you may find useful when working with matplotlib:

The full example is available on github.

First, we construct a figure and an axes object:

fig1 = plt.figure(facecolor='white')
ax1 = plt.axes(frameon=False)

The Axes object is a container that holds the axes, the ticks, the labels, the plot, the legend, etc. You will always have an Axes object, even if the axes are not visible! The keyword argument frameon=False turns the frame off. An alternative method is:

Python: lists to tuples and tuples to lists

Sat, 02 Jul 2011 17:07:00 +0000

Get the code for this example The zip function in Python accepts a number of sequences (such as lists or arrays) and returns a list of tuples. The first element of each tuple comes from the first sequence, the second from the second sequence, and so on. For example: [code language=“Python”]a = range(0, 5) b = range(5, 10) c = range(10, 15) sequence_of_tuples = zip(a, b, c) print(sequence_of_tuples)[/code]

[(0, 5, 10), (1, 6, 11), (2, 7, 12), (3, 8, 13), (4, 9, 14)]

A self-contained Fortran linear equation solver

Wed, 16 Mar 2011 21:33:27 +0000

I’ve just released a self-contained Fortran module that solves a system of linear equations using the LU decomposition. Download the Fortran linear solver from github This module is based on code that was implemented and released on the Web by Jean-Pierre Moreau. His implementation was based on one of the Numerical Recipes books. I updated his code to a more strict Fortran 90 standard and added the necessary comments so that it can be built as a Python module using f2py. I replaced Jean-Pierre’s Fortran test program with a simpler, self-contained program. I also included a Python script that implements the same test case. I created this module because sometimes a self-contained routine is more appropriate than a full library. I am compiling a library that implements a custom boundary condition for a proprietary computational fluid dynamics solver (CFD-ACE+). The library has to be written in Fortran, and it has to be built using a proprietary set of build scripts. I could either try to reverse-engineer the build process and to modify it to link to a shared library like LAPACK, or I could implement a self-contained solver. Since Pierre had already implemented the solver, I was able to slightly modify his code and get it working relatively quickly.

Sage: open-source mathematical software

Mon, 07 Mar 2011 14:07:19 +0000

I’ve recently gained a lot of experience with Sage, an open-source alternative to MATLAB, Mathematica, Maple, MuPAD, and Magma. Here are a couple of links to check out:

Public notebook servers--try it online right now! Interactive examples with Sage

Mathematical publications citing Sage

Sage vs. Mathematica

My experience with mathematical software started my freshman year at the University of Illinois when I signed up for a calculus class that was taught almost entirely with Mathematica. I grew to love Mathematica’s symbolic computation and plotting capabilities, although I found its programming language to be cumbersome. Once I was no longer a student, Mathematica was no longer an option due to the hefty licensing fees. With the caveat that I haven’t used Mathematica in many years, I will say that Sage compares very favorably to my experience with Mathematica. Sage has a notebook interface that allows you to integrate code, results, text, typeset equations, and graphics in an interactive document that can be viewed with any standards-compliant web browser. I don’t know how Sage stacks up against Mathematica in areas of advanced mathematics. I have heard that Sage is far ahead of Mathematica in number theory, because the lead developer of Sage is a number theorist.

Linear system simulation with Python

Fri, 17 Dec 2010 21:33:16 +0000

Linear time-invariant (LTI) systems are widely used in the field of signal processing. Scipy contains powerful tools for simulating LTI systems in the scipy.signal package, but they are not well documented. I will provide a simple example that demonstrates how to use a few of the core classes and functions in scipy.signal for simulating LTI systems with Python.

Define an LTI system

You will need to have Scipy installed, and you will need to have Matplotlib as well to make plots. We will start with an example of a first-order LTI system, which is characterized by a single parameter known as the time constant. scipy.signal defines the class lti to represent a linear system in Python.

Constrained least-squares fitting with Python

Fri, 05 Mar 2010 19:27:25 +0000

Scipy contains a good least-squares fitting routine, leastsq(), which implements a modified Levenberg-Marquardt algorithm. I just learned that it also has a constrained least-squared routine called fmin_slsqp(). I am using simple upper and lower bound constraints, but it’s also possible to specify more complex functional constraints. What I did not realize, at first, is that fmin_slsqprequires a different type of objective function thanleastsq. leastsqrequires you to write a function that returns a vector of residuals, andleastsqautomatically squares and sums the residuals.fmin_slsqp is actually more flexible, in that it can use any objective function that returns a single scalar value. To implement least-squares curve fitting, your objective function will need to find the residual at each data point, square the values, and sum them up. Hopefully this tip will save you some time. Check out the scipy optimization tutorial for more examples. Here is the original paper by Dieter Kraft which introduces the algorithm used by fmin_slsqp.

Storing large Numpy arrays on disk: Python Pickle vs. HDF5

Sun, 10 Jan 2010 23:00:20 +0000

In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: [code language=“Python”]#!/usr/bin/env python from numpy import array

f2py: binding Fortran and Python

Wed, 23 Sep 2009 22:56:08 +0000

I have recently started using f2py to call Fortran from Python. I have found this useful for two reasons: speeding up Python scripts by calling compiled Fortran code, and using Python as a unit testing framework for Fortran modules. Unfortunately, the documentation for f2py is rather sparse, and may not be completely up to date. In this note, I will hopefully prevent you from wasting a lot of time figuring out how to pass array arguments, and return array results.

Profiling memory usage of Python code

Fri, 17 Apr 2009 22:08:52 +0000

In a previous post, I explained how to use the Python profiler. The profile is great for finding out which parts of the code run the slowest, or are called most often. However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed. If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude. On the positive side, your code may run much faster if it fits entirely in the processor cache. In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

Lookup tables and spline fitting in Python

Wed, 08 Apr 2009 23:36:09 +0000

Lookup tables and spline fitting are widely used by scientific programmers. A particular function may not have an analytic solution–in other words, it can’t be expressed as an equation of elementary functions. This might happen if the function were empirically determined from experimental data, or if the equation can’t be algebraically solved for one variable. If the analytical function is available, but takes a long time to evaluate, a lookup table or spline approximation can be considerably faster. In a previous post, I showed how to use the function interp1d from scipy.interpolate as a lookup table. In a later post, I showed that interp1d is actually rather slow, and scipy.interpolate.UnivariateSpline is much faster. Now, I will show some benchmark results, and explain a potential pitfall when using UnivariateSpline. The two functions I will use for this demonstration are not very complicated. Each function is a piecewise approximation of a more complex function. Here is one of the functions. For small z, the function uses one approximation, and for large z, it uses a far-field approximation. For intermediate values, a polynomial is used to smoothly match the two solutions.

3D Plotting Software for Python::Part 1::PyX

Fri, 20 Mar 2009 23:08:55 +0000

There are lots of good open-source tools that you can use to make high-resolution, publication-quality 2D plots. Personally, I like to use Python, numpy, and matplotlib. Unfortunately, it is much harder to find a good tools to make 3D plots. Older versions of matplotlib had rudimentary 3D support, but this was removed in version 0.98. In this post, I will review a Python 3D plotting library called PyX.

Preparing the data

Figuring out how to store the data to be plotted was actually the hardest part of learning to use PyX. The data format for 3D plots is not well documented. PyX requires a list of (x,y,z) lists like this:

Updated Python class for writing Paraview (VTK) (.vtu) files

Fri, 23 Jan 2009 23:28:05 +0000

I have released a new version of my Python class that generates VTK data files in the .vtu format, which is compatible with Paraview and other VTK applications. If you have downloaded the old one, please get the latest version, which incorporates some bug fixes and has been more thoroughly tested.

Deploying Python applications on Windows

Thu, 15 Jan 2009 16:38:56 +0000

Writing applications in Python on a Linux system is almost too easy. Deploying Python apps on other Linux systems is not hard, because most Linux systems already have Python, with its core libraries and tools, installed. Most Linux systems also have package managers that make it easy to find and install required components. But, what happens when your co-workers who use Windows need to use your app? When you tell them to “go to the command line and…” you’ve pretty much lost them at “command line.” How do you package Python in a way that’s easy for a Windows user to install? Here is a process that worked for me:

Scipy.integrate ODEPACK import error solved!

Wed, 14 Jan 2009 16:46:16 +0000

I recently found a solution to a problem that had been vexing me for about a year. In order to successfully import anything from scipy.integrate, I had edit the file scipy/integrate/__init__.py and comment out the line

from odepack import *

If not, I would get various import errors such as

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python2.5/site-packages/scipy/integrate/__init__.py", line 10,
in <module>
 from odepack import *
 File "/usr/lib/python2.5/site-packages/scipy/integrate/odepack.py", line 7,
in <module>
 import _odepack
ImportError: /usr/lib/python2.5/site-packages/scipy/integrate/_odepack.so:
undefined symbol: daxpy_

I found the solution in Gentoo bug 251165. The problem only occurs when Scipy is built with non-reference versions of BLAS and CBLAS. The solution is to install the reference implementation of BLAS and CBLAS, rebuild Scipy, and then use whichever implementation of BLAS or CBLAS you want. I don’t know why this happens, and I don’t know if it affects distributions other than Gentoo. By the way, Gentoo has a really useful system tool called eselect, which has various modules that are used to choose between different versions or implementations of tools on your system (BLAS, Java virtual machine, kernel sources, OpenGL, etc.)

Optimizing Python code for fast math

Fri, 09 Jan 2009 21:14:38 +0000

I spent some time today profiling a Brownian dynamics simulation written in Python to see how I could make it faster before starting some long runs on a Linux cluster. In the sections below, I have attempted to quantify the speed improvements due to various changes. Keep in mind that the actual speed improvement in your code will vary, depending on where the actual bottlenecks occur. See my post about profiling Python code. Another caveat: I am running Python 2.4.4 because it’s installed on our cluster.

Redirecting standard output from Python: another example

Wed, 07 Jan 2009 21:54:10 +0000

I wrote a previous post about how to redirect standard output from a Python script to a GUI window. In this post, I will give an even simpler example of to redirect standard output to a log file. During the early development and debugging of Python programs, I use print statements to keep me informed of what’s happening. However, printing to the terminal is not always practical–for example, when I run numerical code on a parallel cluster, there is no way to determine which output came from which instance of the program. Here is a class that you can use to redirect standard output to a log file:

Tools for Python software development

Tue, 06 Jan 2009 18:52:45 +0000

I have found a few tools over the years that I find extremely useful for developing software. Python is my language of choice at the moment, but I’m sure these tools will be handy for any language.

Subversion is an open-source version-control system. Version control was designed to allow multiple programmers to work on the same project at the same time without stepping all over one another. However, even though I am a solo developer, I find version control to be extremely helpful.
1. When I commit changes to the repository, I can document what I’ve changed and why. This is a great help when I introduce a bug and have to go back and find it.
2. The repository is stored on a remote server that is backed up nightly.
3. It’s easy to make an “unstable” branch for implementing new features. When I make changes that don’t work, it’s easy to revert to a previous version that works.
4. It is easy to deploy my code to the Linux cluster and make sure that the cluster is running the latest version of my software.

Go read the documentation on the Subversion web site to find out what it can do for you.RapidSVN is a GUI client for a Subversion server. By default, Subversion comes with a command-line client that does everything you need. However, sometimes it’s easier to stay organized when everything is presented visually. Here is a screenshot of RapidSVN:

RapidSVN screenshot

A lookup table for fast Python math

Thu, 11 Dec 2008 23:17:06 +0000

Numerical programming frequently requires the use of look-up tables. A look-up table is a collection of pre-computed values. When given an “x” value, the table returns a pre-computed “y” value. Look-up tables can be used to speed up numerical codes, when it is faster to look up a value in the table than it is to compute the value. They are also used when the data in the table cannot be computed–for example, experimental data or averaged results from an ensemble of Monte Carlo simulations. Another application is to compute a value when a function cannot be solved algebraically. Assume that you have a formula for a function q(h). You need the value of h for a given value of q, but the formula cannot be algebraically solved to get h(q). Instead, choose a range of h values, compute the function q(h), and store each value in a look-up table. Now you can get h(q) for any value stored in the table. The major limitation of a look-up table is that it cannot return valid results for any value of q which is outside the range of those stored in the table. Depending on its implementation, the table may be able to interpolate to return values between known points.

Update 2: building 64-bit Numpy with Intel compilers and MKL

Tue, 09 Dec 2008 19:33:12 +0000

NOTE: these instructions are obsolete. Please see Building NumPy on a 64-bit Red Hat Cluster with Intel MKL. In a previous post I described how I built Numpy with Intel compilers and the Math Kernel Library on a 64-bit cluster. Today I upgraded to Numpy-1.2.1 and I made a few improvements to my install process. Please read the previous post, since I will not duplicate some important information, and then read on. This time, I made use of a site.cfg file. Copy the file site.cfg.exampletosite.cfg and edit. At the end of the file, uncomment the [mkl] section and set the path to your library. Mine looks like:

Using Python to generate XML files for visualization in Paraview

Thu, 13 Nov 2008 19:00:23 +0000

VTK is an open-source software system for “3D computer graphics, image processing, and visualization” developed by by Kitware. VTK is the foundation of Paraview, an industrial-strength CFD visualization tool that I have found to be very useful. I generate “second generation” XML-based files from my Python code and import them into Paraview for visualization. I am in the process of creating some Python classes to do, and I hope to publish them soon. Until then, I want to share some useful resources. The VTK file formats are specified in this document. It’s a pretty good specification, but it lacks some examples. Soon I will post an example of a valid unstructured, serial .vtu file. Each VTK file includes data from only one time step, so you have to keep track of time yourself (the filename is an easy solution). Paraview can read in data from multiple time steps, but you have to specify them in a .pvd file. This is also an XML file, with the following format: (reference)

Unexpected integer/float math behavior in Python

Fri, 07 Nov 2008 00:21:16 +0000

I wasted some time today tracking down a bug in one of my programs. It turned out to be “unexpected behavior” rather than a bug. I was aware of this aspect of the language, but I made an assumption and got bit. Read on for a valuable lesson. Python handles integer math differently than floating point math. If you type a number without a decimal point, Python treats it as an integer. All math performed only with integers results in integers.For example, 1/2 evaluates to 0 while 1./2. evaluates to 0.5. If you mix integers and floats, Python will produce a floating point result (1/2.=0.5), but you must be very careful. For example, you might expect the expression 4/3*3.14159 to yield a floating point result. It does yield a floating point number, butnotthe one you were expecting! 4/3*3.14159 yields 3.14159. What happened? Python works from left to right. 4/3 evaluates to the integer “1”. 1*3.14159 evaluates to 3.14159. For comparison, 4./3.*3.14159 evaluates to 4.1887866. Here’s the problem with this particular aspect of Python: according to the rules of math, 4/3*3.14159 is exactly the same expression as 4*3.14159/3, but in Python they yield different results if you forget the decimal points! 4*3.14159 evaluates to a floating point, so (4*3.14159)/3 yields the “correct” floating point value. Lesson Learned: be explicit about specifyingall floats if you are doing floating-point math! Sometimes I get lazy and leave a trailing decimal point off of a number when doing a floating point calculation, knowing that the results are “upcast” into floats. Not any more! Note: this unexpected behavior goes away in Python 3.0

Profiling Python code

Fri, 24 Oct 2008 21:37:24 +0000

“Speed” is a complicated term when used in the context of software. Does it mean raw speed of execution, or reducing the amount of time until a correct result is obtained? Python is not the first language that comes to mind when people think of “fast software.” It is true that pure Python will usually not execute as quickly as the same algorithm directly coded in C or Fortran. However, when you define speed as “least amount of time until you get the right answer,” then Python is pretty fast. It is so easy to develop correct code in Python, when compared to low-level compiled languages, that Python is often the fastet route to a correct answer, even if the execution time is longer. Having said that, there aretimes when code has to execute quickly, and that’s why I will introduce you to profiling Python code. Like all things Python, profiling is easier than you think. Python 2.4 has the profile module, and Python 2.5 has both profile and cProfile. cProfile is written in C for lower overhead, and it’s the recommended version. I am stuck with profile, because the cluster that I am working with still uses Python 2.4. You can read the docs for more details, but I will quickly outline what I find to be the most helpful usage. For example, say I want to profile the filerun_sim.py. I use the following command line:

Updated: building 64-bit Numpy with Intel compilers (icc)

Fri, 17 Oct 2008 19:36:38 +0000

I had to re-build Numpy because our cluster was upgraded and the Intel compilers and libraries were moved to a different directory. This turned out to be a half-day affair of trial-and-error. I learned a few important things, which I will try to list here:

*Delete the numpy-1.0.4/build directory after every build attempt. Doing “python setup.py clean” isnoteffective. I kept getting errors about undefined symbols when I tried to “import numpy” on the Python command line. It was looking for symbols in the old locations, even though I had just rebuilt the code using the new library locations. It turned out that I needed to delete the build directory in order to force a complete bottom-up rebuild.*The use of “setup.py” from distutils is not well documented online. The best thing to do is run “python setup.py –help-commands” to get a list of available commands. Then run “python setup.py <cmd> –help” to get help for that specific command. You can string commands together on the command line, as I will show in the example below.When you test the new numpy, make sure you arenotin the numpy-1.0.4 directory! If you are in the numpy source directory, when you import numpy, you will get the message “Running from numpy source directory.” and you will not be able to load any symbols from numpy. On 64-bit architectures, you need to compile position-independent library code. For some reason, distutils does not do this automatically, and the compilation will fail with an error similar to the following:

Even faster collision detection in Python using Numpy

Thu, 16 Oct 2008 19:44:15 +0000

Last night, in the shower, I realized that my collision detection routine could be even faster. Here is a representative snippet of code from my previous post:

d2 = (x-self.x[0:i])*(x-self.x[0:i]) + (y-self.y[0:i])*(y-self.y[0:i]) + (z-self.z[0:i])*(z-self.z[0:i])

For some reason, I used the code (x-self.x)*(x-self.x) instead of (x-self.x)**2. Upon further reflection, I realized that (x-self.x)*(x-self.x) computes the difference between array elements twice, and then multiplies the results. Using a “power function” should enable the interpreter to compute the difference only once, and then multiply each element times itself. Here is the updated code, using Python’s power operator:

Speeding up Python math with Numpy: collision detection example

Sun, 12 Oct 2008 18:10:20 +0000

Python is a very-high-level language. That makes it easy to write code quickly, but the program may not be as fast as a program compiled from a lower-level language. For this reason, many scientific programs are written in Fortran or C++. However, it has always been my experience that the majority of time on a project is spent in writing, modifiying, and debugging code, rather than executing. Fortunately, if written correctly, the time-critical parts of Python code can execute almost as fast as compiled software. Here is an example of a collision-detection algorithm which achieved almost a ten-fold increase in speed when written to use Numpy.

Installing numpy with the Intel Math Kernel Library (mkl)

Fri, 19 Sep 2008 21:24:58 +0000

Today I installed numpy on a cluster. Normally, as a Gentoo admin, I just install things with emerge, and all the details are taken care of automagically. However, this cluster runs Red Hat Enterprise, and I don’t have admin privileges, so I had to install numpy in my home directory. I installed 1.0.4, to match the version used on another system. You may not need to do this for more recent versions of numpy, which may have an improved setup script. The overall process is:

Python Pickle: Painless binary storage for Python objects

Mon, 15 Sep 2008 22:23:18 +0000

The pickle module provided with Python is so useful that I’m surprised I haven’t used it before. Pickle allows you to save an entire data structure (such as an object) to disk as a binary file in a effortless (and fairly efficient) manner. For example, in my latest project I have created a Monte Carlo simulation that can take quite a bit of time to run. I also need to make multiple runs to get statistics on the results. At the end of each run, I need to dump the resulting data to disk so that it can be read in later by an analysis program. If I had to write data in a format that could be interchanged with other scientific software, I’d use the hdf5 format with the pytables package. However, right now I just need to get something working, and the pickle module is perfect. Here is how I save an object called box:

Python threads are easy (with example)

Tue, 19 Aug 2008 22:50:07 +0000

It’s remarkably easy to spawn a Python thread. However, before doing so, I caution you that a Python thread is not the same thing as an OS thread. Python threads run within the Python interpreter, but the Python interpreter always executes in a single process. The reasons why have already been explained elsewhere, so I refer you to the thread module documentation to learn about the Global Interpreter Lock. You probably have objections to this state of affairs, and I assure you they have already been voiced by Juergen Brendel and responded to by Guido van Rossum (creator of Python). Anyway, the upshot is that Python can only utilize one core of a multi-core CPU. This isn’t such a big deal for me because I’m a scientific programmer, and if I really need to write parallel code it’s going to have to run on a cluster or a grid. Threads don’t help with that. Having said all that, threads in Python are still useful. I will detail one example in which I spawn a thread to load a large binary file. While this doesn’t spread the work across multiple CPU cores, it does enable the GUI to remain interactive while the file loads. All you have to do to create a Python thread is create a class that is derived from Thread. In the example below, I derived a class called Loader, which “wraps” the function that actually reads the binary files. The __init__ method accepts the filename and other options as arguments. The run() method is required. Don’t call run() directly–instead, call the start() method (inherited from the base class) to start the thread.

Redirecting text output from Python functions

Sat, 16 Aug 2008 20:47:05 +0000

Two posts ago, I described how I wrote a function in Python that reads in a binary file from Labview. In my last post, I described using wxPython to write a GUI to process the data from those binary files. Naturally, I called the binary-file-reader function from the GUI. The problem is that the file reader prints a lot of information to the terminal, using Python printstatements. None of this goes to the GUI, requiring the user to run the GUI from a terminal and keep an eye on the text output, which is inconvenient. However, I don’t want to modify the file reader to include GUI-specific code, because that would be less modular and less re-usable. Instead, I learned that Python has a very easy facility to redirectstdout, the default destination of the print statement. I modified the file reader as follows:

Fun with threads in Python and wxPython

Mon, 11 Aug 2008 22:12:07 +0000

I have finally gotten back to programming in the last couple of days. Our project has finally started to generate a lot of data, so I’ve been refactoring and improving my code that reads data stored in LabView binaries. Today I spent a lot of time creating a GUI for browsing data. Arguably, this wasn’t the best use of my time, but I learned a lot about multi-threaded Python GUI programming with wxPython. You can find a gold mine of information on the multi-threaded wx programming at the wxPython wiki. Because the LabView binary data has to be read sequentially, and the files are rather large, it takes a long time to read in a file. I spawn a thread to handle the file reading, while allowing the GUI to remain responsive. The thread posts messages to the GUI window, which are used to update the user on the status of the file reading operation. When the file is read, a final message containing the data is posted to the window. It’s really pretty slick now that I’ve figured out how to do it. I will soon post a clever scheme to capture text output from the file-reading function, and display it in the GUI, without making substantial changes to the file-reading function.

Reading Labview binary files with Python

Wed, 25 Jun 2008 16:04:46 +0000

My research group uses Labview 7.1 to write custom data acquisition (DAQ) software. I code everything else in Python, so I need to get data from Labview into Python for processing. Our DAQ program produces Labview binary files, so I had to find a way to read them with Python. Binary files are nice because they are a compact way to store numerical data as compared to ASCI or (heaven forbid) XML, but they are much harder to read. The binary format used by Labview is documented only indirectly, so I had to hack a little. The first thing to realize is that the Labview binary file is a direct dump of the data that was stored in RAM. How Labview stores data in memory is documented here. Indirectly, this documents how binary files are stored on disk. Our DAQ program writes a rather complex “cluster” (Labview’s version of a C structure) to disk. The elements of the cluster are stored contiguously as a sequence of bytes, and there’s no way to know which byte goes with which element, unless you know the size of each element and the order in which they are stored in the cluster. So, the first step is to document the cluster that’s being written to disk. You can use the context help in Labview to view the data type of the wire that leads to the VI that writes the file. With this in hand, you are ready to write Python code. First, make sure you open the file in binary mode: