Software Development on shocksolution.com

Multi-Page Blueprints for Confluence

Wed, 05 May 2021 20:37:05 +0000

A free demo app is available in the Atlassian Marketplace which supports multi-page blueprints for Confluence. This app works around numerous bugs in the official Atlassian blueprint tutorials and a key Atlassian API bug.

Background on Confluence

Confluence is one of the leading enterprise wiki products. Its built-in feature set is already very powerful, and it can be extended with applications available in the Atlassian Marketplace. You can also write your own applications for private, internal use. Atlassian provides the atlassian-connect-express toolkit for building apps with Node.js, which reduces development time and effort. They also provide a number of sample projects for Jira and Confluence apps on Bitbucket. Unfortunately, many of these examples are obsolete or deprecated, and do not function correctly with the latest version of atlassian-connect-express or the latest release of Confluence and Jira.

Node.js: Connecting to MySQL on a socket with Sequelize

Thu, 22 Apr 2021 18:54:25 +0000

According to its official description, “Sequelize is a promise-based Node.js ORM for Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server.” Sequelize is widely used in Node.js web applications to abstract the database layer. I recently found a “corner case” for Sequelize which is possible to accomplish, but is not well documented anywhere on the web. I need to use Sequelize to connect to a MySQL server via a UNIX socket. I’m developing some plugins for Confluence that use the atlassian-connect-express toolkit, which has a datastore that wraps Sequelize. The way Atlassian has chosen to wrap Sequelize is rather unfortunate, probably because they are trying to maintain backwards compatibility with JugglingDB. Essentially, the Atlassian Connect Express only looks at the URL, and ignores any option passed to Sequelize. Therefore, you have to pass everything you need via the URL, and this is where it’s tricky. Here’s the form of the URL that you need to connect to MySQL via a UNIX socket with Sequelize:

Configuring Laravel/Lumen applications to connect to SQL database sockets

Mon, 14 May 2018 18:58:47 +0000

The Laravel/Lumen framework documentation does not explain how to connect an application to a database using UNIX sockets instead of a TCP-based network connection. I recently had to configure the Polr URL shortener (built on the Lumen microframework by Laravel) to connect to Google Cloud SQL with a UNIX socket. Since all of Polr’s configuration takes place in the .env file, and there is no environment variable that’s specific to database sockets, this took some research. I finally found the answer in an obscure StackOverflow response. Previous TCP connection: [code] DB_HOST=some-server-name DB_PORT=3306 [/code] Socket-based SQL connection: [code] DB_HOST=localhost;unix_socket=/cloudsql/cloud-project-name:us-east1:sql-instance-name [/code] This approach will work with any UNIX socket; you just need to give it the absolute path to the socket.

Service account credentials with the Python client for the Google Drive API (v3)

Mon, 23 Apr 2018 12:23:29 +0000

EDIT: Get the full code for this post on Github. This article only contains important snippets of code that require explanation.

There are numerous ways to authenticate against the Google Drive API. If you have an application running on Google Compute Engine that needs to access Drive, a Service Account is probably the easiest way to do it. One use case is for an application to write reports or log files to Drive so that users can see them without logging into a server. Before you try this example, go through all of the steps in Google’s Using OAuth 2.0 for Server to Server Applications guide and save your service account’s private key locally in JSON format. Getting credentials from a service account file is easy:

"Exporting" a project from a Git repository

Fri, 24 Jan 2014 18:48:31 +0000

What do you do when you want to distribute or release source code that is stored in a Git repository? Obviously, if your target audience is using Git, you can just compress the directory that contains the repository and distribute the copies, or give the users a way to clone your repository (such as GitHub). However, your audience may not be Git users, or the hidden .git directory may be very large and you don’t want to distribute it. The solution is the git archive command, which packs the files from a tree-ish into an achive (ZIP or TAR). By “tree-ish”, they mean that you can specify a branch, commit, HEAD, etc. git archive is somewhat analagous to the svn export command. I find the most useful form of this command to be: cd example git archive --output ~/example.zip --format=zip --prefix=example/ HEAD Do not forget the trailing slash after the directory that you specify with the --prefix flag! REFERENCE: How to do a “git export” (like svn export)

Collaborative Git workflow: Shared Repository on a File Server

Fri, 10 Jan 2014 20:55:26 +0000

GitHub is a great tool for collaborating on projects. However, sometimes it is necessary to mimic the “GitHub workflow” using a shared repository on a local Linux server. The following example shows how I shared an example repository with multiple users. We are also using the Git flow model for branching, aided by the handy git flow plugin.

On my workstation

I started by creating a repo on my local workstation and setting it up to use the git flow plugin.

pickle, hickle and HDF5

Sun, 03 Feb 2013 19:33:57 +0000

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. The hickle developers have made a good start, and they have a long way to go before hickle will be useful to a wider audience. Right now, hickle can only store NumPy ndarrays and Python list objects. If you only need to store lists and arrays, you might as well use HDF5 bindings for Python such as PyTables or h5py. The power of the pickle module is that you can immediately serialize almost any Python object of arbitrary complexity, store it on disk, and retrieve it. hickle will only achieve its full potential once it replicates this functionality, and I’m not sure how difficult this will be. Ideally, you might be able to derive a class from Pickler that uses Picker’s methods to serialize an object, and then add your own method to write the serialized object to an HDF5 file. In a future post, I’ll describe some of the practical problems with using pickle files to store data, and try to organize some thoughts about how they might be solved.

Tricks for Writing XML with Python 3

Thu, 27 Sep 2012 16:35:49 +0000

I’ve added a Python 3 XML example to my Shocksolution_Examples repo on GitHub. The new example shows how to generate an XML file which functions as a template for building a GUI with wxGlade. However, this example should be helpful for anyone who needs to create XML files with Python. The full example is on GitHub, so I’m just going to highlight a few interesting snippets. Use the SubElement factory function to create a new Element instance and add it to an existing element. Here, I create an element called templatedata and add it to the root element. I then create another element called author and add it to templatedata.

Python string format examples

Thu, 03 Nov 2011 22:38:10 +0000

The format method for Python strings (introduced in 2.6) is very flexible and powerful. It’s also easy to use, but the documentation is not very clear. It all makes sense with a few examples. I’ll add more as I have time:

Formatting Numbers in Python Strings

Formatting a floating-point number

[code language=“python”] “{0:.4f}".format(0.1234567890) “{0:.4f}".format(10.1234567890) [/code] The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field” 0 indicates the first argument to method format : indicates the start of the format specifier .4 indicates four decimal places f indicates a floating-point number

Parsing INI configuration files with FORTRAN

Tue, 23 Aug 2011 21:14:44 +0000

Fortran would not be my first choice for working with text, in any form! However, sometimes even numerical codes need to read data from configuration files. The easiest way to read a configuration file from a Fortran 90 routine is by using namelist I/O (I really need to add an example of that). If you’re stuck with INI files, I found an INI file parser written in Fortran buried in an index of free Fortran routine. I don’t even remember how I even found it, since it’s not well labeled and doesn’t come up in the first few pages of Google results, so I thought I’d better write a post about it in case I ever need such a thing in the future.

A git branching strategy suitable for large projects

Sat, 13 Aug 2011 21:57:14 +0000

Git is an amazing tool…but what is the best way to use it? Like any tool that gives you great power and flexibility, it’s up to you to use the tool in the best way to suit your purpose. A friend of mine who manages an enterprise-class software development team recommended the git branching strategy explained in this blog post. I highly recommend reading it.

Replacing text in place with GNU sed

Mon, 11 Apr 2011 17:39:38 +0000

sed is a stream editor, which means that it accepts a stream of text, processes it, and spits out another stream of text. sed can process files that are too large to load into memory, and it is a completely command-line tool that can easily be integrated into shell scripts. This excellent sed tutorial was written for an old version of sed provided by Sun Microsystems, and it doesn’t cover one of the most useful features of GNU sed. GNU sed accepts a -icommand line argument that tells sed to replace the text file in place, rather than writing the output stream to another file. Another nice feature is thatsed -i will create a backup file before processing if you provide a backup suffix:

A self-contained Fortran linear equation solver

Wed, 16 Mar 2011 21:33:27 +0000

I’ve just released a self-contained Fortran module that solves a system of linear equations using the LU decomposition. Download the Fortran linear solver from github This module is based on code that was implemented and released on the Web by Jean-Pierre Moreau. His implementation was based on one of the Numerical Recipes books. I updated his code to a more strict Fortran 90 standard and added the necessary comments so that it can be built as a Python module using f2py. I replaced Jean-Pierre’s Fortran test program with a simpler, self-contained program. I also included a Python script that implements the same test case. I created this module because sometimes a self-contained routine is more appropriate than a full library. I am compiling a library that implements a custom boundary condition for a proprietary computational fluid dynamics solver (CFD-ACE+). The library has to be written in Fortran, and it has to be built using a proprietary set of build scripts. I could either try to reverse-engineer the build process and to modify it to link to a shared library like LAPACK, or I could implement a self-contained solver. Since Pierre had already implemented the solver, I was able to slightly modify his code and get it working relatively quickly.

Finding a value in an unordered Fortran array

Fri, 11 Mar 2011 16:27:51 +0000

I have been optimizing some Fortran code that involved searching for an integer value in an unordered array (we know the value occurs only once). Since there is no intrinsic procedure to accomplish this, I thought I’d try a couple of approaches to see which was fastest. The simple answer is that, in this case, brute force beats elegance, even when the target value is near the end of the array. Download the full example from GitHub

Sage: open-source mathematical software

Mon, 07 Mar 2011 14:07:19 +0000

I’ve recently gained a lot of experience with Sage, an open-source alternative to MATLAB, Mathematica, Maple, MuPAD, and Magma. Here are a couple of links to check out:

Public notebook servers--try it online right now! Interactive examples with Sage

Mathematical publications citing Sage

Sage vs. Mathematica

My experience with mathematical software started my freshman year at the University of Illinois when I signed up for a calculus class that was taught almost entirely with Mathematica. I grew to love Mathematica’s symbolic computation and plotting capabilities, although I found its programming language to be cumbersome. Once I was no longer a student, Mathematica was no longer an option due to the hefty licensing fees. With the caveat that I haven’t used Mathematica in many years, I will say that Sage compares very favorably to my experience with Mathematica. Sage has a notebook interface that allows you to integrate code, results, text, typeset equations, and graphics in an interactive document that can be viewed with any standards-compliant web browser. I don’t know how Sage stacks up against Mathematica in areas of advanced mathematics. I have heard that Sage is far ahead of Mathematica in number theory, because the lead developer of Sage is a number theorist.

Linear system simulation with Python

Fri, 17 Dec 2010 21:33:16 +0000

Linear time-invariant (LTI) systems are widely used in the field of signal processing. Scipy contains powerful tools for simulating LTI systems in the scipy.signal package, but they are not well documented. I will provide a simple example that demonstrates how to use a few of the core classes and functions in scipy.signal for simulating LTI systems with Python.

Define an LTI system

You will need to have Scipy installed, and you will need to have Matplotlib as well to make plots. We will start with an example of a first-order LTI system, which is characterized by a single parameter known as the time constant. scipy.signal defines the class lti to represent a linear system in Python.

Storing large Numpy arrays on disk: Python Pickle vs. HDF5

Sun, 10 Jan 2010 23:00:20 +0000

In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: [code language=“Python”]#!/usr/bin/env python from numpy import array

f2py: binding Fortran and Python

Wed, 23 Sep 2009 22:56:08 +0000

I have recently started using f2py to call Fortran from Python. I have found this useful for two reasons: speeding up Python scripts by calling compiled Fortran code, and using Python as a unit testing framework for Fortran modules. Unfortunately, the documentation for f2py is rather sparse, and may not be completely up to date. In this note, I will hopefully prevent you from wasting a lot of time figuring out how to pass array arguments, and return array results.

Profiling memory usage of Python code

Fri, 17 Apr 2009 22:08:52 +0000

In a previous post, I explained how to use the Python profiler. The profile is great for finding out which parts of the code run the slowest, or are called most often. However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed. If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude. On the positive side, your code may run much faster if it fits entirely in the processor cache. In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

Lookup tables and spline fitting in Python

Wed, 08 Apr 2009 23:36:09 +0000

Lookup tables and spline fitting are widely used by scientific programmers. A particular function may not have an analytic solution–in other words, it can’t be expressed as an equation of elementary functions. This might happen if the function were empirically determined from experimental data, or if the equation can’t be algebraically solved for one variable. If the analytical function is available, but takes a long time to evaluate, a lookup table or spline approximation can be considerably faster. In a previous post, I showed how to use the function interp1d from scipy.interpolate as a lookup table. In a later post, I showed that interp1d is actually rather slow, and scipy.interpolate.UnivariateSpline is much faster. Now, I will show some benchmark results, and explain a potential pitfall when using UnivariateSpline. The two functions I will use for this demonstration are not very complicated. Each function is a piecewise approximation of a more complex function. Here is one of the functions. For small z, the function uses one approximation, and for large z, it uses a far-field approximation. For intermediate values, a polynomial is used to smoothly match the two solutions.

3D Plotting Software for Python::Part 1::PyX

Fri, 20 Mar 2009 23:08:55 +0000

There are lots of good open-source tools that you can use to make high-resolution, publication-quality 2D plots. Personally, I like to use Python, numpy, and matplotlib. Unfortunately, it is much harder to find a good tools to make 3D plots. Older versions of matplotlib had rudimentary 3D support, but this was removed in version 0.98. In this post, I will review a Python 3D plotting library called PyX.

Preparing the data

Figuring out how to store the data to be plotted was actually the hardest part of learning to use PyX. The data format for 3D plots is not well documented. PyX requires a list of (x,y,z) lists like this:

The Python configparser: a way to read simple data files

Tue, 03 Feb 2009 23:42:21 +0000

My simulation library, which is written mostly in Python, needs a lot of data and parameters in order to run. In some cases, I just hard-code the values in the script that calls the library, and in other cases I load a pickle file containing a Python objext. What if I want to read in data or configuration parameters from a human-editable text file? If the information is extensive or complex, XML might be a good choice, but XML is overkill for simple configuration or data files. Fortunately, a standard Python library called ConfigParser has already defined a configuration file format, and provides methods to interact with such files. Here is a sample of the format used in a config file:

How to put formatted, highlighted code in a Wordpress post

Sat, 17 Jan 2009 21:49:36 +0000

I found two complementary plugins that enable me to put highlighted formatted code in a Wordpress page or post. Here’s an example of what they do:

[sourcecode language="python"]
# Plot flux at continuum boundary
pylab.figure()
pylab.hold(True)
pylab.plot(nd_times, Jl_BD, 'b-', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD+Jl_BD_std, 'b.', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD-Jl_BD_std, 'b.', label="Flux from BD simulation")
[/sourcecode]

Test successful! Note that Javascript must be enabled to see the highlighting. Get the plugins here:

Deploying Python applications on Windows

Thu, 15 Jan 2009 16:38:56 +0000

Writing applications in Python on a Linux system is almost too easy. Deploying Python apps on other Linux systems is not hard, because most Linux systems already have Python, with its core libraries and tools, installed. Most Linux systems also have package managers that make it easy to find and install required components. But, what happens when your co-workers who use Windows need to use your app? When you tell them to “go to the command line and…” you’ve pretty much lost them at “command line.” How do you package Python in a way that’s easy for a Windows user to install? Here is a process that worked for me:

Optimizing Python code for fast math

Fri, 09 Jan 2009 21:14:38 +0000

I spent some time today profiling a Brownian dynamics simulation written in Python to see how I could make it faster before starting some long runs on a Linux cluster. In the sections below, I have attempted to quantify the speed improvements due to various changes. Keep in mind that the actual speed improvement in your code will vary, depending on where the actual bottlenecks occur. See my post about profiling Python code. Another caveat: I am running Python 2.4.4 because it’s installed on our cluster.

Redirecting standard output from Python: another example

Wed, 07 Jan 2009 21:54:10 +0000

I wrote a previous post about how to redirect standard output from a Python script to a GUI window. In this post, I will give an even simpler example of to redirect standard output to a log file. During the early development and debugging of Python programs, I use print statements to keep me informed of what’s happening. However, printing to the terminal is not always practical–for example, when I run numerical code on a parallel cluster, there is no way to determine which output came from which instance of the program. Here is a class that you can use to redirect standard output to a log file:

Tools for Python software development

Tue, 06 Jan 2009 18:52:45 +0000

I have found a few tools over the years that I find extremely useful for developing software. Python is my language of choice at the moment, but I’m sure these tools will be handy for any language.

Subversion is an open-source version-control system. Version control was designed to allow multiple programmers to work on the same project at the same time without stepping all over one another. However, even though I am a solo developer, I find version control to be extremely helpful.
1. When I commit changes to the repository, I can document what I’ve changed and why. This is a great help when I introduce a bug and have to go back and find it.
2. The repository is stored on a remote server that is backed up nightly.
3. It’s easy to make an “unstable” branch for implementing new features. When I make changes that don’t work, it’s easy to revert to a previous version that works.
4. It is easy to deploy my code to the Linux cluster and make sure that the cluster is running the latest version of my software.

Go read the documentation on the Subversion web site to find out what it can do for you.RapidSVN is a GUI client for a Subversion server. By default, Subversion comes with a command-line client that does everything you need. However, sometimes it’s easier to stay organized when everything is presented visually. Here is a screenshot of RapidSVN:

RapidSVN screenshot

Using Python to generate XML files for visualization in Paraview

Thu, 13 Nov 2008 19:00:23 +0000

VTK is an open-source software system for “3D computer graphics, image processing, and visualization” developed by by Kitware. VTK is the foundation of Paraview, an industrial-strength CFD visualization tool that I have found to be very useful. I generate “second generation” XML-based files from my Python code and import them into Paraview for visualization. I am in the process of creating some Python classes to do, and I hope to publish them soon. Until then, I want to share some useful resources. The VTK file formats are specified in this document. It’s a pretty good specification, but it lacks some examples. Soon I will post an example of a valid unstructured, serial .vtu file. Each VTK file includes data from only one time step, so you have to keep track of time yourself (the filename is an easy solution). Paraview can read in data from multiple time steps, but you have to specify them in a .pvd file. This is also an XML file, with the following format: (reference)

Unexpected integer/float math behavior in Python

Fri, 07 Nov 2008 00:21:16 +0000

I wasted some time today tracking down a bug in one of my programs. It turned out to be “unexpected behavior” rather than a bug. I was aware of this aspect of the language, but I made an assumption and got bit. Read on for a valuable lesson. Python handles integer math differently than floating point math. If you type a number without a decimal point, Python treats it as an integer. All math performed only with integers results in integers.For example, 1/2 evaluates to 0 while 1./2. evaluates to 0.5. If you mix integers and floats, Python will produce a floating point result (1/2.=0.5), but you must be very careful. For example, you might expect the expression 4/3*3.14159 to yield a floating point result. It does yield a floating point number, butnotthe one you were expecting! 4/3*3.14159 yields 3.14159. What happened? Python works from left to right. 4/3 evaluates to the integer “1”. 1*3.14159 evaluates to 3.14159. For comparison, 4./3.*3.14159 evaluates to 4.1887866. Here’s the problem with this particular aspect of Python: according to the rules of math, 4/3*3.14159 is exactly the same expression as 4*3.14159/3, but in Python they yield different results if you forget the decimal points! 4*3.14159 evaluates to a floating point, so (4*3.14159)/3 yields the “correct” floating point value. Lesson Learned: be explicit about specifyingall floats if you are doing floating-point math! Sometimes I get lazy and leave a trailing decimal point off of a number when doing a floating point calculation, knowing that the results are “upcast” into floats. Not any more! Note: this unexpected behavior goes away in Python 3.0

Even faster collision detection in Python using Numpy

Thu, 16 Oct 2008 19:44:15 +0000

Last night, in the shower, I realized that my collision detection routine could be even faster. Here is a representative snippet of code from my previous post:

d2 = (x-self.x[0:i])*(x-self.x[0:i]) + (y-self.y[0:i])*(y-self.y[0:i]) + (z-self.z[0:i])*(z-self.z[0:i])

For some reason, I used the code (x-self.x)*(x-self.x) instead of (x-self.x)**2. Upon further reflection, I realized that (x-self.x)*(x-self.x) computes the difference between array elements twice, and then multiplies the results. Using a “power function” should enable the interpreter to compute the difference only once, and then multiply each element times itself. Here is the updated code, using Python’s power operator:

Speeding up Python math with Numpy: collision detection example

Sun, 12 Oct 2008 18:10:20 +0000

Python is a very-high-level language. That makes it easy to write code quickly, but the program may not be as fast as a program compiled from a lower-level language. For this reason, many scientific programs are written in Fortran or C++. However, it has always been my experience that the majority of time on a project is spent in writing, modifiying, and debugging code, rather than executing. Fortunately, if written correctly, the time-critical parts of Python code can execute almost as fast as compiled software. Here is an example of a collision-detection algorithm which achieved almost a ten-fold increase in speed when written to use Numpy.

Python Pickle: Painless binary storage for Python objects

Mon, 15 Sep 2008 22:23:18 +0000

The pickle module provided with Python is so useful that I’m surprised I haven’t used it before. Pickle allows you to save an entire data structure (such as an object) to disk as a binary file in a effortless (and fairly efficient) manner. For example, in my latest project I have created a Monte Carlo simulation that can take quite a bit of time to run. I also need to make multiple runs to get statistics on the results. At the end of each run, I need to dump the resulting data to disk so that it can be read in later by an analysis program. If I had to write data in a format that could be interchanged with other scientific software, I’d use the hdf5 format with the pytables package. However, right now I just need to get something working, and the pickle module is perfect. Here is how I save an object called box:

Python threads are easy (with example)

Tue, 19 Aug 2008 22:50:07 +0000

It’s remarkably easy to spawn a Python thread. However, before doing so, I caution you that a Python thread is not the same thing as an OS thread. Python threads run within the Python interpreter, but the Python interpreter always executes in a single process. The reasons why have already been explained elsewhere, so I refer you to the thread module documentation to learn about the Global Interpreter Lock. You probably have objections to this state of affairs, and I assure you they have already been voiced by Juergen Brendel and responded to by Guido van Rossum (creator of Python). Anyway, the upshot is that Python can only utilize one core of a multi-core CPU. This isn’t such a big deal for me because I’m a scientific programmer, and if I really need to write parallel code it’s going to have to run on a cluster or a grid. Threads don’t help with that. Having said all that, threads in Python are still useful. I will detail one example in which I spawn a thread to load a large binary file. While this doesn’t spread the work across multiple CPU cores, it does enable the GUI to remain interactive while the file loads. All you have to do to create a Python thread is create a class that is derived from Thread. In the example below, I derived a class called Loader, which “wraps” the function that actually reads the binary files. The __init__ method accepts the filename and other options as arguments. The run() method is required. Don’t call run() directly–instead, call the start() method (inherited from the base class) to start the thread.

Fun with threads in Python and wxPython

Mon, 11 Aug 2008 22:12:07 +0000

I have finally gotten back to programming in the last couple of days. Our project has finally started to generate a lot of data, so I’ve been refactoring and improving my code that reads data stored in LabView binaries. Today I spent a lot of time creating a GUI for browsing data. Arguably, this wasn’t the best use of my time, but I learned a lot about multi-threaded Python GUI programming with wxPython. You can find a gold mine of information on the multi-threaded wx programming at the wxPython wiki. Because the LabView binary data has to be read sequentially, and the files are rather large, it takes a long time to read in a file. I spawn a thread to handle the file reading, while allowing the GUI to remain responsive. The thread posts messages to the GUI window, which are used to update the user on the status of the file reading operation. When the file is read, a final message containing the data is posted to the window. It’s really pretty slick now that I’ve figured out how to do it. I will soon post a clever scheme to capture text output from the file-reading function, and display it in the GUI, without making substantial changes to the file-reading function.

Reading Labview binary files with Python

Wed, 25 Jun 2008 16:04:46 +0000

My research group uses Labview 7.1 to write custom data acquisition (DAQ) software. I code everything else in Python, so I need to get data from Labview into Python for processing. Our DAQ program produces Labview binary files, so I had to find a way to read them with Python. Binary files are nice because they are a compact way to store numerical data as compared to ASCI or (heaven forbid) XML, but they are much harder to read. The binary format used by Labview is documented only indirectly, so I had to hack a little. The first thing to realize is that the Labview binary file is a direct dump of the data that was stored in RAM. How Labview stores data in memory is documented here. Indirectly, this documents how binary files are stored on disk. Our DAQ program writes a rather complex “cluster” (Labview’s version of a C structure) to disk. The elements of the cluster are stored contiguously as a sequence of bytes, and there’s no way to know which byte goes with which element, unless you know the size of each element and the order in which they are stored in the cluster. So, the first step is to document the cluster that’s being written to disk. You can use the context help in Labview to view the data type of the wire that leads to the VI that writes the file. With this in hand, you are ready to write Python code. First, make sure you open the file in binary mode:

Server move completed and general update

Sun, 22 Jun 2008 19:37:44 +0000

The site is back online after a server move. Actually I can’t blame the server move for the downtime, because I went out of town and then had a lot of catching up to do, and as a result I didn’t switch the domain to point to the new server. I haven’t been blogging much because I haven’t done much software development, Linux admin, or lighting design lately. I have been busy in the lab at work and I’ve been supervising a trainee lighting operator instead of running my own shows. We haven’t been moving forward with plans to purchase a new lighting control console, so no update on that, either. Hopefully, I’ll have something interesting to post soon.

Presentation at BarCamp Orlando 2008

Mon, 07 Apr 2008 03:14:32 +0000

I gave a presentation at BarCamp Orlando on 5 April 2008. Here is a link to my presentation, Data Analysis with Python.

My current Gentoo Linux setup

Mon, 12 Nov 2007 22:15:03 +0000

This post is a reminder to myself more than it is intended for anyone else. Well, that’s most of this blog. Gentoo is a great distribution of Linux, but you have to install everything yourself. This is a good thing, because you get a very lean system, but the downside is that you are re-inventing the wheel somewhat with every install. So, without further ado, here is my current favorite configuration: Global USE flags:X cups dbus hal jpeg png gs dvi gif tiff mpeg bzip2 gtk lapack tetex -gnome -kdePackage list (basic software):* dbus (message bus system)* hal (Hardware Abstraction Layer) gentoolkit vim/gvim x.org xfce4 thunar mozilla-firefox acroreadPackage list (technical software): blas-atlas* lapack-reference numpy (unstable 0.90.1) matplotlib (Matlab-like plotting package for Python) texlive (experimental, hard-masked) lyx evince (dvi, ps, pdf viewer) paraview (3D visualization of datasets, unstable)

ODEPACK re-released

Wed, 26 Sep 2007 03:44:58 +0000

Minutes ago I uploaded a tarball containing my re-release of ODEPACK, a standard numerical tool for the solution of systems of ordinary differential equations. The algorithms and numerical code are identical to the distribution found on Netlib, but everything has been placed into an organized “package” with Makefiles, examples, documentation, etc. The code is set up to be compiled as a shared library that you can link to your code. Download and enjoy! And tell me if you’re using it!

Direct access to C++ containers from Python

Thu, 13 Sep 2007 16:31:01 +0000

In previous examples, I’ve shown how to pass Python lists into C++ using Boost.python. Because Python lists can contain a mixture of objects of different types, C++ has to use extract<type> to determine what kind of data to get from a list item. This approach works well, but it has its faults. In the C++ code, you might need to have duplicate data structures (a boost.python list to interchange data with Python and a C++ container such as a vector to process the data) and the code to convert between them. If you want to use your C++ library with languages other than Python, you want a non-Python-specific interface. Wouldn’t it be great to define your core data structures using a standard C++ container (like vector) and then access and modify them in Python? You can, but it’s non-obvious and it’s not well explained in the Boost.python docs. Read the docs for the indexing suite and take a look at my demonstration code for exposing C++ vectors to Python, and you should be able to figure it out. Once you get it working, it’s a very clean way of doing things. Just remember, because you’re working with C++ containers that accept only the specified type of object Python will throw a TypeError if you try to append the wrong kind of object.

Mixing objects of different types in a Boost.python list

Wed, 08 Aug 2007 15:35:26 +0000

Demonstration Code–mixing objects of different types in a Boost.python list In Python, a list is allowed to contain objects of various types. Using Python’s introspection capabilities, it’s easy to process a mixed list because you can just test each list member to see what kind it is, using isinstance(objectA, TypeB). This creates a problem when the list is passed into C++, where arrays and containers are designed to hold objects of one type. I encountered this problem when writing a small module to parse mathematical expressions. The expression is entered as a string in Python, which is parsed and “compiled” into a stack of objects. The compiled stack is passed to C++ using boost.python, where it is processed when needed. The stack consists of two or more fundamentally different objects: numerical constants and operators. The evaluation routine pops an object. If it’s a numerical constant, the value is placed on the operand stack. If it’s an operator, the appropriate operation is performed on the values at the top of the operand stack. With help from Alex Mohr and Christopher Woods on the C++-sig mailing list, I developed a method that is illustrated in the example. In C++, define a polymorphic base class (has at least one virtual method) and several classes that are derived from the base class. These classes are exposed to Python using Boost.python so that objects can be created in Python. A list of mixed objects is then passed back to C++ for processing. When trying to access a member of the list, C++ doesn’t know what kind of object to expect, so we have to use the Boost.python function extract<>() (see previous post for an introduction to extract). Now we extract a pointer to the base class using: object = extract <BaseClass*> (list[index]); This works because every object in the list is derived from the base class. As shown in the example, I first tried to do this with references (to avoid the whole pointer mess) but it did not work, so it seems that pointers are unavoidable here. It seems that this use of introspection is relatively rare in OOP, and is sometimes actively discouraged. However, for this type of problem it seems like the most logical design. Is there a different design pattern that would work better?

USPTO tries peer review for software patents

Wed, 04 Jul 2007 21:55:06 +0000

The US Patent and Trademark Office (USPTO) has launched a trial program to subject software patents to peer review online. Naturally, the big software companies aren’t exactly publicizing this. We all know that a lot of stupid patents get issued. Well, stop complaining and start reviewing! Let’s make this trial successful!

Example: using an extractor class in Boost.Python

Mon, 16 Apr 2007 03:12:23 +0000

The documentation for the Boost.Python extractor class is not terribly enlightening for a new user. The examples given in the tutorial are code fragments, and it’s never clear exactly what the context is or what they are supposed to accomplish. That’s a shame, because the extractor classes are really cool and easy to use, so people should use them! Here is a Boost Python extractor example that should make things clear.

"Ubuntu" actually means "eats your config files"

Wed, 14 Mar 2007 20:08:52 +0000

The Dapper-to-Edgy upgrade was surprisingly smooth, once I figured out how to do it. Unfortunately, the new version overwrote my “/boot/grub/menu.lst” so I could not boot Windows. I have enough experience with Linux that this was a five-minute fix, but it could be pretty scary for an inexperienced person. Who knows what else got over-written and I just didn’t realize it. I was further disappointed to find out that the version of PIDA in Edgy is broken.

Ubuntu is an African word that means "devours your soul"

Mon, 12 Mar 2007 22:57:45 +0000

When it comes to Linux, I’m a Gentoo guy. Go ahead, make fun of me and call me a ricer. Then go try to do something difficult on Redhat or Ubuntu, and come back and tell me what you think. I needed a Linux development environment at work, so I took my Windows machine and added a Linux partition. Since Ubuntu has been getting a lot of good press, and I wanted something that would install fast, I tried Ubuntu. Specifically, I installed Kubuntu, because Linus Torvalds doesn’t use Gnome. It really did install quickly and everything worked great out of the box. I installed a number of packages, and they worked too! Too good to be true? Absolutely! It all started when I wanted to compile something. Silly me, I thought gcc was a basic part of a basic Linux installation…but no, that must be a Gentoo bias, so I installed gcc. Then I decided to install numpy, the latest numerical toolkit for Python. It’s been around for a couple of years as a beta, so I figured there must be an “unstable” build out there. Nope. To get numpy, you have to upgrade from Ubuntu 6.06 “Dapper” to 6.10 “Edgy”. <sarcasm>Because it makes sense to upgrade gcc, KDE, Firefox, etc just to get a new Python package.</sarcasm> Now, go find instructions on how to upgrade the Ubuntu distribution. Then try those instructrions and realize that they won’t work at all for Kubuntu because the package managers are totally different. While you’re there, read the horror stories on the forums from people who followed the “official” upgrade instructions and ended up with less-than-functioning systems. FYI, the Kubuntu upgrade instructions are actually easy to find once you know where to look. Kubuntu 6.10 is installing right now, so tomorrow we’ll see how it went.

Python calling C++ calling Fortran

Fri, 09 Mar 2007 04:40:03 +0000

I got it working a while ago, but I haven’t had a chance to update my eager audience. I built a simple simple simulation tool using this crazy combination of languages. It’s a library written in C++ that uses the finite volume method to simulate diffusion in one dimenstion. What’s cool is that it’s designed to be called from Python using Boost. Furthermore, it then uses Boost Bindings (not officially part of Boost) to call LAPACK (a Fortran library) to solve linear equations (as shown in a previous post). It’s pretty cool that all this stuff actually works together. Why did I do this? Python is my favorite very-high-level language for doing scientific programming. Compared to C++ or Fortran, it’s just so easy to do things in Python, like making plots or reading in an xml file. I chose C++ for the numerical codes because it’s faster than Python and the Boost library connects the two. I could have used Fortran, but the Python interfaces doesn’t seem to be as clean, probably because Fortran isn’t object-oriented. Finally, since the reference implementation of LAPACK is still in Fortran, I just used the Boost bindings to call it. I might switch to a native C++ linear algebra library if a clear leader emerges from the confusing alternatives that are available today. More complaints on that later…

Subtle aspects of using Boost and LAPACK

Tue, 27 Feb 2007 23:43:59 +0000

I spent part of the afternoon wrestling with LAPACK integration in a C++ program, using the unofficial Boost bindings. I learned a couple of interesting things. test.C is a file that demonstrates them. 1. You need to create column-major matrices to pass to LAPACK. The subtle part about this is that the matrices are internally stored as column-major (like Fortran) but are accessed like row-major matrices (like C). In other words, even when you declare A as column-major, A(3,5) refers to the third row and fifth column of A. This was not obvious to me. 2. The LAPACK routines may modify the matrices that are passed to them. For example, when using gesv(A,B) to solve a system of linear equations, the matrix B is modified to contain the solution values–this is obvious. It is not obvious, at first, that the matrix A is also modified (and I don’t know what useful information it contains). 3. There does not seem to be any binding to the specialized LAPACK solvers for banded and tri-diagonal matrices. I would like to know why, since my matrix is tri-diagonal and it can probably be solved considerable more quickly.

The need for 3D technical drawing software

Mon, 26 Feb 2007 21:42:17 +0000

Today I was reminded again of the need for a new kind of technical drawing software. Here is the problem I am facing: I want to create a vector drawing of a three dimensional structure. Right now, I can either create it in CAD and export a bitmap, or I can manually draw a 3-D representation using 2-D vector software such as Dia or Xfig. For publication-quality graphics, the first approach is usually out. The second approach works, but what if I spend an hour drawing, only to realize that I should have chosen a different perspective or viewpoint to show a critical feature of the drawing? I have to modify the entire drawing just to change the viewpoint. Or, what if I need to show two views to capture all the details? I have to create both from scratch. I envion a new type of tool, or CATD (Computer Aided Technical Drawing). My vision starts with a 3-D CAD tool, which could be built around lines, points, and surfaces (like AutoCAD) or a solid modeling kernel (like SolidWorks). The user should have the ability to choose surfaces to be transparent or solid, and create cut planes to show internal features. However, the 3-D model is only the first step. Once the user has created the model and chosen a viewpoint that shows its important features, the 3-D model is projected onto a 2D surface to create a vector drawing that can be further manipulated. Ideally, the 2-D vectors created from each “object” in the CAD model would remain grouped together for easy manipulation. The final output would be a publication-quality PostScript, PDF, or SVG file. I don’t know if this is commercially available, but it certainly isn’t available as open source. As a starting point, I propose creating a plugin for Blender that fulfills this purpose. It would take the 3-D model, as displayed on the screen, and create a vector 2-D representation that can be imported into other software for manipulation. Ultimately, the tool should allow users to import different types of CAD models. The text should be somehow integrated with LaTeX for seamless typesetting of equations and numbers.

LISP is cool but annoying

Mon, 26 Feb 2007 05:09:14 +0000

I spent today organizing my personal finances in GnuCash. I also spent a few hours customizing a report, which unfortunately means hacking in Scheme, a “dialect” of LISP. Don’t get me wrong, I actually like LISP/Scheme quite a bit. What I don’t like is writing code in “normal” languages, like C++ and Python, all week long and then switching to Scheme. Scheme has a totally different syntax and requires a different way of thinking about a problem. Normal language: result = myfunction (argument1, argument2) Scheme code: (myfunction (argument1 argument2)) If you have never seen a LISP language, find one and check it out. Here are the links I always use: