shocksolution.com

Accessing a Mulesoft API mock with curl

Fri, 23 May 2025 20:40:15 +0000

Using curl to access a Mulesoft API mock requires a little trick. Mulesoft gives you the ability to test out a mock of your API design. The easiest way to try the API is to click on the Documentation button on the right side of the API Designer, select an endpoint, and click “Try it.” If you have made the mock public, you can also use curl to test, but there is a little trick to using curl.

Contact Craig

Mon, 05 Sep 2022 03:39:21 +0000

Google Apps Script: Spreadsheet to Document

Thu, 19 May 2022 02:38:02 +0000

For an explanation of this script, see Jeffrey Everhart’s excellent introduction to Apps Script.

function onOpen() {
 const ui = SpreadsheetApp.getUi();
 const menu = ui.createMenu('AutoFill Docs');
 menu.addItem('Create New Docs', 'createNewGoogleDocs')
 menu.addToUi();

}

function createNewGoogleDocs() {
 //This value should be the id of your document template that we created in the last step
 const googleDocTemplate = DriveApp.getFileById('1It9mTOqk0dVyPVaxmjRXa5tXdWMzgsaDxvb4Nm11nsM');

 //This value should be the id of the folder where you want your completed documents stored
 const destinationFolder = DriveApp.getFolderById('1jeg1V6q1ZJ6OWz-A4LNS5f8GtE9x4SQ4')
 //Here we store the sheet as a variable
 const sheet = SpreadsheetApp
 .getActiveSpreadsheet()
 .getSheetByName('Data')

 //Now we get all of the values as a 2D array
 const rows = sheet.getDataRange().getValues();

 //Start processing each spreadsheet row
 rows.forEach(function(row, index){
 //Here we check if this row is the headers, if so we skip it
 if (index === 0) return;
 //Here we check if a document has already been generated by looking at 'Document Link', if so we skip it
 if (row[5]) return;
 //Using the row data in a template literal, we make a copy of our template document in our destinationFolder
 const copy = googleDocTemplate.makeCopy(`${row[1]}, ${row[0]} Employee Details` , destinationFolder)
 //Once we have the copy, we then open it using the DocumentApp
 const doc = DocumentApp.openById(copy.getId())
 //All of the content lives in the body, so we get that for editing
 const body = doc.getBody();
 //In this line we do some friendly date formatting, that may or may not work for you locale
 const friendlyDate = new Date(row[3]).toLocaleDateString();

 //In these lines, we replace our replacement tokens with values from our spreadsheet row
 body.replaceText('{{First Name}}', row[0]);
 body.replaceText('{{Last Name}}', row[1]);
 body.replaceText('{{Hours}}', row[2]);
 body.replaceText('{{Date Completed}}', friendlyDate);

 //We make our changes permanent by saving and closing the document
 doc.saveAndClose();
 //Store the url of our new document in a variable
 const url = doc.getUrl();
 //Write that value back to the 'Document Link' column in the spreadsheet. 
 sheet.getRange(index + 1, 6).setValue(url)

 })

}

Google Cloud Build for Cloud Functions

Fri, 29 Apr 2022 18:57:01 +0000

This post aims to add some missing information to Google’s docs about setting up CI/CD for Cloud Functions with Cloud Build.

Global vs Regional

Cloud Build triggers are globalby default. You can create a regional trigger in the Cloud Console by changing the location when editing or creating a trigger.NOTE: I can’t find any way at all to create a regional trigger with Terraform.

However, functions must be regional. The Google docs indicate that you must specify a region in your cloud build config file. If you’re using a global trigger, the `$LOCATION` substitution variable is set to global, and you’ll get a permission error when trying to deploy to a regional function. There are two options that I know of, and neither is great:

Terraform: for_each on a list of resources

Thu, 10 Mar 2022 18:25:02 +0000

Terraform provides a very simply way to use for_each to iterate over a list of resources. If you have a list of strings, use the toset() function to convert the list to a set of strings.

Example: assign a unique role on each resource

My use case is setting up a number of dev environments in Google Cloud Platform. The number may change as the size of the dev team increases, so I don’t want to hard-code the number of resources anywhere in my Terraform code. The number of environments is stored in the num_envs variable. For this example, I want to create a group of resources (Google Cloud Storage buckets). Each environment has its own service account, and I want each environment’s service account to have a specific role on that environment’s bucket. I also want to leverage official Google-supported Terraform modules whenever possible.

Using SSL Certificates with the Apache Tomcat Web Server

Tue, 30 Nov 2021 03:20:26 +0000

Creating PKCS12 Files

PKCS #12 is a format for storing multiple cryptography objects in a single archive file. You can store arbitrarily complex objects within a PKCS #12 archive, but the most common use is to store a single private key and its certificate chain. Create a PKCS12 file from PEM files:

openssl pkcs12 -export -in ssl_cert.pem -inkey key.pem -certfile bundle.crt -name "*.example.com" -out example.com.p12

Depending on the product you’re working with, the documentation may call for a .pfx file instead of a .p12 file. PFX is an older format that was a predecessor to PKCS #12. In most modern systems, A PFX FILE AND A P12 FILE ARE EXACTLY THE SAME THING! You can just change the extension if needed and that will generally work. For example, the docs for ESET Security Management Center call for a pfx file, but a PKCS12 file will work just fine.

Auto-Create Multiple Blocks in a Terraform Resource

Fri, 12 Nov 2021 16:35:50 +0000

Learn how to automatically create multiple Terraform resources, and multiple blocks within one Terraform resource, using the for_each meta-argument and dynamic blocks.

Creating Multiple Resources

When using Terraform in a realistic environment (e.g. not a lab or tutorial), you often need to create automatically create multiple resources based on a list or map of data. This article reviews the options for creating resources based on data.There are two general methods to choose from when creating multiple resources:

The Best Mental Model for Writing Terraform Code

Tue, 19 Oct 2021 03:48:29 +0000

This article explains my current mental model of Terraform, in the hope that it will save you some time in your learning process. The foundation of using any programming language or software tool correctly is to develop a valid mental model for it, and refine your model as you learn more. There isn’t one correct mental model, and your mental model must evolve as your understanding grows.

Your introduction to writing Terraform code is usually through a very simple example. Unfortunately, simple examples can obscure some of the fundamental concepts of the language. For example, when I first started with Terraform, I did not understand the difference between a variable and a local. Variables need to be declared in a .tf file and defined (assigned a value) in a .tfstate file (or through other means). Variables seem complicated compared to locals. You can just assign a variable to a local and start using it, like a variable in Python. Why use variables when locals seem so much easier?

Define a Google Load Balancer and Cloud Storage bucket with Terraform

Thu, 30 Sep 2021 18:11:54 +0000

Here’s an example of using Terraform to define resources to host static content in a Google Cloud Storage bucket, fronted by a Cloud Load Balancer with a custom URL and SSL certificate. This example uses some other advanced features, such as Google Secrets and a map variable to define the SSL certificates. It’s pulled from a larger project, so this block of code isn’t guaranteed to run as-is. At a minimum, you’ll need to define the variables and set values.

How to Enable Multi Factor Authentication (MFA) on your IEEE Email Account

Wed, 08 Sep 2021 04:03:20 +0000

If you’ve been around IEEE for a long time, you remember when your email address ____@ieee.org was just an alias that forwarded all email to your actual email provider. However, in 2013 the IEEE partnered with Google to create an actual Gmail account for your IEEE email address. While the intent behind this benefit is certainly appreciated, it was actually an inconvenience for many of us who were already established with an email provider. We didn’t want to migrate our entire digital live to IEEE’s new Gmail account. I wrote a popular blog post back in 2013 about how to configure your IEEE Gmail account to forward 100% of all email to your actual email provider.

Multi-Page Blueprints for Confluence

Wed, 05 May 2021 20:37:05 +0000

A free demo app is available in the Atlassian Marketplace which supports multi-page blueprints for Confluence. This app works around numerous bugs in the official Atlassian blueprint tutorials and a key Atlassian API bug.

Background on Confluence

Confluence is one of the leading enterprise wiki products. Its built-in feature set is already very powerful, and it can be extended with applications available in the Atlassian Marketplace. You can also write your own applications for private, internal use. Atlassian provides the atlassian-connect-express toolkit for building apps with Node.js, which reduces development time and effort. They also provide a number of sample projects for Jira and Confluence apps on Bitbucket. Unfortunately, many of these examples are obsolete or deprecated, and do not function correctly with the latest version of atlassian-connect-express or the latest release of Confluence and Jira.

Node.js: Connecting to MySQL on a socket with Sequelize

Thu, 22 Apr 2021 18:54:25 +0000

According to its official description, “Sequelize is a promise-based Node.js ORM for Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server.” Sequelize is widely used in Node.js web applications to abstract the database layer. I recently found a “corner case” for Sequelize which is possible to accomplish, but is not well documented anywhere on the web. I need to use Sequelize to connect to a MySQL server via a UNIX socket. I’m developing some plugins for Confluence that use the atlassian-connect-express toolkit, which has a datastore that wraps Sequelize. The way Atlassian has chosen to wrap Sequelize is rather unfortunate, probably because they are trying to maintain backwards compatibility with JugglingDB. Essentially, the Atlassian Connect Express only looks at the URL, and ignores any option passed to Sequelize. Therefore, you have to pass everything you need via the URL, and this is where it’s tricky. Here’s the form of the URL that you need to connect to MySQL via a UNIX socket with Sequelize:

Enable Port Statistics on a Netgear M4200/M4300 Switch

Mon, 18 Jan 2021 20:12:08 +0000

You can enable detailed port statistics via the command-line interface (CLI) on switches in the Netgear M4200/M4300 family. I will explain how to set up a time range and port group, and apply them to a port or range of ports to gather throughput data. This post will make up for some shortcomings in Netgear’s CLI documentation about enabling and gathering switching statistics.

NOTE: you must enter commands enable and configure to enter administrative configuration mode prior to setting a time range or port statistics.

Use an SSL/TLS Certificate with ESET Security Management Center Appliance

Wed, 11 Nov 2020 21:19:29 +0000

ESET Security Management Center (ESMC) is the replacement for the ESET Remote Administrator (ERA) Server. You can install ESMC as a “virtual appliance” which will run on hosts such as VMWare or Linux/KVM/QEMU. Under the hood, this appliance is a CentOS 7 server running the Apache Tomcat web server. ESMC is implemented as a Java application that runs as a systemd service. In my experience, ESET is a solid security product, and the free ERA/ESMC provides some rudimentary endpoint management capabilities. Unfortunately, everything about ESET is confusing, from the naming of their products to their documentation. The documentation is thorough…it’s just hard to find what you’re looking for, which is usually in multiple overlapping documents that each contain part of the puzzle.

Alert Policies for Log Metrics on Google Stackdriver Monitoring

Tue, 08 Sep 2020 16:31:08 +0000

Google Cloud Operations, formerly known as Stackdriver Logging and Monitoring, can be very confusing to set up. It’s easy to monitor something simple, but more complex cases quickly get confusing. One of the more flexible but confusing types of alert policies in Stackdriver Monitoring is a Logs-Based Metrics policy, which gives you the ability to monitor entries in Stackdriver Logging.

Create a Logs-Based Metric

The first step is to go to Stackdriver (Operations) Logging and create a user-defined metric. Go to the Logs Viewer and build a query to return the subset of log entries that you want to monitor (the details of query building are beyond the scope of this article). As of Sept. 2020, I highly recommend enabling the Preview Mode of the Logs Viewer interface, which is much better at guiding you through the process of building the query. If you must use the Classic interface, click the little black arrow at the right side of the filter box, near the top of the Logs Viewer screen. Select “Convert to advanced filter” from the drop-down menu, and create an advanced filter using the query language. Once you have created the right filter (query) that selects the log entries that you want to monitor, proceed to the next step:

Testing Access to Google Bigtable

Fri, 05 Jun 2020 20:14:15 +0000

This is a minimal PHP script that’s useful for verifying connectivity and permissions to read data from a Google Bigtable instance. Unlike some of the sparse examples in Google’s PHP SDK, this script will work for any table, and you don’t need to know anything about the structure of the table. This script tests the critical elements that need to work together to enable Bigtable access:

Credential file GRPC extension for PHP Protobuf extension for PHP Scopes enabled on Compute Engine instances

Dynamic node groups with Salt

Mon, 13 Apr 2020 12:53:00 +0000

Salt is a great choice for a configuration management system. I have found that it scales well, both in terms of number of nodes, and complexity of configurations. However, I have found one aspect of Salt that remains under-developed: Salt doesn’t have great support for custom groupings of minions. Minion groups (which could also be called node groups, host groups, instance groups, etc.) allow you to target a predefined group of minions.

How to use SSL/TLS certificates on a Netgear M4200/4300 switch

Fri, 03 Apr 2020 03:50:18 +0000

If you operate a Netgear switch in an environment that is subject to compliance requirements such as PCI or HIPAA, you are probably doing vulnerability scans, and the HTTP management interface of your switches will generate medium vulnerabilities (at least from Nessus):

SSL Self-Signed CertificateSSL Certificate Cannot Be Trusted* SSL Certificate Expiry

It IS possible to use signed certificates to secure the HTTPS interface on a Netgear switch, but the process to do so is insanely obscure. I do not know how someone had time to figure this out! Apparently, even Netgear tech support didn’t have any idea.

curl or libcurl: SSL certificate problem: unable to get local issuer certificate

Tue, 10 Mar 2020 17:00:17 +0000

curl, or an application that uses libcurl, may have a problem with an SSL certificate that works fine when using a web browser to access the same URL. Typical error output from curl looks like this:

$ curl -v https://my-subdomain.mysecuresite.com

 Trying xxx.xxx.xxx.xxx:443…
 TCP_NODELAY set
 Connected to my-subdomain.mysecuresite.com (xxx.xxx.xxx.xxx) port 443 (#0)
 ALPN, offering h2
 ALPN, offering http/1.1
 successfully set certificate verify locations:
 CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: none
 TLSv1.3 (OUT), TLS handshake, Client hello (1):
 TLSv1.3 (IN), TLS handshake, Server hello (2):
 TLSv1.2 (IN), TLS handshake, Certificate (11):
 TLSv1.2 (OUT), TLS alert, unknown CA (560):
 SSL certificate problem: unable to get local issuer certificate
 Closing connection 0
 curl: (60) SSL certificate problem: unable to get local issuer certificate
 More details here: https://curl.haxx.se/docs/sslcerts.html 
 curl failed to verify the legitimacy of the server and therefore could not
 establish a secure connection to it. To learn more about this situation and
 how to fix it, please visit the web page mentioned above.

Troubleshooting Strategy

A good starting point for any SSL error on a public-facing URL is to analyze the URL at SSL Labs.
Does the error happen for all certificates issued by a specific Certificate Authority (CA)? If so, the system running curl may need to have a root certificate for that CA added or updated in its certificate repository. That’s a relatively rare problem, but might occur if the system running curl is very old. There’s a comprehensive thread about this issue on Stack Overflow.
If the error only happens for one specific site, it’s likely that the site is missing an intermediate certificate. The command to diagnose this issue is also found in that Stack Overflow thread:

openssl s_client -connect myhost.com:443 -servername myhost.com -showcerts

The output should show a series of certificates, starting with the site certificate, and ending with the root certificate for the Certification Authority. If this chain only shows the site certificate, that’s the problem.

2020 Ubiquiti Wireless Access Point (WAP) Roundup

Tue, 21 Jan 2020 18:05:28 +0000

Ubiquiti publishes detailed data sheets for all of their products, but it’s hard to find all the information you need in one place for comparison purposes. That’s the purpose of this article.

First Generation Access Points

It can be confusing to distinguish between the generations of UniFi access points; you have to look at the access point label, or access it via the UniFi software to see the version. The performance difference between generations are significant. The datasheet for first generation Ubiquiti UAP wireless access points is still available.

Monitoring a switch or router port via SNMP

Mon, 16 Dec 2019 20:19:52 +0000

Most business-class switches and routers support monitoring via SNMP. The main barrier to implementing SNMP monitoring is that so many parameters can be monitored, and it can be very hard to isolate the most important parameters. This example shows how to monitor the status of a specific port. Note that the term “port” includes link aggregation groups (LAGs) or other logical groups of ports like VLANs.

Use Case

A top-of-rack switch has several important link aggregation groups (LAGs). For example, one LAG has connections to two identical routers in a VRRP cluster. Other LAGs provide redundant network paths to independent NICs on the same server. I want to be alerted if the port status changes on any critical LAG. The ISP delivers Internet to a single port on the switch, which I also want to monitor.

openssl unable to read/load/import SSL private key from GoDaddy

Mon, 02 Dec 2019 13:24:00 +0000

openssl is the standard open-source, command-line tool for manipulating SSL/TLS certificates on Linux, MacOS, and other UNIX-like systems. I recently ran into an interesting problem using openssl to convert a private key obtained from GoDaddy. Someone else used GoDaddy’s “wizard” interface to generate a certificate signing request (CSR) and private key, and saved the files on their Windows workstation. They purchased an SSL cert from GoDaddy, and shared all the files with me for installation on servers. GoDaddy saved the private key in the newer PKCS #8 format (pkcs8), and one system required the key in the older PKCS #1 (pkcs1) format. It’s easy to tell the difference.

Nginx default user changed in RedHat/CentOS package nginx-1.16.1-1.el7

Tue, 12 Nov 2019 02:58:34 +0000

The latest Nginx rpm (nginx-1.16.1-1.el7) from EPEL for CentOS/Redhat is a fairly major update, from version 1.12 to 1.16, and includes some very important fixes to address multiple CVEs. There is another small but critical change in the latest RPM that isn’t mentioned in the bug report. RedHat had configured nginx to run as user “apache” in past releases, but in the EPEL release on 19 October 2019, the configuration was changed to run as user “nginx”. If you ran “yum upgrade” recently (which you really need to do on a regular basis) on any system with nginx installed, you probably have a system that is affected.

mount.cifs error 13 after update (CentOS/RedHat Linux 7.6)

Wed, 23 Oct 2019 16:23:48 +0000

PERMISSION DENIED

PERMISSION DENIED (Photo by Hidde van Esch on Unsplash)

If you recently updated your RedHat or CentOS 7.6 system, you may suddenly start getting “Permission Denied” errors when attempting to mount SMB shares via CIFS. Typical error messages in syslog look like this:

kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE
kernel: CIFS VFS: Send error in SessSetup = -13
kernel: CIFS VFS: cifs_mount failed w/return code = -13

Configuration that triggered the problem

Synology NAS with latest operating system shares a volume via SMBCentOS 7 Linux server mounts SMB share using a local username and password (NOT domain credentials)cifs-utils version 6.2 installed in May of 2019 (this by itself worked fine)libsmbclient just updated to 4.9.1-6libmount just updated to 2.23.2-61CIFS mount options: vers=3.0,credentials=/root/credentials.txt,sec=ntlmsspi* File /root/credentials.txt contained a username and password that are LOCAL to the SMB server

Secure, Automated Network Device Backups

Tue, 03 Sep 2019 16:38:30 +0000

You probably have at least a few business-class switches and routers silently do their job, year after year. Network devices can be so reliable that we configure them once, and then forget that they exist. However, forgetting them is a really bad idea, since they require regular OS or firmware updates to patch vulnerabilities, and at some point, they will become unreliable and fail. If your device fails, do you have the configuration available to apply to its replacement? Are you sure the configuration that’s in your archive or documentation is up to date? This post deals with automating backups from Netgear switches and Ubiquiti EdgeRouters (and possibly switches, if they run EdgeOS). Please check out the accompanying repo at https://github.com/craigafinch/network-backup-scripts

Another Reason to Enable Private IP Addresses on Google Cloud SQL

Wed, 26 Jun 2019 18:16:22 +0000

Google Cloud SQL recently introduced the capability for an instance to have an IP address within the private address space of your VPC network. Previously, all Cloud SQL instances had IP addresses within the address space of the public Internet. Obviously, the public IP was a security and privacy concern, as well as a potential violation of some compliance and governance frameworks. Now, each instance may have a public IP, private IP, or both. There’s another potential benefit to using a private IP address: performance!

QNAP NAS: Backup & Sync to Google Cloud Storage

Sat, 01 Jun 2019 21:17:40 +0000

I have a QNAP NAS at home which I use for file storage. It shares several folders with the family’s Mac, Linux, and Windows computers. Many of these files are are irreplaceable, especially our digital photos, so I back them up to the cloud. QNAP provides the Hybrid Backup and Sync application, which supports multiple cloud storage endpoints. QNAP also provides specific documentation for backing up and syncing to Google Cloud Storage. However, QNAP’s instructions don’t explain how to set up a Google Cloud Storage bucket destination, so here is the procedure I used to set up Google Cloud Storage to back up my QNAP NAS.

Disk space utilization monitoring in Google StackDriver

Tue, 28 May 2019 18:22:43 +0000

Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).

File System Type Filter

Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.

Managing remote servers with salt-ssh

Mon, 13 May 2019 21:20:37 +0000

I love using Salt to configure and manage servers at scale. Typically, a Salt master server manages an army of servers which are on the same private network. A salt-minion daemon runs on each client server and communicates with the master. Sometimes you need to manage a server which is outside of your internal network, such as a marketing web server running on a service like AWS LightSail, Linode, or Digital Ocean. That’s when salt-ssh comes in handy.

Find Last Login Time for Wordpress Users in the SQL Database

Tue, 16 Apr 2019 16:49:21 +0000

Most answers say to install some plugin, and there are dozens that will show the date and time when each Wordpress user last logged in. However, plugins are a problem, because they always need to be kept up to date, and many plugins get abandoned. Now, you have a potential security problem. Also, this approach assumes that your Wordpress front-end is up and running; if your site has been hacked or accidentally broken, you need to go into the database to do some forensics.

MySQL replication fails with WSREP error after restart

Mon, 11 Mar 2019 12:16:20 +0000

Scenario

A single Percona MySQL 5.7 server replicates asynchronously with a cluster of Percona MySQL 5.7 servers. After rebooting the single server, replication failed with the following symptoms:

mysql> show slave status \G**************************** 1. row ************** Slave_IO_State: Waiting to reconnect after a failed registration on master Master_Host: my-replication-partner Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000120 Read_Master_Log_Pos: 3848512 Relay_Log_File: my-host-hame-relay-bin.000012 Relay_Log_Pos: 4 Relay_Master_Log_File: mysql-bin.000120 Slave_IO_Running: Connecting Slave_SQL_Running: Yes

I checked my MySQL text log (/var/log/mysql.log in my case, but the path may vary, or messages may go to syslog) and found the following error:

Updating to Wordpress 5 on a Linux Host: Requesting FTP Credentials

Mon, 14 Jan 2019 17:10:43 +0000

Wordpress 5 didn’t install automatically on sites hosted on my CentOS 7 Wordpress hosting server. That’s by design, because I’m selfish and I’d rather wait for a few bug-fix releases before making a major upgrade. When I chose to upgrade manually on a test site, Wordpress asked for my FTP credentials:

Connection Information: FTP credentials

If you’re hosting Wordpress on your own server, it does not need FTP credentials. This page is a symptom of a permissions issue with the underlying server. The web server (typically Apache) is unable to write the updated files to the Wordpress directory. From a security standpoint, this is actually a good thing. You generally don’t want to allow a public-facing service to make changes on your server. This problem occurs in other situations, such as configuring the WordFence plugin, so I’ve written a page devoted to troubleshooting Wordpress permissions on Linux hosts. That page describes how to fix the permissions issue without compromising the general security of your Wordpress host.

Troubleshooting Wordpress permissions errors on Linux hosts

Mon, 14 Jan 2019 16:59:21 +0000

Wordpress permission errors usually manifest as a request for FTP credentials with the error message, “Connection Information: To perform the requested action, WordPress needs to access your web server. Please enter your FTP credentials to proceed. If you do not remember your credentials, you should contact your web host.” You generally do not need to enter these credentials. If you’re hosting Wordpress on your own Linux server, read on to see what might be causing the problem, and how to fix it.

Netgear GS728TS, GS728TPS, GS752TS, and GS752TPS Switch Configuration Guide

Sun, 30 Dec 2018 20:05:41 +0000

Overview of the Netgear GS Switch Series

Netgear switch models GS728TS, GS728TPS, GS752TS, and GS752TPS (Gigabit Smart Switches) are value-priced edge switches (about $1000 on Amazon for the GS752TPS 48-port model with PoE) with limited Layer 3 routing capabilities. These switches are best deployed as edge switches in a small office environment with a limited budget. The fan noise is annoying but not deafening, so they should be located in a network closet or noise-insulated rack. In the case of budget constraints, the GS7 series could be used as core switches. They can be stacked for high availability, and they support LAGs and bonding to allow equipment with multiple NICs to run one cable to each switch, allowing the equipment to stay up even if one switch unit dies. However, these switches are not really designed for “enterprise” use as core switches.

Creating Kubernetes Secrets Using TLS/SSL as an Example

Fri, 14 Dec 2018 16:15:11 +0000

Creating Kubernetes secrets isn’t intuitive the first time you do it. A common reason to use a secret is to add a SSL/TLS certificate to a cluster. Kubernetes provides two ways to add a secret: directly on the command line, and from a YAML source file. First, let’s generate a test certificate to work with and select our cluster.

Prerequisites

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com"

This command produces two files: tls.key and tls.cert. In production, you’d generate a key file and use it to obtain a certificate from a certificate authority.

Yum/RPM Package Signing Key Fingerprints

Fri, 19 Oct 2018 21:44:41 +0000

The Importance of Checking Keys

RPM packages are signed using public-key cryptography. A package maintainer uses their private key to sign a package when they create it, and they make the corresponding public key available to the world. When you install a package with Yum or RPM, RPM uses the public key to verify the integrity of the package. If this process weren’t in place, it would be very easy to tamper with packages to install malware. When you install a package and you haven’t used that package’s public key before, yum or RPM prompts you to accept the public key by showing you the public key’s fingerprint. The fingerprint is a short sequence of bytes used to identify a longer public key. You should check the fingerprint presented by rpm or yum against a trusted fingerprint (usually published on the package maintainer’s web site). If you don’t check the fingerprint, you could be importing a public key from a malicious actor, and then yum will happily install whatever malicious payload that actor has inserted into RPM files.

Blog

Tue, 28 Aug 2018 01:42:00 +0000

Home

Tue, 28 Aug 2018 01:42:00 +0000

Welcome to your site! This is your homepage, which is what most visitors will see when they come to your site for the first time.

Optimize the Wordfence Firewall: Needs FTP Credentials

Sun, 26 Aug 2018 20:17:21 +0000

I use Wordfence as a Wordpress security solution. Even the free version has a comprehensive set of features, and the premium version is even better. It’s very easy to install and configure Wordfence as a regular plugin. However, the step to “Optimize the Wordfence firewall” fails for many people. This is not the fault of Wordfence, since different steps may be required depending on how your web server is configured. My solution is for those who are hosting their own Wordpress, but may be helpful to other situations. Please note that these instructions affect the security of your system, and are aimed at users with intermediate to advanced Linux knowledge.

How to run Shadowrun: Dragonfall on Ubuntu Linux

Sat, 16 Jun 2018 15:55:16 +0000

I’m very happy that Harebrained Schemes chose to release its Shadowrun game series (Shadowrun Returns, Shadowrun: Dragonfall, and Shadowrun: Hong Kong) for Linux. It can be a little difficult to get this stuff running on Linux. In this post, I will explain the errors you may see when trying to run Shadowrun on Linux.

Installation Procedure

I purchased and downloaded Shadowrun:Dragonfall on sale from Humble Bundle. 1. Verify file integrity. The md5 checksum is provided on Humble’s download page-check the output from this command against what Humble provides. This is how you make sure that you aren’t installing a corrupted or infected application on your system: [code language=“bash”] md5sum shadowrun-dragonfall-linux.tar.gz_2.0.9.zip [/code] 2. Unzip the downloaded file: [code language=“bash”] unzip shadowrun-dragonfall-linux.tar.gz_2.0.9.zip [/code] 3. Move the unzipped file (shadowrun-dragonfall-linux.tar.gz) to wherever you want to install it. I put it in a subdirectory called “Games” in my home directory: [code language=“bash”] mv ~/Downloads/shadowrun-dragonfall-linux.tar.gz ~/Games/ [/code] 4. Unpack the TAR archive. Note that there is a minor error on the part of the developers or Humble; the file you get after unzipping has the wrong file extension, so you have to use a different command than you normally would. The file should be called “shadowrun-dragonfall-linux.tar” because it’s not compressed with gzip. [code language=“bash”] tar xf shadowrun-dragonfall-linux.tar.gz [/code] 5. Run the game. This probably won’t work the first time; see the appropriate section below to install required packages on your system. [code language=“bash”] ~/Games/Shadowrun\ Dragonfall/Dragonfall Dragonfall: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory It seems Shadowrun Dragonfall has exited unsuccessfully (error = 127). Please verify that you have the latest drivers installed before filing a bug report here: http://harebrained-schemes.com/shadowrun/bug-reports [/code] 6. Install required packages on Ubuntu 16.04 (Xenial) 64-bit (x86_64). Shadowrun is a 32-bit game, but it runs on 64-bit Ubuntu because of Ubuntu/Debian’s multilib feature. You just need to install two specific 32-bit libraries that probably aren’t on your system by default: [code language=“bash”] sudo apt-get install libxcursor1:i386 libglu1-mesa:i386 [/code] You also need the appropriate graphics card drivers for your hardware. I have an Nvidia card and I use the proprietary drivers available directly from Nvidia (not the open-source version, which has caused problems for me with other games). If you had to do something differently on another version of Ubuntu, please leave a comment below.

New Relic APM on Fedora/CentOS/RedHat with SELinux Enforcing: Solved

Thu, 14 Jun 2018 03:27:28 +0000

I found the solution for getting the New Relic APM Agent to run on a CentOS 7 system with SELinux Enforcing. The only SELinux modification you have to make is to change the context of the log file directory /var/log/newrelic. You can temporarily make these changes with the chcon command: [code lang=“bash”] chcon -R -t httpd_log_t /var/log/newrelic/ [/code] Kill any running instances of newrelic-daemon. If you’re using New Relic’s recommended default setup, make sure that the file /etc/newrelic/newrelic.cfg does not exist. Verify that all required values in /etc/php.d/newrelic.ini are correct (especially the license key and the application name). Finally, restart httpd: [code lang=“bash”] systemctl restart httpd [/code] If that works, you should see output in both /var/log/newrelic/newrelic-daemon.log and /var/log/newrelic/php_agent.log. Now, make this file context change persistent, and verify the result: [code lang=“bash”] semanage fcontext –add –type httpd_log_t “/var/log/newrelic(/.*)?” restorecon -RF /var/log/newrelic/ ls -laZ /var/log/newrelic/ [/code] When you restart Apache, it spawns the newrelic-daemon processes, which communicate with APM via the socket file /tmp/.newrelic.sock. You really don’t want to run the newrelic-daemon processes via systemd if you can avoid it. The resulting process tree should look like this:

Internet Service Comparison: Spectrum Cable Modem vs AT&T Fiber

Tue, 05 Jun 2018 13:24:10 +0000

I’m moving to a new address in Central Florida which has multiple Internet Service Providers (ISPs) available. Spectrum (formerly BrightHouse) provides speeds up to 100Mbps via cable modem. AT&T just installed fiber in the neighborhood, so I was excited to find out what they would offer.

Spectrum Cable Internet

Here’s a cost breakdown by service:

Lightning Internet (200Mbps download data rate, 30Mbps upload): $90/month Static IP address: $10/month Modem rental: $4/month (you must rent a modem to get a static IP) Taxes: $0.26

Running Unity games via Steam in Linux

Mon, 28 May 2018 19:15:33 +0000

There seems to be a lot of confusion about how to run games built on the Unity engine distributed by Steam on a Linux system. I recently solved this problem after backing the promising survival sandbox game Planet Nomads on Kickstarter. Here is how to configure an Ubuntu Linux system and launch Planet Nomads successfully.

Prerequisites

If you have an Nvidia card, you must download and install the official proprietary drivers provided by Nvidia. I could not get Unity to launch with the open-source drivers. Once I installed the proprietary (closed source) drivers, I was able to launch Planet Nomads from the Steam application.

Audit terminology: SAS 70, SSAE 16, SSAE 18, SOC1, SOC2, Type 1, Type 2

Mon, 21 May 2018 22:08:17 +0000

If you are involved in information technology and compliance in a heavily regulated industry, or work with larger organizations, you have probably run across the terms SAS 70, SSAE 16, SSAE 18, SOC 1 report, SOC 2 report, Type 1 Report, and Type 2 Report. These terms are frequently abused and misunderstood, even by compliance “experts,” so I’ve written this brief one-page summary. Everything I state below has a direct reference on the AICPA site; I have learned not to trust third-party sources in regards to compliance.

Configuring Laravel/Lumen applications to connect to SQL database sockets

Mon, 14 May 2018 18:58:47 +0000

The Laravel/Lumen framework documentation does not explain how to connect an application to a database using UNIX sockets instead of a TCP-based network connection. I recently had to configure the Polr URL shortener (built on the Lumen microframework by Laravel) to connect to Google Cloud SQL with a UNIX socket. Since all of Polr’s configuration takes place in the .env file, and there is no environment variable that’s specific to database sockets, this took some research. I finally found the answer in an obscure StackOverflow response. Previous TCP connection: [code] DB_HOST=some-server-name DB_PORT=3306 [/code] Socket-based SQL connection: [code] DB_HOST=localhost;unix_socket=/cloudsql/cloud-project-name:us-east1:sql-instance-name [/code] This approach will work with any UNIX socket; you just need to give it the absolute path to the socket.

Configure CentOS/RedHat VMs with Kickstart files on Virtualbox

Sat, 05 May 2018 18:54:50 +0000

Kickstart is a type of file that’s used to automatically install RedHat or CentOS Linux on a physical or virtual server. If you are managing more than a few servers, it’s a good idea to configure the servers via kickstart files instead of logging in and configuring each one manually. However, some unfortunate choices were made when defining the Kickstart file format, and it’s not the easiest thing to use. When creating a new Kickstart or making major changes, it’s good to have way to quickly iterate and test your changes. Iterating on a Kickstart file is slow and inconvenient in an enterprise environment in which a freshly booted server obtains an image and Kickstarter file via PXEboot. To speed up Kickstart testing and debugging, I’ve developed a simple way to use Kickstart files with CentOS/RedHat guest virtual machines that run in VirtualBox on my Mac. This process should also work with Windows or Linux hosts. When the RedHat installer is booting, it looks for a file called ks.cfg on a volume with label OEMDRV. First, we will create this volume as a small virtual disk image. On a Mac, you can create this image with Disk Utility as shown below. Note that the Name needs to be OEMDRV (though you can make the filename anything you want).

Create OEMDRV disk image OSX Disk Utility

Service account credentials with the Python client for the Google Drive API (v3)

Mon, 23 Apr 2018 12:23:29 +0000

EDIT: Get the full code for this post on Github. This article only contains important snippets of code that require explanation.

There are numerous ways to authenticate against the Google Drive API. If you have an application running on Google Compute Engine that needs to access Drive, a Service Account is probably the easiest way to do it. One use case is for an application to write reports or log files to Drive so that users can see them without logging into a server. Before you try this example, go through all of the steps in Google’s Using OAuth 2.0 for Server to Server Applications guide and save your service account’s private key locally in JSON format. Getting credentials from a service account file is easy:

General error: 2006 MySQL server has gone away

Tue, 10 Apr 2018 03:19:57 +0000

“MySQL Server has gone away” is a cryptic error that can be hard to troubleshoot (look at all the various responses on Stack Overflow!) Many problems can cause this error; I would like to document one specific case. In this example, the client is a PHP app using the Phalcon framework:

[Mon, 09 Apr 18 03:34:08 -0400][ERROR] SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /path/to/ModelBase.php:
Stack trace:
#0 [internal function]: PDOStatement->execute()
...
#17 {main}

This error can be tricky to troubleshoot. I looked in three logs:

Viewing logs for a cluster of instances on Google Stackdriver Logging

Tue, 27 Mar 2018 14:01:51 +0000

StackDriver Logging is a great feature of Google Compute Engine (GCE). You pretty much need a centralized logging solution if you are taking maximum advantage of the features offered by GCE. For example, most production applications will run on a cluster of web servers. If you set up the cluster as a managed instance group on GCE, Google can auto-scale the size of the cluster based on traffic. The challenge is that it’s much harder to troubleshoot errors across a cluster. The requests that caused the error could be spread across any number of servers, with randomly assigned names. If load drops and the server pool contracts, you will entirely lose any log data on a server that’s auto-deleted. StackDriver Logging is the answer to this problem. Configure all servers to send all logs to StackDriver, and you can view all of your web server logs in one interface, with the entries in chronological order. View StackDriver Advanced Filter as a Gist on GitHub The problem is that the Stackdriver docs only explain how to select an instance by the instance ID, which is a random-looking string that is not visible in the Console. Further, if you have a large cluster of disposable servers, you would have to look up the ID for every instance. If an instance was deleted when the cluster was auto-scaled, you won’t even know what to look up. Therefore, it’s much better to search for the logs based on a human-readable server name. For example, assume that your servers are Google Compute Engine instances that are named [region]-[zone]-web-[id]. Here’s a filter to match on that naming convention: [sourcecode language=“plain”] resource.type=“gce_instance” logName=“projects/my-project-name/logs/apache-error” labels.“compute.googleapis.com/resource_name”:"-web-" textPayload:“execution time” [/sourcecode]

Protecting a RESTful JSON API from a CSRF attack

Sun, 07 Jan 2018 13:40:57 +0000

“Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated” (OWASP definition)

CSRF is related to Cross-Site Scripting (XSS), but does not require the execution of Javascript or any other front-end code. In fact, APIs are increasingly vulnerable to CSRF attacks due to the emergence of single-page web applications. A single-page web application is a Javascript “client” application that executes in the user’s browser. The client application interchanges data with the server via API calls. There are many advantages to this architecture, but it does increase the attack surface for CSRF attacks. When the user authenticates, they receive a unique token with a limited lifetime, which is usually stored as a cookie. The token is passed to the server as a header on every API call, and the server may return a new token with an updated expiration time with every response.

Upgrade Ubiquiti UniFi Access Points (WAP) now to avoid KrackAttack

Tue, 17 Oct 2017 15:39:10 +0000

On October 15, 2017, security researcher Mathy Vanhoef announced the discovery of KrackAttacks, a serious flaw in the WPA2 encryption protocol that encrypts most WiFi connections. Using this method, an attacker can decrypt traffic from almost any wireless access point (WAP) and clients. Every WiFi access point will need to be upgraded with patch that prevents this attack. Ubiquiti has already released a patch for UniFi access points that addresses this vulnerability. However, the upgrade process may not be straightforward, depending on the age of your access points. The following process will ensure that you are running a safe version of the firmware (3.9.3.7537 or later). Note that any upgrade will result in downtime for each access point as it is upgraded, so you will want to perform a rolling upgrade if you have multiple devices that will be upgraded while people are using them.

Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

Sun, 01 Oct 2017 17:24:01 +0000

When you run yum upgrade on a CentOS/RedHat 7 instance, you will be upgraded to 7.4. If you have a FreeRADIUS server, you will be upgraded from version 2 to 3, and your server will likely stop authenticating! Good times! Fortunately, the solution was not complicated for us, because we had good documentation.

Symptoms

Your radius server suddenly starts denying logins.

Diagnosis

The configuration file /etc/raddb/radiusd.conf has changed enough between versions 2 and 3 that you cannot re-use the old version. To get radius debugging output, stop the radiusd service, and run (as root):

Managing persistent disks on Google Compute Engine

Tue, 19 Sep 2017 12:19:24 +0000

It can confusing when you have multiple persistent disks on an instance running on Google Compute Engine. For example, a server may have separate disks for the filesystem root, MySQL data, logs, and /tmp. Once you’ve created the Compute Engine disks and attached each one to the instance, how do you know which Compute Engine disk maps to each volume on the instance? First, describe the instance to see a list of the attached disks:

Migration from GoDaddy to Wordpress.com hosting

Thu, 14 Sep 2017 18:42:54 +0000

I’ve migrated my blog from GoDaddy to a paid plan with Wordpress.com. I decided to trade flexibility for simplicity, since I don’t want to spend time fiddling around with the administration of this site. When I spend all day on IT, doing IT for a blog is no longer on my list of hobbies. With GoDaddy, you are responsible for all aspects of security, including updating Wordpress, updating themes and plugins, and installing a security plugin to block the thousands of known attacks against Wordpress. I have two major security concerns with GoDaddy. I had to allow insecure ciphers in order to connect to the shell on my site via SSH (or SCP). Why on earth haven’t they updated to a recent version of SSH?!? GoDaddy also requires you to purchase an SSL certificate from them to enable TLS on your domain. There’s no way to install a free certificate from Let’s Encrypt. I looked into enabling two-factor auth, but you have to manage it yourself with a plugin. With a Premium plan from Wordpress.com, SSL is enabled automatically with a certificate from Let’s Encrypt. I can use my domain name, and security is mostly handled for me. I can enable two-factor authentication. The downside is that I’ve lost some flexility; I can’t use my own theme or install my own plugins unless I upgrade to a Business plan. Also, I have to set up ads through Wordads.com to try to cover the cost of the site. I was fairly successful in covering the cost of hosting with Google AdSense; I’m not sure how effective WordAds will be.

About

Wed, 30 Aug 2017 14:57:56 +0000

This is an example of an about page. Unlike posts, pages are better suited for more timeless content that you want to be easily accessible, like your About or Contact information. Click the Edit link to make changes to this page or add another page.

Contact

Wed, 30 Aug 2017 14:57:55 +0000

Contact me on LinkedIn: https://www.linkedin.com/in/craigafinch

Is there a "Primary Domain Controller" in Active Directory?

Thu, 29 Jun 2017 14:08:52 +0000

The Historical Answer

With Windows NT, prior to the advent of Active Directory, there was one Primary Domain Controller (DC) per domain, and every other DC was a Backup.

The Modern Answer

When Active Directory was introduced with Windows 2000, domain controllers became fully multi-master. There is no primary domain controller. However, there are two caveats that may confuse you.

FSMO Operations Masters

Though Active Directory is multi-master, certain operations must take place in a way that ensure consistency across the domain controllers. For this reason, one DC will be designated as the Operations Master for each role. This is notthe same thing as a Primary Domain Controller, because there are currently 5 master roles, and a different DC could hold each role. To view the current Operations Masters, run the following at a Windows command prompt: NetDOM /query FSMO In Powershell, you have to run two commands: Get-ADDomain | Select-Object InfrastructureMaster, RIDMaster, PDCEmulator Get-ADForest | Select-Object DomainNamingMaster, SchemaMasterReferences:

MySQL/MariaDB, logrotate, and SELinux

Tue, 17 Jan 2017 03:30:14 +0000

Assumption: You have SELinux Enforcing on your database server. If you’re still solving problems by permanently setting SELinux to Permissive, I don’t think you can really call yourself an IT professional. Here are the commands to set the SELinux context of all text MySQL log files to var_log_t so that they can be rotated by logrotate:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/.*.log(-[0-9]+(.gz)*)*)+"
restorecon -R -v -F /var/lib/mysql

NOTE: This expression will also match the file /var/lib/mysql/tc.log and change its context, which will prevent MySQL from starting. If your version of MySQL uses tc.log, here is one possibility:

Synology DiskStation DS1515+ Review

Sun, 18 Dec 2016 20:19:48 +0000

Summary: The Synology DS 1515+ is a capable little NAS with a large feature set, but it has some software reliability issues and limited technical support.

Hardware

The DS1515+ is a compact unit that feels sturdy. It holds five 2.5" or 3.5" drives, and 3.5" drives can be installed without any tools. You will need a Phillips-head screwdriver to install an additional RAM module, but that’s also a very simple process. The fans are also easily replaceable.

anacron run-parts generates invalid or malformed syslog messages

Sun, 06 Nov 2016 02:43:43 +0000

On RedHat and CentOS 6 and 7, anacron generates syslog messages that are mangled when they are forwarded by rsyslog. I found the cause and a solution in a comment by Tomas Heinrich on this old Fedora bug. Sadly, that bug was closed instead of getting fixed, probably because Fedora is oriented toward desktop users who are not generally forwarding syslog messages! Syslog uses the following default template for messages: "%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" The syslogtag field is truncated to 32 characters, but cronie-anacron includes a lot of data in its tag. The tag gets truncated to 32 characters, resulting in invalid syslog. If you try to send them to Logstash, you’ll get errors like this: {"message":"Nov 4 19:01:01 my-web-03 run-parts(/etc/cron.hourly)[1858 starting 0anacron","@version":"1","@timestamp":"2016-11-05T00:01:01.146Z","port":55456,"type":"syslog","tags":["_grokparsefailure"],"host_ip":"192.168.17.23"} Unfortunately, the problem is with the sending host, not Logstash, so you have to update the rsyslog config on every host that generates anacron messages. I prefer to do this in /etc/rsyslog.d to keep the configs more manageable. Add these two lines, which add a template that has no restriction on the length of the tag field, and activate the template when forwarding: $Template fwd,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%" $ActionForwardDefaultTemplate fwd I placed these lines in /etc/rsyslog.d/logstash.conf along with my other Logstash-specific rsyslog configs.

Failed to schedule Software Protection service for re-start: Error Code: 0x80041316

Sat, 08 Oct 2016 17:35:20 +0000

This post documents how to resolve the following error message, which may appear for no particular reason, and flood the Windows event log: Failed to schedule Software Protection service for re-start at 2116-09-14T16:54:27Z. Error Code: 0x80041316. The following steps will resolve the error on Windows Server 2012r2, but based on other links, similar steps should work for Windows 8, 8.1, or Server 2012.

Root Cause

The root cause, in my case, was a corruption in the XML files that control task scheduling for the Software Protection service. I am not sure what caused this error, but it appeared to start after a domain controller was not shut down cleanly. Microsoft’s official documentation reports that another cause may be a mismatch between the permissions used to run the task and the permissions on the files that control the task. However, most people have reported that the problem is caused by corrupted XML files. Also, note that the Microsoft documentation is inaccurate when it states that the task must run as the NETWORK SERVICEaccount. If the task is triggered by an interactive user logon, it should run as accountInteractive.

Making Windows work for Linux and OS X admins

Wed, 25 May 2016 04:10:17 +0000

If you are a Linux or OS X power user, then you’re used to having all the necessary tools built into your OS. When you log into a Windows system (What! No command line?) you may feel lost. These tools and shortcuts will help you be productive on Windows systems.

Windows Shortcuts

Alt-x is a magic shortcut key on Windows 8 and Server 2012. It pops up a little menu in the lower right corner of the screen which contains just the items that an admin needs. Try it!

Windows Programs

If you want Windows to work a little bit like Linux or OS X, you need to install a few of these free programs. Some are gratis, and some are libre.

Microsoft Baseline Security Analyzer Troubleshooting

Fri, 25 Mar 2016 03:30:36 +0000

Microsoft Baseline Security Analyzer (MBSA) is a useful tool for auditing the configuration and update status of Windows computers. Most of the time, its reports are useful and easy to understand. However, some of its responses are baffling, and some of its suggested solutions haven’t been updated since Server 2003. Here is my collection of odd MBSA reports, and how to resolve them.

Internet Explorer

MBSA Reports:

The use of Internet Explorer is not restricted for administrators on this server

Solution: Enable IE Enhanced Security Configuration in the Server Manager

Troubleshooting SSSD, realm, kerberos, and SSH

Thu, 10 Mar 2016 19:51:20 +0000

SSSD (System Security Services Daemon) allows Linux systems (specifically, Red Hat, CentOS, and Fedora) to verify identity and authenticate against remote resources. If you have a CentOS or Red Hat enterprise system, and you need to authenticate against a domain controller such as FreeIPA or Active Directory, SSSD is the way to go. I use SSSD on CentOS 7 systems, but it is now available on CentOS 6 as well. A few years ago, adclient (an open-source project from Centrify) was your only option to make a CentOS 6 server authenticate against Active Directory. adclient seems to have reached end of life, so SSSD is definitely the path forward. I won’t repeat the procedure for using Active Directory as an identity provider on a Red Hat 7 system. Instead, I want to provide a few troubleshooting tips, since limited information is available on SSSD and related tools.

ODEPACK: Numerical Solutions to ODEs

Wed, 22 Jul 2015 23:52:21 +0000

ODEPACK is a library for the numerical solution of systems of ordinary differential equations. It is written in Fortran 77, but can be accessed from C and C++ with only a little effort. The permanent home of ODEPACK is in the ODEPACK directory at netlib. Surprisingly, there are very few open-source libraries that perform the functions of ODEPACK.

A friendlier distribution of ODEPACK

https://github.com/cfinch/odepack

ODEPACK is distributed as fixed-format Fortran 77 source with plain text files offering documentation and examples. It is not the friendliest distribution, because there are no makefiles and even the examples are buried in commented-out sections of the solver source code. It seems that ODEPACK was designed to be used in source form, perhaps by copying relevant functions and pasting them into your own Fortran program, or compiling ODEPACK right into your program. In an effort to get ODEPACK running for my own use, I have developed a friendlier distribution. I created a Makefile that will compile ODEPACK as a shared library, using the Gnu compiler collection (gcc), and separated the demonstration code into a directory of standalone programs. I intended to make it publicly available…but I never did. Ask me if you want it!

LVM device-mapper: reload ioctl failed: Invalid argument

Fri, 19 Dec 2014 17:23:19 +0000

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

GPG Public Key

Fri, 12 Dec 2014 02:55:06 +0000

For secure communication, please use this public key with a program such as GPG.

Is it worthwhile to embed structured data in Web content?

Sun, 09 Nov 2014 18:03:16 +0000

Short answer: not really! A few years ago, I was fascinated with the idea of microformats. The concept is to add structural data to common HTML tags so that certain types of data on a web page can be clearly identified. Adding structural information makes it easier to automate the parsing and indexing of web content. The original microformats included common data sets that contain well-defined data, such as contact information or a calendar event. In the intervening years, microformats were replaced by microformats2, with additional draft standards for more items such as recipes, resumes, etc. I decided to use my resume as a “use case” for structured HTML data. My original plan was to use the h-resume draft specification from microformats2. I modified my plan when I realized that major search engines (Bing, Google, Yahoo! and Yandex) have decided to parse another semantic technology, microdata (microdata W3C spec). There is no point in creating semantic web pages if nobody is going to parse them, so I used the microdata specifications that are available at schema.org and placed them into an h-resume framework. I used the event class to represent each position I held or educational achievement, because the event class is the only one with start and end dates. Is it worth the effort to add structured data to HTML pages? Generally, no. Here’s why:

Monitoring with SNMP, Part 3: Automate active monitoring with Nagios

Mon, 12 May 2014 12:29:08 +0000

My last post showed how to monitor networked devices with SNMP. You could try to remember to manually check the status of things periodically, but that would be missing the point of computers. Instead, automate your monitoring with Nagios, a web-based monitoring tool for Linux that automates the process of actively querying devices and doing something with the information. Nagios is available as free open source software (Nagios Core), and the company offers additional non-free products with premium features. The open-source version is fine for getting started and setting up basic monitoring. Nagios does a lot more than just SNMP monitoring. I’ll refer you to the Nagios Core documentation to get Nagios up and running, and I’ll focus on how to set up Nagios to actively monitor devices with SNMP.

Monitoring with SNMP, Part 2: Command-line tools for active SNMP

Fri, 02 May 2014 18:27:02 +0000

In Part 1, I summarized the basic concepts of SNMP and defined the terms and acronyms used in this post. Now, I will show how to use SNMP to monitor actual devices. As an example, I will monitor an enterprise-grade uninterruptible power supply (UPS) and power distribution unit (PDUs) from Tripp-Lite. These devices have an SNMPWEBCARD installed to support communication over Ethernet. Command-line tools for SNMP communication should be available for any Linux distribution (or any other UNIX-derived OS). Documentation for the basic SNMP tools is available online. The challenge with SNMP is figuring out what parameters are supported by a particular device. Most devices support a set of standard OIDs that return basic information such as device name, uptime, etc.

Monitoring with SNMP, Part 1: Fundamentals of SNMP

Wed, 05 Mar 2014 16:14:49 +0000

SNMP is a protocol for conveying information and controlling devices over a network. SNMP can be used in two ways:

Active: a device sends a command to set a parameter or request information for another device Passive: a device sends an alert (called a trap) to another device, which is configured to receive traps and do something with the information.

The “payload” of an SNMP message is called an Object Identifier, or OID. An OID is an ordered list of non-negative numbers, such as:

"Exporting" a project from a Git repository

Fri, 24 Jan 2014 18:48:31 +0000

What do you do when you want to distribute or release source code that is stored in a Git repository? Obviously, if your target audience is using Git, you can just compress the directory that contains the repository and distribute the copies, or give the users a way to clone your repository (such as GitHub). However, your audience may not be Git users, or the hidden .git directory may be very large and you don’t want to distribute it. The solution is the git archive command, which packs the files from a tree-ish into an achive (ZIP or TAR). By “tree-ish”, they mean that you can specify a branch, commit, HEAD, etc. git archive is somewhat analagous to the svn export command. I find the most useful form of this command to be: cd example git archive --output ~/example.zip --format=zip --prefix=example/ HEAD Do not forget the trailing slash after the directory that you specify with the --prefix flag! REFERENCE: How to do a “git export” (like svn export)

Running network experiments on the GENI project

Fri, 17 Jan 2014 20:50:04 +0000

The GENI Project is a networking testbed that is used by researchers studying novel networking technologies. While the technology is fascinating, the web site is, unfortunately, a confusing mess. Here are some pointers to get you started (or refresh your memory). This post will be updated as I learn more.

Key GENI Links

GENI Portal

This is where you log into the GENI Project. Your institution must have Shibboleth enabled and be part of the InCommon Federation. Click on the “Use GENI” button, enter the name of your institution into the search box, and you will be redirected to your institution’s login page.

Collaborative Git workflow: Shared Repository on a File Server

Fri, 10 Jan 2014 20:55:26 +0000

GitHub is a great tool for collaborating on projects. However, sometimes it is necessary to mimic the “GitHub workflow” using a shared repository on a local Linux server. The following example shows how I shared an example repository with multiple users. We are also using the Git flow model for branching, aided by the handy git flow plugin.

On my workstation

I started by creating a repo on my local workstation and setting it up to use the git flow plugin.

Linux configuration management roundup

Mon, 09 Dec 2013 19:52:20 +0000

Our high performance compute cluster (HPCC) has fairly primitive tools for managing the deployment of the operating system on the compute nodes. Our current tools are “aspencopy,” which takes an “image” of a the filesystem of a running server and saves it as a .tar.gz file (NOT a disk image). “aspenrestore” is its counterpart, which deploys an “image” to another server. The utility is smart enough to update things like the host name, IP address, host SSH keys, etc. However, the images are essentially “black boxes,” in the sense that there is no system for keeping track of which configuration changes have been applied to which image, and no way to know which image is running on each server. The next cluster that I am responsible for purchasing must include a configuration management/data center automation system, such as:

Configuring GRUB2 on Ubuntu to boot from another Linux partition

Mon, 02 Dec 2013 13:28:30 +0000

My recent Ubuntu installation was my first experience with the new GRUB 2.x series of bootloaders. Unforunately, the process of manually configuring GRUB2 on Ubuntu is not well documented in the case that everything doesn’t work “automagically.” I had to solve two problems: the blank screen at boot, and getting GRUB to boot to an existing partition with CentOS 5 installed.

Blank screen at boot

I got a blank screen after selecting an OS from the GRUB menu. GRUB seemed to work fine: it presented a menu of OS selections and allowed me to select one. Some text would then scroll by quickly, and then the monitor would then go blank and go into power-saving mode. This PC has an old NVIDIA card (I did not install any NVIDIA drivers). I found that I need to edit the file /etc/default/grub and set the nomodesetoption: GRUB_CMDLINE_LINUX="nomodeset" I then ran the commandupdate-grub to create the file /boot/grub/grub.cfg. You don’t want to edit this file by hand, since it gets over-written by configuration scripts.

Configuring XFCE4 and LightDM on a minimal Ubuntu installation

Mon, 25 Nov 2013 03:07:43 +0000

I recently installed Ubuntu on an older PC with 1GB of RAM and 80GB of hard drive space, so I wanted a lightweight desktop interface. I chose XFCE, since it is both lightweight and usable, and I have used it extensively. You can get Ubuntu pre-made with XFCE (xubuntu), but there are some disadvantages. The ISO is slightly too large to fit on a CD, and it comes with a lot of applications that I don’t need. Instead, I installed Ubuntu from a minimal CD and then used apt-get to install XFCE and the LightDM display manager. I learned that a couple of extra steps must be taken to get XFCE to play with lightdm. I got the error “can’t find session ubuntu”. Don’t bother changing the .dmrc file in your home directory; this file is overwritten every time lightdm starts! I configured lightdm to use an XFCE session with the command:

Preventing "soft" failures due to memory fragmentation in Linux

Wed, 20 Nov 2013 13:27:41 +0000

A previous post documented that a Linux server running a pre-2.6.24 kernel can fail to allocate large chunks of memory after its memory has been fragmented by a “thrashing” incident. In this post, I will point out some ways to prevent this problem.

Use a Newer Kernel

We have some servers running RHEL 5.9 with the kernel updated to 2.6.34.14. After a thrashing incident, these servers do not experience the same problem with allocating large blocks of memory. I think the fix is documented in the release notes for kernel 2.6.24. Section 2.4 talks about “anti-fragmentation patches” and includes a link to this article about Linux memory management, which links to this thorough documentation of the anti-fragmentation patches.(BTW, here is the full list of 2.6 kernel changelogs) My plan is to deploy RHEL 5.9 with the updated kernel to all the compute nodes in our cluster. However, this still doesn’t solve the problem of a user who requests some portion of the RAM on a node and then proceeds to consume more memory than requested. This is unfair to another user whose job is running on the same node.

Memory fragmentation degrades performance in Linux kernels < 2.6.24 (RHEL 5.4)

Fri, 15 Nov 2013 21:30:38 +0000

I have data from the STOKES High Performance Compute Cluster which definitively shows that kernel versions prior to 2.6.24 can suffer significant performance degradation due to memory fragmentation. I noticed the problem on servers running Red Hat Enterprise Linux (RHEL) 5.4 with kernel version 2.6.18-164.el5. This post will document my findings. The graphs were taken from our Ganglia monitoring system.

This node has 24GB of RAM. As long as processes do not request more than 23GB of RAM, the node operates normally. Processes can use 23GB of RAM all day long:

Insert an EPS file into an Asymptote Vector Graphics document

Wed, 23 Oct 2013 13:08:06 +0000

It’s surprisingly difficult to find out whether it is possible to include an image from an Encapsulated PostScript (EPS) file into an Asymptote vector graphics document. It turns out that it is easy, but difficult to find in the Asymptote docs (I finally found the answer, via Google, in the FAQ). It turns out that you use the label function to insert an image into an Asymptote document. Here is a snippet of code that I used to assemble a multi-part figure from several EPS documents:

How to fix your IEEE email alias so that it delivers all messages

Wed, 16 Oct 2013 18:18:11 +0000

I and others who use ___@ieee.org email addresses have a serious problem in which we occasionally fail to receive certain email messages that are sent our ieee.org addresses. This is a serious problem, and I’ve finally found a solution.

First, some background. Before the summer of 2013, the IEEE email address was simply an alias, which forwarded email to an actual email account. Some “spam” filtering was provided by IEEE, which could be modified or disabled through the IEEE web portal. In the summer of 2013, IEEE moved to Google Apps. Your ieee.org email now points to a Google Apps account. If you were previously using the alias service, the Google Apps account is configured to simply forward all email to the other account. However, the Google Apps account is also performing spam filtering on your email! When I finally logged in to my IEEE Google Apps account, I found several non-spam messages which had never been delivered because they were mistakenly sent to the spam folder in the Google Apps account (which I never check). Here is how to disable this dangerous and irritating “feature” that may be causing you to lose valuable email messages.

Using the MATLAB Distributed/Parallel Computing Toolbox without modifying your cluster

Wed, 02 Oct 2013 19:44:53 +0000

This post shows how I set up MATLAB to perform parallel calculations on a multi-core server which is part of an existing cluster that is managed using Torque and Moab. I did this without making any changes to the cluster submission system. The installation documentation for the MATLAB Parallel Computing Toolbox and Distributed Computing Server is poor. The instructions make a lot of assumptions that don’t fit into an existing “production” cluster environment. For example, they assume that a GUI is available to configure the cluster (this step is not required for my method!) Further, the instructions give the impression that MATLAB components must be “installed” on both the head node and compute nodes by an administrator. THIS IS FALSE! Ordinary users can run concurrent (multicore) MATLAB jobs without administrative privileges!

PHPUnit on CentOS 6

Wed, 28 Aug 2013 13:09:16 +0000

In my last post, I showed how to install PHPUnit with PHP 5.3 on a CentOS 5.9 system. Today, I will tackle the much simpler task of installing PHPUnit on a CentOS 6.4 system. The task is easier since PHP5.3 is standard on CentOS 6. On my system, PEAR was already installed. Steps: 1. Install the PHPUnit channel for PEAR:

yum install php-channel-phpunit

2. Set channel to auto-discover:

Installing PHPUnit on CentOS 5.9

Tue, 20 Aug 2013 02:13:21 +0000

I recently had to install PHPUnit on a CentOS 5.9 system. This turned out to be a little more complicated than I had anticipated, so I wrote down the procedure in case I had to do it again. Here are my notes: 1. Install php53 packages, not php (php packages install the old 4.x series) 2. Install PEAR, PHP’s component management system. Here is the complete Yum log for a PEAR installation: Jun 03 22:49:56 Installed: php53-common-5.3.3-13.el5_8.x86_64 Jun 03 22:49:57 Installed: php53-cli-5.3.3-13.el5_8.x86_64 Jun 03 22:49:58 Installed: php53-5.3.3-13.el5_8.x86_64 Jun 03 22:51:24 Installed: php53-devel-5.3.3-13.el5_8.x86_64 Jun 03 22:51:25 Installed: 1:php-pear-1.4.9-8.el5.noarch 3. Install php53-xml: Jun 03 23:30:40 Installed: php53-xml-5.3.3-13.el5_8.x86_64 4. Install php54-xml and restart httpd to get the DOM extension for PHP 5. Install PHPUnit with PEAR:

Very helpful PDF manipulation tools on Linux

Fri, 14 Jun 2013 21:25:15 +0000

You can do some really cool stuff with Ghostscript on the command line on a computer running Linux or OS X. However, it’s hard to remember all that stuff when you don’t use it very often. I recently learned about a set of command-line PDF manipulation tools called PDFjam. If you are a geek like me, you probably already have pdfLaTeX installed. I can’t summarize the capabilities any better than the description on the PDFjam site:

When root cannot run X applications from a local terminal

Sun, 02 Jun 2013 03:08:56 +0000

I run XWindows on a CentOS 5.9 box as a non-root user. To perform administrative tasks, I occasionally log in as root using “su” in a terminal window. However, I found that I was unable to start any GUI applications as root. I got the following error:

Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified
E233: cannot open display Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

This can happen when xhost isn’t configured to allow local, non-network connections. As the user who started the xwindows session (not root) run:

The cause of init: Id "co" respawning too fast, and how to fix it

Fri, 05 Apr 2013 20:26:03 +0000

I recently found the following message in the system logs for one of the compute nodes in the STOKES cluster:

init: Id "co" respawning too fast: disabled for 5 minutes

This caught my attention, because this OS image on this particular node should be identical to the image that is deployed on the rest of the nodes in the cluster. Why was it the only one producing this strange warning message? I searched the web and learned that the following line in /etc/inittab is causing the warning:

How to plot .xvg files from GROMACS on Windows

Tue, 12 Mar 2013 18:00:51 +0000

GROMACS produces graphical output in the form of .xvg files. These are designed to be viewed with a classic UNIX/Linux plotting program called Grace. If you happen to be using Linux and you have Grace installed, it is very easy to plot the data with the command

xmgrace my_file.xvg

If you aren’t using Linux, plotting .xvg files is quite a bit more difficult. Gnuplot is the only free plotting program that I have found that can handle .xvg files. Gnuplot is available for Windows (there is a direct download link near the top of the download page), but I will caution you that Gnuplot is not easy to learn. It has a command-line interface, and there are no point-and-click options to do basic operations such as labeling the axes of the plot. If you are using Gnuplot, you can plot the contents of an .xvg file with the command:

Opportunity for postdoctoral research associate in high performance computing

Mon, 04 Mar 2013 22:51:12 +0000

My current employer, the STOKES Advanced Research Computing Center (STOKES ARCC), is hiring a postdoctoral research associate to conduct research in high performance computing with an emphasis on next-generation networking technologies. The ARCC has internal funding that will be used to upgrade our research network to the Internet2 Innovation Platform standard. We are also seeking external funding to extend the research network across the UCF campus. We are looking for a candidate with an interest in topics such as defining a “Science DMZ,” Internet2, GENI, perfSONAR, software-defined networks, etc. Please use the link above to apply for the position. Feel free to contact me if you have questions-my contact information is on the about page.

Updated GROMACS tutorials

Thu, 28 Feb 2013 21:40:54 +0000

I have published up-to-date versions of two classic GROMACS tutorials on GitHub. The Getting Started section of the GROMACS online documentation contains some helpful tutorials. Unfortunately, these tutorials have not been updated in a while. They also don’t explain how to set up an efficient workflow to run large molecular dynamics simulations on a shared cluster using a resource manager such at Torque. I have created a set of files that implement the speptide tutorial from the GROMACS documentation.You can use my files and follow along with the explanations in the GROMACS manual. The speptide directory has two subdirectories: one for MDP files (which control the simulations) and one for simulation files. I have found this to be an efficient setup because many MD experiments involve simulating multiple variants of a molecule (such as mutants of a protein) and comparing the results. All variants should be run with the same parameters, so the same MDP files are used for each simulation. Keeping the MDP files in their own directory and creating symbolic links from the run directory to the MDP files ensures that all simulations are run with identical parameters. In the run directory, there is a Bash script called setup_GROMACS_job.sh. At the top of the script, the user sets variables to set simulation parameters such as the box size. The script can be run with the command

Installing Lumerical FDTD on a linux cluster

Mon, 25 Feb 2013 20:25:43 +0000

Most of the time, RPM (especially in conjunction with yum) is a decent package management solution. However, I can think of two common circumstances when you don’t want to let RPM install a package:

You don’t have root permissions on a system such as a shared cluster You are an administrator on a shared cluster and you can’t risk having a package over-write system-critical files

One approach is to extract the files from the RPM package and install them manually. I used this method to install FDTD Solutions from Lumerical on the STOKES Linux cluster. The application is distributed as an RPM package. I never install a third-party RPM as root, because a badly constructed package might over-write a system-critical file that some user is depending on. I downloaded the appropriate TAR file from Lumerical, uncompressed it, and looked at the contents. There is a simple script called install.sh which checks to make sure the user is root and then tries to install a hardware key driver and an RPM that contains the FDTD software. Here are my recommendations for installing Lumerical products on a cluster:

OpenMPI, Intel Compilers and RedHat 5: cannot find -lnuma

Wed, 20 Feb 2013 23:21:42 +0000

I found an interesting quirk when trying to build an OpenMPI application on a visualization node with a “stock” version of Red Hat Enterprise Linux 5.8. I used mpicc to compile the application and got the following error:

$ mpicc hello_world_mpi.c -o hello_world
/usr/bin/ld: cannot find -lnuma

This was rather surprising, since this node mounts a directory via NFS that contains OpenMPI and Intel Composer 2013, and these applications are known to work on other nodes. To find the source of the problem, I used the showme option to see the command that is actually run by mpicc:

pickle, hickle and HDF5

Sun, 03 Feb 2013 19:33:57 +0000

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. The hickle developers have made a good start, and they have a long way to go before hickle will be useful to a wider audience. Right now, hickle can only store NumPy ndarrays and Python list objects. If you only need to store lists and arrays, you might as well use HDF5 bindings for Python such as PyTables or h5py. The power of the pickle module is that you can immediately serialize almost any Python object of arbitrary complexity, store it on disk, and retrieve it. hickle will only achieve its full potential once it replicates this functionality, and I’m not sure how difficult this will be. Ideally, you might be able to derive a class from Pickler that uses Picker’s methods to serialize an object, and then add your own method to write the serialized object to an HDF5 file. In a future post, I’ll describe some of the practical problems with using pickle files to store data, and try to organize some thoughts about how they might be solved.

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Wed, 16 Jan 2013 22:29:19 +0000

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

A list of system and application profiling tools

Mon, 07 Jan 2013 14:21:04 +0000

This article from Admin Magazine has a handy list of profiling tools.

A trend for 2013: the democratization of manufacturing

Tue, 01 Jan 2013 21:40:00 +0000

3D printers have been getting more attention in the mainstream media in the past few months. 3D printers were mentioned in a story about tech trends on National Public Radio today, and my father showed me a newsletter from a mutual fund company with a short article about 3D printing. You know that something is hitting the mainstream when retired people learn about it. Of course, 3D printing is old news to those of us in the Maker community. We know that just because consumers can buy 3D printers for their homes doesn’t mean that they should. Today’s 3D printers aren’t exactly consumer products, and someone who believes the hype and spends a couple thousand dollars without doing a lot of research is probably going to be disappointed. The technology is not at the point where you can print objects that look as good or work as well as similar items you can buy at the store. Likewise, designing your own custom objects sounds good until you discover the price (and learning curve) of 3D design software. Focusing on 3D printers risks missing a broader trend in the democratization of manufacturing. From the late 1970’s to the early 1990’s, computing power became affordable to the masses. From the late 1990’s to the present, the Internet has given the common man access to an amount of information that would have been inconceivable two decades ago. These trends democratized the means of production for media, music, video, and information technology. Over the past few years, the barriers to entry for manufacturing are becoming lower at a rapid pace. Now you can print goods in your home, or send your design out for rapid prototyping, at an affordable price. For example, I recently spent $25 to buy a mount for my smartphone through Kickstarter. The business that is developing this item consists of one man with a 3D CAD program. The prototypes were fabricated using the Ponoko 3D printing service and the production tooling will only require an up-front investment of $18,000. Karl Marx developed communism because he perceived that the common man was oppressed by the few wealthy people who owned the means of production. Today, the common man doesn’t need to own the means of production because he can make use of them at an affordable price. Our concept of work is being transformed in a post-industrial revolution, and no one describes it better than Seth Godin. If you haven’t read any of his books, Linchpin is a good place to start. We’re living in an exciting time, and I’m looking forward to seeing what 2013 will bring!

Installing and configuring Infiniband on a Red Hat system

Wed, 12 Dec 2012 23:15:54 +0000

This post will take you through the installation and configuration of an Infiniband card on a server running Red Hat Enterprise Linux 5.4. These steps are applicable to any version of Red Hat 5, and will probably work with version 6 as well. It has been surprisingly hard to find all of these steps in one document.

Required packages

openib-1.4.1-6.el5.noarch libibverbs-1.1.3-2.el5.x86_64 libnes-0.9.0-2.el5.x86_64 libibumad-1.3.3-1.el5.x86_64 opensm-libs-3.3.3-2.el5.x86_64 swig-1.3.29-2.el5.x86_64 ibutils-libs-1.2-11.1.el5.x86_64 ibutils-1.2-11.1.el5.x86_64 (provides ibdiagnet and others) opensm-3.3.3-2.el5.x86_64 libibmad-1.3.3-1.el5.x86_64 infiniband-diags-1.5.3-1.el5.x86_64 (provides handy tools like ibstat and ibstatus) libibverbs-utils-1.1.3-2.el5.x86_64 (provides handy tools ibv_devinfo and ibv_devices) libibverbs-devel-1.1.3-2.el5.x86_64

The Infiniband troubleshooting quick reference

Tue, 04 Dec 2012 16:59:34 +0000

Glossary of Infiniband Terminology

GID: Global Identifier GUID: Global Unique Identifiers (also known as Direct Address) HCA: Host Channel Adapter LID: Local Identifier TCA: Target Channel Adapter SM: Subnet Manager

Infiniband: The Host Perspective

Every host on an Infiniband fabric has three identifiers: GUID, GID, and LID. A GUID is similar in concept to a MAC address because it consists of a 24-bit manufacturer’s prefix and a 40-bit device identifier (64 bits total). The Global Identifier (GID) is a 128-bit identifier similar to an IPv6 address (technically, a GID is a valid IPv6 identifier with restrictions). The GID consists of the 64-bit GUID plus an additional 64-bit EUI-64 identifier, for a total of 128 bits. The GID is used for routing between subnets. The default GID prefix is 0xfe80::0. Finally, there is the local identifier (LID), which is assigned by the subnet manager. The LID is a 16-bit identifier that is unique within a subnet. Hosts have an LID between 0 and 48,000, usually expressed in hexadecimal notation (such as 0xb1). Routing within a subnet is managed by LID. The GUID, GID, and LID for a Linux server are stored in text files. The exact path to this text file will depend upon which Infiniband driver is used on the system. For a system with the MLX4 driver (such as RedHat/CentOS 5.x), the commands are:

Managing an HPC cluster or cloud infrastructure: alternatives to xCAT

Sun, 04 Nov 2012 19:01:15 +0000

xCAT is the eXtreme Cloud Administration Toolkit from IBM. It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC). I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5). Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster. In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good. There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru). If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems. We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach. For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense. The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools. I’ll list some of the key differences between the IBM and Aspen systems:

Mac OS X hack: download multiple files with curl

Mon, 29 Oct 2012 11:03:26 +0000

Updated 2017-08-01 to fix typo and use new demo URL. Also note that the Homebrew package manager has matured since this post was first published in 2012. Once you’ve configured Homebrew, installing a tool like wget is as simple as running brew install wget wget is an incredibly useful GNU tool on Linux. Unfortunately, it doesn’t come with OS X (as of Mountain Lion). OS X includes curl, which is a very handy tool but lacks at least one important feature of wget: the ability to use wildcards to get multiple files at the same time. For example, let’s say you want to download a subset of files from an FTP server. With wget, you could type:

An Improved Ebuild for OpenFOAM on Gentoo Linux

Mon, 22 Oct 2012 09:38:00 +0000

OpenFOAM is a notoriously difficult piece of software to compile, install and run. OpenCFD (the authors of OpenFOAM) have chosen to require the use of recent versions of gcc that are not available on most stable enterprise-class systems (ie Red Hat Enterprise Linux). To make things worse, until recently, OpenCFD also bundled a large number of libraries and helper applications (like VTK and ParaView) with the OpenFOAM source instead of using libraries and tools that are already on the system. Fortunately, OpenCFD has now moved the extra tools to a separate tarball, and the wizards at Gentoo have managed to create an ebuild for OpenFOAM. This is why I run Gentoo on my desktop workstation!

Run DSSP with GROMACS 4.5.5 on Red Hat Linux

Mon, 15 Oct 2012 10:00:50 +0000

UPDATE

A new version of GROMACS (4.6 series) has been released since this post was written. Please try installing the latest version of GROMACS before attempting the steps in this post.

Fixing GROMACS

There is a bug in the do_dsspcommand in GROMACS 4.5.5 that prevents the analysis of secondary structure using DSSP . Attempting to rundo_dssp will result in a segmentation fault. The bug has been patched since 4.5.5, but this version has not been released (see this post on the GROMACS mailing list). To get the patched version, follow the instructions from this post.

Tricks for Writing XML with Python 3

Thu, 27 Sep 2012 16:35:49 +0000

I’ve added a Python 3 XML example to my Shocksolution_Examples repo on GitHub. The new example shows how to generate an XML file which functions as a template for building a GUI with wxGlade. However, this example should be helpful for anyone who needs to create XML files with Python. The full example is on GitHub, so I’m just going to highlight a few interesting snippets. Use the SubElement factory function to create a new Element instance and add it to an existing element. Here, I create an element called templatedata and add it to the root element. I then create another element called author and add it to templatedata.

Firefox in CentOS/Fedora/Red Hat Enterprise Linux can only access Google

Mon, 10 Sep 2012 03:54:24 +0000

The Problem

I ran into what was probably the strangest error I have ever encountered as a Linux admin. I created a fresh CentoOS 6 virtual machine (VM) using VirtualBox on my OS X Mountain Lion system. To my great surprise, I was only able to access Google and a handful of other web sites from Firefox in the VM. I assumed that there was an issue with the networking of the VM, but after several frustrating hours of checking my setup I realized that the network in the VM was working fine. I was able to ping any web site by name on the command line, so clearly DNS was working. I could navigate to <<www.google.com&gt>; and perform searches, but clicking on most links resulted in the browser waiting forever. I could navigate to any site by IP, although the sites didn’t always load correctly (such as centos.org).

Hack of the Day: running CFD-ACE+ on Gentoo Linux

Tue, 24 Jul 2012 18:50:49 +0000

CFD-ACE+ is a multiphysics and computational fluid dynamics (CFD) simulation tool that was originally developed by CFD Research Corp. and is now distributed by ESI Software. The only platforms officially supported by CFD-ACE+ are Red Hat Enterprise Linux, SUSE Linux and Windows. Fortunately, it seems that ACE+ runs on other Linux distributions with only a little hacking. I just installed and tested CFD-ACE+ successfully (albeit not very thoroughly) on an up-to-date Gentoo Linux system. The process will require hacking some config files to build an external library from source. I started by downloading the files intended for Red Hat 6. Each package is a strange self-extracting executable file that unpacks itself when run. They also provide a C-shell script called install.com that automates the install process. The install script is very well behaved in that it doesn’t try to spew libraries and binaries throughout your filesystem, and you can run it as an ordinary user and install the file in your home directory. The tricky part about the installation is that the unpacking software (which is bundled into the self-extracting executable) needs the library selinux.so.1 in order to run. That library isn’t normally installed on a Gentoo system unless you’ve specifically set up SELinux. In order to avoid the potential problems of installing this library in my system paths, I created srcandincludedirectories in my home directory. I downloaded source tarballs for libsepol and libselinux from the download page of the selinux project. First, build libsepol usingmake. Then, copy the libsepol.so.1 binary from the src directory to the lib directory you just created in your home directory. Copy the include/sepol directory with all of its contents to the include directory in your home directory. When you are done, you should have:

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

Mon, 02 Jul 2012 17:14:54 +0000

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

How to dump/print/display/show UEFI configuration for IBM hardware

Thu, 07 Jun 2012 22:00:48 +0000

Newer enterprise-class hardware is starting to use UEFI instead of BIOS. UEFI is a big improvement, but there isn’t much documentation on the subject. Today I had to configure the UEFI for a new node in our cluster. I wanted to configure it exactly like an existing node that works. It was challenging just to find out how to display the configuration of the old node without rebooting the node and paging through dozens of configuration screens with the keyboard. Then I discovered IBM’s Advanced Settings Utility software, which fortunately was already installed on our cluster. It appears to be free for download, but with a restricted license. To display the complete UEFI configuration for a server, ssh to the server and run

Mini Maker Faire Orlando Report: Electronics

Sun, 03 Jun 2012 19:28:31 +0000

In my previous post about the Mini Maker Faire Orlando, I described some of the cool hardware that was on display. In this post, I’m going to describe some of the electronics available from local vendors.

Electrimod

Electrimod MiniStack

Local vendor Electrimod (Clermont, FL) was on hand to showcase their pluggable modules for PIC microcontroller development. Basically, Electrimod is developing products that are equivalent to Arduino shields. You start with the PIC module, and then plug in whatever other modules you need to build your prototype in the form of a stack. These brand-new products aren’t available yet, but they should be shipping by June or July 2012.

Mini Maker Faire Orlando Report: Hardware

Sun, 27 May 2012 02:20:43 +0000

I had a lot of fun at the Orlando Mini Maker Fair today! In this post I’m going to highlight some of the hardware vendors who took the time to come out and show their stuff to the community.

80/20: The Industrial Erector Set

80/20 is a company that makes modular aluminum framing systems. Their motto is “The Industrial Erector Set,” and it’s a good description of their product. Our local distributor is Florida Motion and Control. If you ever might need to build any kind of machine or robot, you need to browse the 80/20 web site and bookmark it. I have never used it, but I am desperately looking for a project that will require this!

Benchmarking BLAS Part 1: Building BLASbench with gcc

Fri, 25 May 2012 22:37:22 +0000

I recently installed ATLAS on my new workstation and I’ve been wondering what kind of performance gain I would get compared to the reference BLAS. After some searching, I finally found a BLAS benchmarking tool called BLASbench, which is part of the LLCbench suite. This tool has a rather strange build process, so follow the instructions on the web page. You have to create a file called sys.def which contains build instructions for your system. Here’s the sys.def file I used for my Gentoo system:

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

Tue, 24 Apr 2012 16:40:40 +0000

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL). This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

We won Startup Weekend Orlando!

Sat, 07 Apr 2012 17:27:16 +0000

Our team (Mole Shopper) won Startup Weekend Orlando! It was a great experience, and I highly recommend that anyone who is interested in entrepreneurship attend a Startup Weekend event. An MBA might tell you about how business is supposed to work, but launching a business in 52 hours will show you how business _actually_works. We owe a great debt to the volunteer mentors that helped us formulate our business plan and refine our pitch. We won $5,000 in seed funding from Startup Orange County (Florida), three months of incubation at the UCF Venture Lab, 4 months of internet marking and social media consulting by Pro2Go Designs, 2 months of legal services by The Entrepreneurship Law Firm, a $500 credit for VoiceBunny and a subscription to Startup Threads.

Partial derivative notation in Sage

Tue, 06 Mar 2012 18:36:26 +0000

This post will explain the notation used to denote partial derivatives in the output from Sage. It’s confusing at first, but a simple example will make it clear. Here is the input to Sage:

var('x y')
function('test', x, y)
show(test(x, y))
show(diff(test(x, y), x))
show(diff(test(x, y), x, 2))
show(diff(test(x, y), y))
show(diff(test(x, y), y, 2))

I defined a function of two variables called test. When you run this in the notebook interface, you get:

Publication-quality plots with Excel

Tue, 28 Feb 2012 21:36:11 +0000

Microsoft Office tools are not the best choice for scientific publications, but sometimes you have to use them. Some journals actually require Word documents, and many of my collaborators simply don’t do LaTeX. The process outlined below is they best way that I’ve found to make high-quality plots and charts with Microsoft Excel. High-quality graphics with consistent size, aspect ratio, typeface, and font size will make your publications look much more professional. You can do this with Excel, but it requires some effort. You will need to install the free Bullzip PDF Printer before you begin. 1. Create and format the chart correctly in Excel. It’s better to insert a “floating” chart in an existing worksheet, as shown below, rather than creating a new worksheet that contains only the chart. Set the size of the chart by right-clicking near the edge of the chart and selecting Format Chart Area. Then select Size and set the actual size and aspect ratio that you want (try 3 inches by 4 inches). If you want a set of plots to have consistent formatting, it is very important that they are the exact same size in Excel! If the finished plots must be different sizes but have the same formatting, make the outer dimensions for all the charts big enough to accommodate the largest plot. Then shrink the plot within the frame, leaving whitespace which will be cropped out at a later step. Also set correct font sizes, line widths, colors, etc.

Installing PETSc and libMesh on CentOS 5

Fri, 20 Jan 2012 20:06:18 +0000

Prerequisites: boost-devel, openmpi-devel, valgrind-devel I installed PETSc and libMesh in my user directory, since I have a single-user workstation.

Installing PETSc for use with libMesh

First, ensure that OpenMPI is installed and the system paths have been configured correctly. You will need to add the directory containing OpenMPI binaries to your $PATH, and the directory containing OpenMPI libraries to $LD_LIBRARY_PATH. CentOS does not do this automatically! Type mpicc on the command line to make sure the shell can find the binary. If mpicc and mpif77 run, then you should not need to use the comnand-line flags to the configure script for PETSc. If you are going to use libMesh with PETSc, you need to enable C++ with a command-line option to the configure script. Here is the process I used (it might be a good idea to set PETSC_ARCH and PETSC_DIR in your .bashrc so they are consistent for all builds).

Paraview 3.12 on 64-bit CentOS 5.7

Thu, 19 Jan 2012 19:23:15 +0000

I finally got around to installing Paraview on my CentOS 5 box. There are no official RPMS for CentOS 5, so I was expecting a painful build process. To my great surprise, I was able to download the Paraview binary for Linux (64-bit) from Kitware, unpack the archive, and run it! As root, I placed the entire ParaView-3.12.0 directory under /opt/Paraview. You could also keep it in your home directory if you are running a single-user system. I added /opt/Paraview/ParaView-3.12.0/bin to the $PATH in the .bashrc file in my home directory. Now I can type paraview at the command line and it runs! What if you want Paraview to appear in the Applications menu in your desktop environment? Most modern desktops (I use XFCE4) construct the Applications menu “on the fly” based upon the files in a standard directory (/usr/share/applications on CentOS). The Free Desktop Project has created a standard for desktop entry files. You may also find this summary of the standard to be helpful. To add Paraview to the menu, you simply need to create a new file in the standard location. If you installed Paraview in your user directory, you may want to place the desktop file in $HOME/.local/applications. Here are the contents of a file I called paraview.desktop:

Portfolio

Sat, 31 Dec 2011 19:29:18 +0000

Past Research and Development Projects

Used nonlinear optimization tools to develop signal processing devices Modeled the effect of parasitics from bond wires and packaging on the performance of radio-frequency devices, using a combination of lumped-element and finite-element models Created a reduced-order model of the accuracy and penetration performance of armor-piercing projectiles, for use in a OneSAF tactical simulation Developed a basic flight-training scenario for the Orlando Executive Airport, including custom 3D scenery objects

Installing Sage 4.7 on CentOS 5

Mon, 07 Nov 2011 23:07:59 +0000

I recently upgraded my desktop workstation from an old 32-bit version of Gentoo to 64-bit CentOS 5. I downloaded and installed the latest version of Sage, and the process went smoothly. If you find this post helpful, please check out the Sage Beginner’s Guide at Packt Publishing. Since I don’t use Sage every day, I actually refer to my own book on a regular basis!

Download

Since CentOS is designed to be binary compatible with Red Hat Enterprise Linux, the correct binary to download is sage-4.7.2-linux-64bit-red_hat_enterprise_linux_server_release_5.6_tikanga-x86_64-Linux.tar.gz

Python string format examples

Thu, 03 Nov 2011 22:38:10 +0000

The format method for Python strings (introduced in 2.6) is very flexible and powerful. It’s also easy to use, but the documentation is not very clear. It all makes sense with a few examples. I’ll add more as I have time:

Formatting Numbers in Python Strings

Formatting a floating-point number

[code language=“python”] “{0:.4f}".format(0.1234567890) “{0:.4f}".format(10.1234567890) [/code] The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field” 0 indicates the first argument to method format : indicates the start of the format specifier .4 indicates four decimal places f indicates a floating-point number

Linux kernel 3.1 config for Gentoo guest on VMWare Fusion

Mon, 24 Oct 2011 03:23:48 +0000

The following kernel configuration was posted by Alessandro Di Marco as a comment on a previous post in which I provided a Linux kernel config for an earlier version of the kernel. In the interest of sharing his contribution, I’m placing it in its own post. I don’t know the author personally and I have not tried this config. Like any other free software, you are using it at your own risk and neither Alessandro nor I provide any warranty. Download the file from the link below, change the file extension from .txt. to .config and load the config into one of the standard kernel configuration tools (such as make menuconfig) and look over all the options before using it. Gentoo Kernel 3.1 Config

Alternate implementations of Python

Sun, 09 Oct 2011 20:58:20 +0000

This is a “skeleton” of a page that I want to write about various versions of Python that are optimized for various applications. However, I now recommend that you just check out this article about all the different implementations of Python (CPython, Jython, IronPython, PyPy, etc.) by Charles Marsh. Stackless Python is an implementation of Python that adds lightweight microthreads with scheduling and interthread communication. It’s aimed at concurrent programming. Note that Stackless microthreads do not take advantage of multicore or multiprocessor architectures. The threads all run within one Python process. Jython is an implementation of the Python programming language that runs on the Java Virtual Machine (JVM). PyPy is an alternative Python interpreter than incorporates a just-in-time (JIT) compiler that optimized bytecode to optimize performance. Cython is a language that allows you to write C extensions for Python in a language that is very much like Python.

TeXLive and Asymptote on CentOS 5

Mon, 03 Oct 2011 18:18:50 +0000

Tex Live

For reasons unknown, a TeX Live package is not available for Red Hat Enterprise Linux/Centos 5 from the major repositories (EPEL or DAG). I consider this to be a glaring omission, since TeXLive is a great improvement upon teTeX. Since I don’t have time right now to package it myself, I installed TeX Live manually in my user directory and it’s working fine. I used the network install process, which starts with downloading a command-line installer and then following the detailed installation instructions with the base path set to $HOME/texlive/2011.

How to install Octave video tools on CentOS 5

Wed, 14 Sep 2011 15:57:44 +0000

This post describes how to get GNU Octave up and running on a CentOS 5 Linux system for use in reading, processing, and writing video files. You will need to use the EPEL and DAG/rpmforge repositories, but I won’t explain how to do that here. NOTE: I had to go through some trial and error to get this working. I tried to summarize only the necessary steps, but I can’t guarantee I got it completely right until I try a fresh install on another system which does not have any of the dependencies already installed. Please leave a comment if you encounter any problems. As a superuser, use yumto install octave. You need octave-devel and ncurses-devel in order to install any Octave packages. You will need to have the EPEL repository enabled: yum install octave octave-devel ncurses-devel As a superuser, useyum to install ffmpeg and ffmpeg-devel from the rpmforge repo. The video package for octave uses the ffmpeg libraries (avilib) to perform the actual video processing. You will need the ffmpeg-devel package, because Octave builds the video package from source.

How to build ScipPy with Python 2.7.2 on CentOS5

Thu, 25 Aug 2011 00:17:57 +0000

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path. This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib). EDIT 2: you should use the –prefix=/home/yourusername instead of –user. The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages. The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up! I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5. I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system. Here is how I did it. First, with superuser privileges, use yum to install the following packages. You may need to set up epel as an alternate repository:

Parsing INI configuration files with FORTRAN

Tue, 23 Aug 2011 21:14:44 +0000

Fortran would not be my first choice for working with text, in any form! However, sometimes even numerical codes need to read data from configuration files. The easiest way to read a configuration file from a Fortran 90 routine is by using namelist I/O (I really need to add an example of that). If you’re stuck with INI files, I found an INI file parser written in Fortran buried in an index of free Fortran routine. I don’t even remember how I even found it, since it’s not well labeled and doesn’t come up in the first few pages of Google results, so I thought I’d better write a post about it in case I ever need such a thing in the future.

Visualization

Tue, 23 Aug 2011 03:14:18 +0000

Open-source visualization tools

This list is not exhaustive, and I haven’t used all these tools. Hopefully, you’ll find something useful.

asymptote chaco CDAT (Climate Data Analysis Tools) ImageVis3D (volume rendering) matplotlib (html 5 canvas) tikz/pgf (plotting for TeX) VisIt VisTrails (a workflow tool that utilizes other visualizers)

VTK (Mayavi, Paraview)

LaTeX

Sun, 21 Aug 2011 03:52:06 +0000

LaTeX is the the best way to create technical documents. I’m using it to write my dissertation at the University of Central Florida (I have already used it to write my Master’s thesis). This time, I’m writing my own document class. The existing unofficial UCF class uses a lot of raw TeX to achieve its formatting goals. It’s hard to read and does not work well with modern LaTeX packages. My new document class is much cleaner and simpler, uses no TeX, and requires only standard LaTeX packages that come with every distribution.

Removing an axis or both axes from a matplotlib plot

Wed, 17 Aug 2011 22:40:14 +0000

Sometimes, the frame around a matplotlib plot can detract from the information you are trying to convey. How do you remove the frame, ticks, or axes from a matplotlib plot?

matplotlib plot without a y axis

Some books you may find useful when working with matplotlib:

The full example is available on github.

First, we construct a figure and an axes object:

fig1 = plt.figure(facecolor='white')
ax1 = plt.axes(frameon=False)

The Axes object is a container that holds the axes, the ticks, the labels, the plot, the legend, etc. You will always have an Axes object, even if the axes are not visible! The keyword argument frameon=False turns the frame off. An alternative method is:

A git branching strategy suitable for large projects

Sat, 13 Aug 2011 21:57:14 +0000

Git is an amazing tool…but what is the best way to use it? Like any tool that gives you great power and flexibility, it’s up to you to use the tool in the best way to suit your purpose. A friend of mine who manages an enterprise-class software development team recommended the git branching strategy explained in this blog post. I highly recommend reading it.

Running Sage on Windows with VirtualBox

Thu, 28 Jul 2011 21:40:40 +0000

The easiest way to run Sage on Windows is with a virtual machine. Sage is a collection of tools that were primarily developed for UNIX-based operating systems (such as Solaris or Mac OS X) or UNIX-like operating systems (BSD, Linux, etc). Since a standard Windows installation lacks many of the features provided by a UNIX-like environment, Sage is packaged as a Linux virtual machine. You need free virtualization software–VMWare Player or VirtualBox--to run the virtual machine. I find VMWare Player easier to work with, but it’s not an open-source solution. VirtualBox is truly free software, but it’s also harder to configure. When you download Sage, you can choose between a .ova file (for VirtualBox) or a zipped VMware virtual machine. The easiest thing to do is choose the appropriate format for your VM. However, it’s also possible to run the VMware image on Virtualbox, and I’ll show you how. If you want to learn more about Sage, check out the Sage Beginner’s Guide from Packt Publishing. Download, install, and run VirtualBox, and then click the “New” button to get started. Also, unzip the .zip file that contains the VMware virtual machine.

VirtualBox Manager

Numbering equations in Word 2007

Tue, 12 Jul 2011 20:00:35 +0000

Microsoft Word isn’t the best software for technical writing, but sometimes we have to use it. Here is the best way I’ve found to emulate the automatic equation numbering that LaTeX has been doing for years:

Insert an equation (old-style)

Use the Object button on the Insert tab to bring up the dialog box. Choose Microsoft Equation 3.0 and enter an equation.

Insert equation

In Word 2007 and later, you can also use the new style of equations, but you’ll need to use a different method for numbering the new equations. Be warned that the new style is not compatible with Powerpoint 2007 or Word for Mac, which is why I’m not using them yet.

Python: lists to tuples and tuples to lists

Sat, 02 Jul 2011 17:07:00 +0000

Get the code for this example The zip function in Python accepts a number of sequences (such as lists or arrays) and returns a list of tuples. The first element of each tuple comes from the first sequence, the second from the second sequence, and so on. For example: [code language=“Python”]a = range(0, 5) b = range(5, 10) c = range(10, 15) sequence_of_tuples = zip(a, b, c) print(sequence_of_tuples)[/code]

[(0, 5, 10), (1, 6, 11), (2, 7, 12), (3, 8, 13), (4, 9, 14)]

Blog infrastructure updated

Fri, 24 Jun 2011 15:15:06 +0000

I’m almost finished with a round of updates to the infrastructure of this site. I upgraded Wordpress and changed my hosting from a friend’s virtual server to Linux hosting with GoDaddy.com. The price is right and so far the service has been great. I’ve used GoDaddy for domain name registration for many years. [ad#GoDaddy Hosting] Comments are now protected with reCaptcha to try to reduce the amount of link spam that I have to sort through. If you have a problem posting a comment, or notice any other aspect of the site that is “broken,” please contact me at the email address shown on the About page. I still have a bit of work to do on updating my theme to take advantage of the features of Wordpress 3.

Finding dimensions of a bounding box in Jmol

Mon, 13 Jun 2011 20:22:35 +0000

The Jmol applet (or Java application) is widely used to visualize the structure of molecules. It is very powerful, but not that well documented. If you don’t have Jmol running, open the page for Glucose Oxidase (1CF3) from the Protein Databank so you can follow the instructions for finding the size of the bounding box. Right-click on the applet to get a pop-up menu with lots of options. Under the Stylesubmenu, click onBoundboxto show a bounding box around the molecule:

Show bounding box

Sage Beginner's Guide: Code and Errata

Sat, 04 Jun 2011 18:50:59 +0000

Sage is a free open-source mathematics software system. It’s a GPL alternative to commercial software such as Magma, Maple, Mathematica and MATLAB. Sage is a research-grade tool, cited in nearly 100 academic research articles (with dozens more in press). I’ve written a book to help engineers, scientists, and students get started with Sage. It’s available in print and e-book formats. Please check it out!

Code

I am aware that there are a number of issues with the code bundle that is available on the Packt site, but I have not been able to get the publisher to replace it.

Photography

Sat, 14 May 2011 21:26:28 +0000

These are links to places where my photos have appeared online or in print:

Steam tractors on Dark Roasted Blend Stage lighting article for Technologies for Worship Magazine (also in print) Schmap.com Guide to the Florida Keys (one of several photos of Mallory Square) The Center for Educational Technology in Israel used a photo of a cell phone tower disguised as a tree in a geography book for 7th and 8th grade students

IGES file reader

Thu, 14 Apr 2011 18:26:47 +0000

I am creating a Python library for reading CAD files in IGES format. I have barely started on the project–so far, it should be able to parse arbitrary IGES files without crashing, and it can extract useful information for line segments and rational B-spline curves. That’s all I need to get my work done right now, so I don’t feel justified in spending too much more “company time” on this. I am hoping that other people will help me by adding Python classes to represent the types of entities that they need.

CAD/CAM

Thu, 14 Apr 2011 18:26:11 +0000

Projects related to computer aided design/drafting (CAD) and computer aided manufacturing (CAM).

Replacing text in place with GNU sed

Mon, 11 Apr 2011 17:39:38 +0000

sed is a stream editor, which means that it accepts a stream of text, processes it, and spits out another stream of text. sed can process files that are too large to load into memory, and it is a completely command-line tool that can easily be integrated into shell scripts. This excellent sed tutorial was written for an old version of sed provided by Sun Microsystems, and it doesn’t cover one of the most useful features of GNU sed. GNU sed accepts a -icommand line argument that tells sed to replace the text file in place, rather than writing the output stream to another file. Another nice feature is thatsed -i will create a backup file before processing if you provide a backup suffix:

A self-contained Fortran linear equation solver

Wed, 16 Mar 2011 21:33:27 +0000

I’ve just released a self-contained Fortran module that solves a system of linear equations using the LU decomposition. Download the Fortran linear solver from github This module is based on code that was implemented and released on the Web by Jean-Pierre Moreau. His implementation was based on one of the Numerical Recipes books. I updated his code to a more strict Fortran 90 standard and added the necessary comments so that it can be built as a Python module using f2py. I replaced Jean-Pierre’s Fortran test program with a simpler, self-contained program. I also included a Python script that implements the same test case. I created this module because sometimes a self-contained routine is more appropriate than a full library. I am compiling a library that implements a custom boundary condition for a proprietary computational fluid dynamics solver (CFD-ACE+). The library has to be written in Fortran, and it has to be built using a proprietary set of build scripts. I could either try to reverse-engineer the build process and to modify it to link to a shared library like LAPACK, or I could implement a self-contained solver. Since Pierre had already implemented the solver, I was able to slightly modify his code and get it working relatively quickly.

Finding a value in an unordered Fortran array

Fri, 11 Mar 2011 16:27:51 +0000

I have been optimizing some Fortran code that involved searching for an integer value in an unordered array (we know the value occurs only once). Since there is no intrinsic procedure to accomplish this, I thought I’d try a couple of approaches to see which was fastest. The simple answer is that, in this case, brute force beats elegance, even when the target value is near the end of the array. Download the full example from GitHub

Sage: open-source mathematical software

Mon, 07 Mar 2011 14:07:19 +0000

I’ve recently gained a lot of experience with Sage, an open-source alternative to MATLAB, Mathematica, Maple, MuPAD, and Magma. Here are a couple of links to check out:

Public notebook servers--try it online right now! Interactive examples with Sage

Mathematical publications citing Sage

Sage vs. Mathematica

My experience with mathematical software started my freshman year at the University of Illinois when I signed up for a calculus class that was taught almost entirely with Mathematica. I grew to love Mathematica’s symbolic computation and plotting capabilities, although I found its programming language to be cumbersome. Once I was no longer a student, Mathematica was no longer an option due to the hefty licensing fees. With the caveat that I haven’t used Mathematica in many years, I will say that Sage compares very favorably to my experience with Mathematica. Sage has a notebook interface that allows you to integrate code, results, text, typeset equations, and graphics in an interactive document that can be viewed with any standards-compliant web browser. I don’t know how Sage stacks up against Mathematica in areas of advanced mathematics. I have heard that Sage is far ahead of Mathematica in number theory, because the lead developer of Sage is a number theorist.

Moving example code to GitHub

Fri, 04 Mar 2011 18:09:32 +0000

As this site has grown, the example code has gradually become unmanageable. I’ve posted snippets and fragments here and there over the years, and the original code is scattered in various locations across several computers. Further, as people have pointed out bugs or ways to improve the examples, I’ve had a hard time making changes. This is problem is crying out for version control with a central repository. Since I’ve been using Git for the past year or so, I decided to try GitHub. So, I am gradually moving all the examples from the site to: https://github.com/cfinch/Shocksolution_Examples When I put code in a post, I will provide a link to GitHub where you can browse or download the code. You won’ t need to use Git, or even learn anything about it.

Linear system simulation with Python

Fri, 17 Dec 2010 21:33:16 +0000

Linear time-invariant (LTI) systems are widely used in the field of signal processing. Scipy contains powerful tools for simulating LTI systems in the scipy.signal package, but they are not well documented. I will provide a simple example that demonstrates how to use a few of the core classes and functions in scipy.signal for simulating LTI systems with Python.

Define an LTI system

You will need to have Scipy installed, and you will need to have Matplotlib as well to make plots. We will start with an example of a first-order LTI system, which is characterized by a single parameter known as the time constant. scipy.signal defines the class lti to represent a linear system in Python.

Using Microsoft Word for Technical Documents

Wed, 18 Aug 2010 23:30:34 +0000

Microsoft Word is not the best tool for doing technical writing. However, sometimes we are required to use Word because we need to collaborate with others who want to use Word. Right now, I am using Word 2007 on Windows XP to write several mathematical papers. In general, it is a big improvement from previous versions. The new integrated equation editor is outstanding–except for the major bug I’ll discuss below. Here is a brief “FAQ” you will want to read if you are using Word for technical writing. **Q:**How do I enter multi-line equations in Word 2007?**A:Press Shift-Enter where you want a line break to appearQ:**How do I prevent a page break from splitting a table into two parts?**A:**Select the table. On the Home tab, click the little box in the lower-right corner of the Paragraph box. On the paragraph dialog, choose the “Lines and Page Breaks” tab. Check the box for “Keep Lines Together.”**Q:**How do I automatically number headings in Word 2007? For example: Section 1, Subsection 1.1, 1.2, 1.3, Section 2, etc.**A:**It’s not obvious. See this page from dummies.com about number headings.**Q:**In Word 2007, why do equations sometimes appear as blank spaces or question marks when I print or save my document as a PDF file?**A.**This occurs when Word is installed on Windows XP Pro. See the following Microsoft tech support item to find out how to install missing scripts: The characters in an equation are not printed… You may also have an outdated printer driver: Microsoft Support Item 920228**Q:**Why do equations created in Word 2007 disappear when I open the document in Word 2008 for Mac?A. Word for Mac does not support equations written in the new Word 2007 equation editor. Unfortunately, neither does PowerPoint 2007 on the PC. You can work around this by inserting equations into Word the old-fashioned way: go to the Insert tab, click on Object (found in the “Text” box towards the right side of the tab), and choose “Microsoft Equation 3.0” from the list in the dialog box.

Reading an array from a text file with Fortran 90/95

Fri, 21 May 2010 20:59:26 +0000

If you’re used to coding in more modern languages, Fortran I/O can seem a little bizarre. Strings in Fortran are much more difficult to work with, since they are fixed-length rather than null-terminated. The following example illustrates a simple way to read an array of numbers from a text file when the array length is unknown at compile time.

program io_test
 real, dimension(:), allocatable :: x
 integer :: n
 open (unit=99, file='array.txt', status='old', action='read')
 read(99, *), n
 allocate(x(n))
 read(99,*) x
 write(*,*) x
end

Here is the text file that the array is read from. The integer on the first line is the number of elements to read from the next line.

Managing a pool of MPI processes with Python and Pypar

Sat, 17 Apr 2010 18:52:13 +0000

MPI is a standard for communication between multiple processes in parallel computing. These processes can be running on different cores, CPUs, or entirely different computers in a grid. MPI is a standard, and there are many implementations available (many are open source). The features of MPI can be accessed from Python with the packages pypar and mpi4py. Here I present a Python script that implements a “process pool” design pattern using pypar. I have observed a pattern often enough in my own work that I wrote this framework to avoid reinventing the wheel every time I come across it. This pattern is useful for any embarrassingly parallel problem. This describes a computing task that can be easily accelerated by running multiple parallel processes that do not not need to interact with one another. For example, I have large scientific data sets from several runs of an experiment that need to be analyzed. Since the data from each run can be analyzed independently from the other runs, I can analyze all the data sets at once on a parallel machine. The code below implements a “master-worker” paradigm that requires at least three processes to accelerate the calculation. The first process be comes the master, which does no calculation but hands out tasks to the workers. The rest of the processes are workers, which receive a chunk of work, finish it, return the result to the master process, and then wait for more work.

Boost.python Numpy example

Wed, 14 Apr 2010 17:28:45 +0000

Before recent advances such as Cython, Boost.Python was one of the best ways to integrate C++ code into Python programs. The following C++ source code exposes a C++ function to Python. The function takes a NumPy array as an argument and extracts a C++ integer type.

#include
#include "boost/python/extract.hpp"
#include "boost/python/numeric.hpp"
#include
using namespace boost::python;
// Functions to demonstrate extraction
void setArray(boost::python::numeric::array data) {
 // Access a built-in type (an array)
 boost::python::numeric::array a = data;
 // Need to array elements because their type is unknown
 std::cout << "First array item: " << extract(a[0]) << std::endl;
}
// Expose classes and methods to Python
BOOST_PYTHON_MODULE(TestNumPy) {
 boost::python::numeric::array::set_module_and_type("numpy", "ndarray");
 def("setArray", &setArray);
}
```---

## Comments

### Comment #1 by ARAMBAM JAMES SINGH

[ARAMBAM JAMES SINGH](mailto:<jamesastrick@gmail.com>) - Aug 2, 2016

I get this error : import TestNumpy as tnp ImportError: dynamic module does not define init function (initTestNumpy)

---

#### Comment #2 by James

[James](mailto:<jameastrick@gmail.com>) - Aug 0, 2016

Hi , I get the error "Segmentation fault: 11" when i run the python code after compiling above c++ code #--------------- # import TestNumPy as nt import numpy as np a = np.array(\[10,20,30\]) nt.setArray(a)

---

Linking Python and C++ with Boost.python

Wed, 14 Apr 2010 15:59:56 +0000

Introduction to Boost

Boost is a project to create open-source libraries for C++. There are a lot of nice tools available through Boost, one of which is Boost.python. Boost.Python provides seamless interoperability between C++ and Python. I have found it to be an extremely useful tool for scientific programming. Parts of a program that require more flexibility than speed can be written in Python, and critical portions can be coded in C++.

Constrained least-squares fitting with Python

Fri, 05 Mar 2010 19:27:25 +0000

Scipy contains a good least-squares fitting routine, leastsq(), which implements a modified Levenberg-Marquardt algorithm. I just learned that it also has a constrained least-squared routine called fmin_slsqp(). I am using simple upper and lower bound constraints, but it’s also possible to specify more complex functional constraints. What I did not realize, at first, is that fmin_slsqprequires a different type of objective function thanleastsq. leastsqrequires you to write a function that returns a vector of residuals, andleastsqautomatically squares and sums the residuals.fmin_slsqp is actually more flexible, in that it can use any objective function that returns a single scalar value. To implement least-squares curve fitting, your objective function will need to find the residual at each data point, square the values, and sum them up. Hopefully this tip will save you some time. Check out the scipy optimization tutorial for more examples. Here is the original paper by Dieter Kraft which introduces the algorithm used by fmin_slsqp.

A preliminary review of the Lehigh Rendition console

Sun, 17 Jan 2010 17:35:09 +0000

In my last post, I explained why I don’t think the ETC Element is a good replacement for the ETC Express. When I was at LDI 2009, I ran across the Rendition series of consoles from Lehigh Lighting. Based on the literature I picked up, and the extensive demonstration that I received from the Lehigh representative, it looks like the Rendition might be the true heir to the Express. It is targeted at a similar market: small theaters, churches, and schools. The physical layout will be familiar to anyone with theater console experience: 24 or 48 submasters on the left, dual cue playback controls in the center, and hardkeys on the right, with channel faders (48 or 96) along the upper right portion of the desk. The submasters are traditional theater-style subs. Each sub records a fixed look rather than an independent cuelist. Shows can be saved to a flash drive via a USB port. While the console hardware is finalized, several additional features will be added in an upcoming version of the operation system. The console currently supports one external monitor, but dual monitors will soon be supported. An Ethernet port is already included on the console. In a future software release, it will be possible to add wings with additional submasters or channel faders. The advantage of Ethernet over USB is that a wing can be placed hundreds of feet from the console, and connected using existing Ethernet lines. This feature would allow a wing to to be used as remote focus unit. Using a wireless Ethernet bridge opens up even more possibilities. The main difference between the two models, the 24/48 and 48/96, is the number of hardware faders. The software capabilities are identical. The price of the console depends on how many channels of conventional dimming are “unlocked.” You can choose from 125, 250, or 512 conventional channels. All models support 1024 channels of moving-light control. The two hardware DMX ports each support 512 channels, but two additional universes can be accessed via Ethernet (I’m not sure of the details on how this works). The console seems to support all the basic and advanced cue and effect functions that you’d expect: multiple cue lists, flexible cue timing options, macros, subroutines, and effects. I can’t really comment further about that, since I would need to spend a lot of time with the console to accurately gauge its reliability and ease of use. Moving lights are also supported, with fixture profiles for easy patching and a trackball for focusing. Once again, it looks promising, but the only way to really evaluate these features is to try to use them and see how intuitive the process is. Download the offline editing software and try it for yourself. A more sophisticated console called the Rendition Pro is expected to be available in the second quarter of 2010. The Pro is aimed at more experienced users, and appears to compete with the ETC Eos and Ion. It will run the same software as the Rendition, so many of the features that are expected to be added to the Rendition are actually being developed for the Pro. The layout is similar to the Rendition, with playback masters on the left side, traditional cue playback controls in the lower center, and keys to the lower right. Two LCD displays with encoder wheels and softkeys occupy the space above the keypad, with no hard channel faders. Further, the submasters have LCD displays and additional buttons–probably “Go” and “Pause” buttons to support a cue stack on each sub. From my limited experience, both of these consoles series are worthy of further investigation. They seem to be solidly built and well-engineered. The Lehigh personnel that I spoke to were helpful and friendly, and I appreciate their assistance in answering my questions.

Storing large Numpy arrays on disk: Python Pickle vs. HDF5

Sun, 10 Jan 2010 23:00:20 +0000

In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: [code language=“Python”]#!/usr/bin/env python from numpy import array

Python Basics: Tutorials and Examples

Mon, 21 Dec 2009 21:23:09 +0000

If you’re new to Python and wondering how to use some of its features, you may find something useful in the following posts:

Storing data with Pickle Tools for Python software development Redirecting standard output from Python Python threads example

My article published in Technologies for Worship Magazine

Tue, 15 Dec 2009 04:49:09 +0000

Head over to Technologies for Worship Magazine and check out my recently-published article about backlighting. It’s my first publication in print (outside of technical journals). I’m looking forward to writing more articles for a general audience!

First impression: ETC Element Console Review

Wed, 09 Dec 2009 04:12:04 +0000

We recently got an ETC Element to replace an ETC Express 72/144 that had become unreliable due to a hardware problem. I’d like to point out that ETC equipment is generally very durable, but this particular console only lasted about 8 years due to abuse by the high-school students who use the console during the week. The Express was a great console. It had a hardware fader for each of the channels that we commonly used, and a bank of 20 submasters. Recording a submaster was as simple as creating a look onstage with the channel faders, then pressing “Record” and hitting the submaster bump button. It had one external LCD screen for output, which had a very simple character-based interface. Just about anyone could learn to use the Express. The Element is, in many ways, a much more sophisticated console. While the Express didn’t have any real capability to run moving lights, the Element has many features to facilitate the use of moving lights and programmable LED fixtures. It supports two external LCD screens with a full graphical interface, a keyboard, and a mouse. Unfortunately, the Element has lost all of the simplicity that made the Express so successful in its target market. Let’s start with the physical design. The greatest disadvantage of the Element, especially for untrained users, is that all of the forty faders (60 in some models) are multi-function. With the bank switch in position 1, the faders represent channels 1-40. Position 2 changes the fader function to channels 41-80, and Position 3 changes the faders to channels 81-120. Position 4 changes the fader function to submasters. While it’s nice to have 40 subs, it’s also extremely confusing to figure out which channel fader or submaster is currently controlling the output. Here’s an example. Let’ s say that you use faders 2, 17, and 31 to set a look on Bank 1. If you switch to Bank 2, faders 2, 17, and 31 are still up, but they are not controlling anything. Bring those faders down, and continue setting up the look on channels 41-80 using Bank 2. Now switch back to Bank 1 to tweak the look on channels 1-40…but the faders are no longer up! The LEDs on the bump buttons will flash to show which channel levels no longer match the fader positions. To take control of these channels, bring the fader up until it matches the current channel level. Then you can control the channel again. Are you confused yet? Did I mention that the faders have no numbers? I find this slows me down when creating a look, and makes it harder to edit a look “on the fly.” To top it off, the bank selector is a little plastic twist switch, and it’s hard to see which bank it’s pointing at. With many consoles, you can work around the limited number of faders by adding an expansion wing with additional controls. Unfortunately, the Element does not support expansions–you have to upgrade to the Ion to get that feature! I’m not thrilled with the layout of the keys on the Element keyboard, either. It’s difficult to guess where a key is going to be–they don’t seem to be grouped in any particular way. Compare to the Hog 1000, in which the keys are grouped as “verbs” and “nouns” for command-line programming. One final gripe about the hardware: it’s difficult to run my finger down the row of bump buttons to find a particular light–this was easy to do on the old Express. So, after all my complaining, I want to leave you with a great feature. I love the “Exclusive” submaster setting that prevents the output from a sub from being recorded in subsequent cues and submasters. I use this all the time to keep some lights up onstage or in the house without having them recorded into cues. I’m sure there are more little gems built into the Express. Here is my conclusion about the Element: it is NOT a replacement for the Express. Do not expect this console to be a “drop-in” replacement for any traditional theater console. The learning curve will be very steep for brand-new users, and slightly less steep for Express users. In the end, the Element will be far more capable than the Express–but that’s not the target market for the Express. If you are buying a console for a school, small church, or community theater where the users are generally inexperienced, there are probably better choices than the Element.

Search

Sun, 29 Nov 2009 18:24:50 +0000

LDI 2009 Report

Mon, 23 Nov 2009 03:15:54 +0000

I attended LDI (Live Design International) last Friday in Orlando, FL. This was my second time at the show (read my report from 2007). Overall, I’d say this year’s show was scaled back from 2007, which is not surprising considering how the economic recession has affected the entertainment industry. There were fewer booths this year, and they weren’t as fancy. A few manufacturers, such as Chauvet and Coemar, brought pretty spectacular booths. Other companies brought large booths (such as ETC and Martin), but they were not as flashy. I’m not sure if Friday is normally a slow day for this conference, but there wasn’t much of a crowd. I didn’t have to wait in line to see anything. In 2007, I went to LDI to try out control consoles to replace our Hog 1000. That plan fell through when the economy tanked, so this year I was just looking around. Apparently a lot of people were, because the sales people were rather aggressive about jumping onto any apparent interest in their product. One girl tried to scan my badge as I walked by–I don’t even know what she was selling! Actually, that’s a common occurrence at LDI–it’s hard to tell what the actual product is in the booth. All the larger booths seemed to consist of truss, truss warmers, banners, drapes, moving lights, and video. I often had to stop and look for a minute to figure out which one was actually their product! I will post some photos when I get a chance.

Building and linking to a shared Fortran library

Mon, 26 Oct 2009 22:34:02 +0000

I’m using GNU Fortran (gfortran) to build several shared libraries, and then dynamically linking to them from a Fortran program. The process is a little different than what I’m used to for C libraries, so I thought I’d explain it. Unlike C, there is no need to #include header files when compiling code that relies on functions defined in an external library. Likewise, there is no need to use -l or -L linking flags to tell the linker about s hared libraries (at least when they’re in the same directory). In fact, the whole process requires a lot less command-line options than I had expected.

X.org configuration for a Gentoo guest on VMWare Fusion

Thu, 22 Oct 2009 03:25:37 +0000

Here is an xorg.conf for a Gentoo guest running in VMWare Fusion on MacOS 10.6 (Snow Leopard). It’s surprisingly simple to set up X when the hardware is fake ;) Gentoo runs great on a Macbook Pro 13". I was concerned at first about the lack of 3D acceleration, but I tried a few 3D applications and found the performance to be acceptable. Blender ran really smoothly with a simple model (once I found out that command-click is equivalent to middle click). Rotating a field of several hundred spheres in Paraview was a little slow, but still very acceptable. I also bought a Logitech wireless mouse, and found that the middle button works just as you’d expect (paste) in Gentoo running in Fusion. I didn’t have to install the Logitech drivers in OS X.

Kernel configuration for a Gentoo guest in VMWare Fusion

Fri, 09 Oct 2009 22:48:23 +0000

I recently installed Gentoo Linux (amd64) as a guest on my Mac (OS 10.6 Snow Leopard) using VMWare Fusion. I thought I’d post the kernel config that I am using, since I didn’t find any out there that I trusted. If you can use this as a starting point, then it should save you some time and trouble. It’s a pretty minimal configuration–I think I removed all the extra drivers and stuff. You could lean it out a little more by removing the audio and a few other extras that I thought I might use. Let me know if you have any trouble with it. Gentoo amd64 kernel config for VMWare Fusion So far I am very pleased with its performance. My only disappointment is that VMWare doesn’t support Linux graphics hardware acceleration.

f2py: binding Fortran and Python

Wed, 23 Sep 2009 22:56:08 +0000

I have recently started using f2py to call Fortran from Python. I have found this useful for two reasons: speeding up Python scripts by calling compiled Fortran code, and using Python as a unit testing framework for Fortran modules. Unfortunately, the documentation for f2py is rather sparse, and may not be completely up to date. In this note, I will hopefully prevent you from wasting a lot of time figuring out how to pass array arguments, and return array results.

Profiling memory usage of Python code

Fri, 17 Apr 2009 22:08:52 +0000

In a previous post, I explained how to use the Python profiler. The profile is great for finding out which parts of the code run the slowest, or are called most often. However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed. If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude. On the positive side, your code may run much faster if it fits entirely in the processor cache. In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

How to make two mice work with xwindows (x.org)

Sat, 11 Apr 2009 04:01:45 +0000

It’s a real pain to surf the Web when the batteries die in your wireless mouse or trackball. I use some old rechargeables that are no longer fit for digital camera service, so this happens to me fairly often. My batteries just died again, so I had to figure out how to configure X.org to use a backup mouse. It’s actually not hard to have an old 3-button mouse plugged into the PS/2 port and tell X.org to use it as a backup. First, define two input devices:

Lookup tables and spline fitting in Python

Wed, 08 Apr 2009 23:36:09 +0000

Lookup tables and spline fitting are widely used by scientific programmers. A particular function may not have an analytic solution–in other words, it can’t be expressed as an equation of elementary functions. This might happen if the function were empirically determined from experimental data, or if the equation can’t be algebraically solved for one variable. If the analytical function is available, but takes a long time to evaluate, a lookup table or spline approximation can be considerably faster. In a previous post, I showed how to use the function interp1d from scipy.interpolate as a lookup table. In a later post, I showed that interp1d is actually rather slow, and scipy.interpolate.UnivariateSpline is much faster. Now, I will show some benchmark results, and explain a potential pitfall when using UnivariateSpline. The two functions I will use for this demonstration are not very complicated. Each function is a piecewise approximation of a more complex function. Here is one of the functions. For small z, the function uses one approximation, and for large z, it uses a far-field approximation. For intermediate values, a polynomial is used to smoothly match the two solutions.

3D Plotting Software for Python::Part 1::PyX

Fri, 20 Mar 2009 23:08:55 +0000

There are lots of good open-source tools that you can use to make high-resolution, publication-quality 2D plots. Personally, I like to use Python, numpy, and matplotlib. Unfortunately, it is much harder to find a good tools to make 3D plots. Older versions of matplotlib had rudimentary 3D support, but this was removed in version 0.98. In this post, I will review a Python 3D plotting library called PyX.

Preparing the data

Figuring out how to store the data to be plotted was actually the hardest part of learning to use PyX. The data format for 3D plots is not well documented. PyX requires a list of (x,y,z) lists like this:

Pypar Documentation

Sat, 21 Feb 2009 04:08:27 +0000

UPDATE 1/2016: It appears that Pypar is no longer maintained, although it probably still works. For new projects, I advise using mpi4py. The following is an online copy of the documentation of Pypar. I am not the author of the text below. I have merely posted it online to make it more accessible to the community. The latest version of Pypar can be found at: The Pypar page on Sourceforge

PyPAR - Parallel Python, no-frills MPI interface

Author: Ole Nielsen (2001, 2002, 2003) Version: See pypar.__version__ Date: See pypar.__date__

Parallel Programming with Python

Sat, 21 Feb 2009 03:49:01 +0000

Background

As an interpreted language, Python cannot compete directly with C or Fortran in execution speed. As I have written elsewhere, there are ways to make Python code run as fast as you need it to run. However, in scientific computing there is often no single processor that can execute a program in a reasonable amount of time. Computationally intensive problems require the use of parallel computers. A message-passing interface such as MPI is used to communicate between processes running on multiple CPUs. There is a Python interface to MPI called Pypar, which allows your Python programs to run in parallel on a system with an implemention of MPI installed. My experience is mainly with Open MPI, but there are several others. Below are some other resources to learn about about MPI and parallel programming in general. Using MPI with Python works around one of Python’s more serious limitations: the Global Interpreter Lock (GIL). Basically, Python uses the GIL to prevent errors that may occur when two threads try to access the same data. What this means is that Python can only utilize a single CPU thread. Note that this is not a fundamental limitation of the language, but of the standard C implementation of the interpreter. New interpreter implementations may be thread-safe, but for now, Python is stuck in a single thread. The GIL will become more of a problem as ordinary desktop CPU’s gain speed by adding cores rather than increasing clock speed. Rather than exposing my ignorance of Python’s internals, here are some links to posts written by people who are more knowledgable about this subject than I am.

Optimizing fast Python math with Numpy and Scipy

Tue, 10 Feb 2009 21:58:31 +0000

Coming soon! The work is already done–I just have to find time to write it up.

Online Documentation and Resources

scipy.weave documentation Performance Python documentation

Blitz++ (array and matrix library for C++) docs

The Python configparser: a way to read simple data files

Tue, 03 Feb 2009 23:42:21 +0000

My simulation library, which is written mostly in Python, needs a lot of data and parameters in order to run. In some cases, I just hard-code the values in the script that calls the library, and in other cases I load a pickle file containing a Python objext. What if I want to read in data or configuration parameters from a human-editable text file? If the information is extensive or complex, XML might be a good choice, but XML is overkill for simple configuration or data files. Fortunately, a standard Python library called ConfigParser has already defined a configuration file format, and provides methods to interact with such files. Here is a sample of the format used in a config file:

Updated Python class for writing Paraview (VTK) (.vtu) files

Fri, 23 Jan 2009 23:28:05 +0000

I have released a new version of my Python class that generates VTK data files in the .vtu format, which is compatible with Paraview and other VTK applications. If you have downloaded the old one, please get the latest version, which incorporates some bug fixes and has been more thoroughly tested.

How to put formatted, highlighted code in a Wordpress post

Sat, 17 Jan 2009 21:49:36 +0000

I found two complementary plugins that enable me to put highlighted formatted code in a Wordpress page or post. Here’s an example of what they do:

[sourcecode language="python"]
# Plot flux at continuum boundary
pylab.figure()
pylab.hold(True)
pylab.plot(nd_times, Jl_BD, 'b-', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD+Jl_BD_std, 'b.', label="Flux from BD simulation")
pylab.plot(nd_times, Jl_BD-Jl_BD_std, 'b.', label="Flux from BD simulation")
[/sourcecode]

Test successful! Note that Javascript must be enabled to see the highlighting. Get the plugins here:

Deploying Python applications on Windows

Thu, 15 Jan 2009 16:38:56 +0000

Writing applications in Python on a Linux system is almost too easy. Deploying Python apps on other Linux systems is not hard, because most Linux systems already have Python, with its core libraries and tools, installed. Most Linux systems also have package managers that make it easy to find and install required components. But, what happens when your co-workers who use Windows need to use your app? When you tell them to “go to the command line and…” you’ve pretty much lost them at “command line.” How do you package Python in a way that’s easy for a Windows user to install? Here is a process that worked for me:

Scipy.integrate ODEPACK import error solved!

Wed, 14 Jan 2009 16:46:16 +0000

I recently found a solution to a problem that had been vexing me for about a year. In order to successfully import anything from scipy.integrate, I had edit the file scipy/integrate/__init__.py and comment out the line

from odepack import *

If not, I would get various import errors such as

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python2.5/site-packages/scipy/integrate/__init__.py", line 10,
in <module>
 from odepack import *
 File "/usr/lib/python2.5/site-packages/scipy/integrate/odepack.py", line 7,
in <module>
 import _odepack
ImportError: /usr/lib/python2.5/site-packages/scipy/integrate/_odepack.so:
undefined symbol: daxpy_

I found the solution in Gentoo bug 251165. The problem only occurs when Scipy is built with non-reference versions of BLAS and CBLAS. The solution is to install the reference implementation of BLAS and CBLAS, rebuild Scipy, and then use whichever implementation of BLAS or CBLAS you want. I don’t know why this happens, and I don’t know if it affects distributions other than Gentoo. By the way, Gentoo has a really useful system tool called eselect, which has various modules that are used to choose between different versions or implementations of tools on your system (BLAS, Java virtual machine, kernel sources, OpenGL, etc.)

Optimizing Python code for fast math

Fri, 09 Jan 2009 21:14:38 +0000

I spent some time today profiling a Brownian dynamics simulation written in Python to see how I could make it faster before starting some long runs on a Linux cluster. In the sections below, I have attempted to quantify the speed improvements due to various changes. Keep in mind that the actual speed improvement in your code will vary, depending on where the actual bottlenecks occur. See my post about profiling Python code. Another caveat: I am running Python 2.4.4 because it’s installed on our cluster.

Redirecting standard output from Python: another example

Wed, 07 Jan 2009 21:54:10 +0000

I wrote a previous post about how to redirect standard output from a Python script to a GUI window. In this post, I will give an even simpler example of to redirect standard output to a log file. During the early development and debugging of Python programs, I use print statements to keep me informed of what’s happening. However, printing to the terminal is not always practical–for example, when I run numerical code on a parallel cluster, there is no way to determine which output came from which instance of the program. Here is a class that you can use to redirect standard output to a log file:

Tools for Python software development

Tue, 06 Jan 2009 18:52:45 +0000

I have found a few tools over the years that I find extremely useful for developing software. Python is my language of choice at the moment, but I’m sure these tools will be handy for any language.

Subversion is an open-source version-control system. Version control was designed to allow multiple programmers to work on the same project at the same time without stepping all over one another. However, even though I am a solo developer, I find version control to be extremely helpful.
1. When I commit changes to the repository, I can document what I’ve changed and why. This is a great help when I introduce a bug and have to go back and find it.
2. The repository is stored on a remote server that is backed up nightly.
3. It’s easy to make an “unstable” branch for implementing new features. When I make changes that don’t work, it’s easy to revert to a previous version that works.
4. It is easy to deploy my code to the Linux cluster and make sure that the cluster is running the latest version of my software.

Go read the documentation on the Subversion web site to find out what it can do for you.RapidSVN is a GUI client for a Subversion server. By default, Subversion comes with a command-line client that does everything you need. However, sometimes it’s easier to stay organized when everything is presented visually. Here is a screenshot of RapidSVN:

RapidSVN screenshot

A lookup table for fast Python math

Thu, 11 Dec 2008 23:17:06 +0000

Numerical programming frequently requires the use of look-up tables. A look-up table is a collection of pre-computed values. When given an “x” value, the table returns a pre-computed “y” value. Look-up tables can be used to speed up numerical codes, when it is faster to look up a value in the table than it is to compute the value. They are also used when the data in the table cannot be computed–for example, experimental data or averaged results from an ensemble of Monte Carlo simulations. Another application is to compute a value when a function cannot be solved algebraically. Assume that you have a formula for a function q(h). You need the value of h for a given value of q, but the formula cannot be algebraically solved to get h(q). Instead, choose a range of h values, compute the function q(h), and store each value in a look-up table. Now you can get h(q) for any value stored in the table. The major limitation of a look-up table is that it cannot return valid results for any value of q which is outside the range of those stored in the table. Depending on its implementation, the table may be able to interpolate to return values between known points.

Update 2: building 64-bit Numpy with Intel compilers and MKL

Tue, 09 Dec 2008 19:33:12 +0000

NOTE: these instructions are obsolete. Please see Building NumPy on a 64-bit Red Hat Cluster with Intel MKL. In a previous post I described how I built Numpy with Intel compilers and the Math Kernel Library on a 64-bit cluster. Today I upgraded to Numpy-1.2.1 and I made a few improvements to my install process. Please read the previous post, since I will not duplicate some important information, and then read on. This time, I made use of a site.cfg file. Copy the file site.cfg.exampletosite.cfg and edit. At the end of the file, uncomment the [mkl] section and set the path to your library. Mine looks like:

Calculating the Pair Correlation Function in Python

Wed, 26 Nov 2008 23:45:55 +0000

The pair correlation function, also known as the radial distribution function, is a way to characterize the distribution of particles on a two-dimensional plane or in a three-dimensional space. Please check out Eric Weeks’ web site for an introduction to pair correlation functions. He has written some routines in IDL to compute these functions. Using his foundation, I have written some simple routines in Python to compute 2D and 3D pair correlation functions.

Visualizing trajectories with Python, VMD, and .vtf files

Wed, 26 Nov 2008 19:48:01 +0000

As a computational scientist who develops Brownian Dynamics (BD) simulations, I have had a difficult time finding software to visualize trajectories from my simulations. There are a lot of software tools for molecular visualization, but the majority of these tools are designed to view “static” molecules that do not change over time. For storing the output of molecular dynamics (MD) simulations, this is logical, since the simulations often run for millions of time steps and it would be impractical to store (and view) all of them. The user is often interested only in finding the lowest-energy configuration, so all the earlier steps can be discarded. While this is also somewhat true for BD simulations, for debugging purposes it is helpful to visualize the trajectory of the particles.

Generating VTK files with Python

Thu, 20 Nov 2008 23:39:54 +0000

Paraview is an industrial-strength, open-source 3D visualization program designed to visualize large data sets created by computational fluid dynamics (CFD) simulations. Although its native data file format (.vtu) is designed for reading in point- and cell-based data from CFD simulations, I have successfully used it to read in and visualized discrete particle data from a Brownian Dynamics (BD) simulation. Below, you can download a Python class that is useful for visualizing particles in motion. It should be easily adaptable to any type of discrete data set that involves discrete objects, vectors, and/or scalars. In addition, this class should be useful with any application based on the VTK toolkit because they also read .vtu files.

Numerical software and examples

Mon, 17 Nov 2008 19:49:59 +0000

Here is a brief index to some of the more useful things I have created or modified.

Compiling LAPACK with the Intel Fortran Compiler ODEPACK: a library for numerically solving a system of ordinary differential equations

Boost.Python Examples and Tutorials

Scientific Computing

Mon, 17 Nov 2008 04:03:51 +0000

My PhD research is to develop tools that predict the interaction of proteins with engineered surfaces. This work has applications in the development of biomedical devices (such as neural implants and point-of-care diagnostics), biosensors, and hybrid neural systems.

My pages about scientific computing

Optimizing fast Python math with Numpy and Scipy (coming soon!) Calculating the pair correlation function with Python Generating VTK files with Python for visualization in Paraview Visualizing trajectories with Python, VMD, and .vtf files Compiling LAPACK with the Intel Fortran Compiler ODEPACK: a library for numerically solving a system of ordinary differential equations

Simulation

Mon, 17 Nov 2008 04:01:17 +0000

Most of my work has been focused on developing simulations for use in research and development. However, I have worked on several projects with applications in military simulation and civilian training.

A WWII Simulation Using OneSAF Objective Framework Custom scenery development for a pilot training scenario

Open-source tool chain

Mon, 17 Nov 2008 02:58:22 +0000

I use open-source software whenever possible. Although there are cases where a proprietary tool is the right choice, open-source tools have many advantages.

Complete control of your critical tools–the vendor will never stop supporting your platform, an upgrade will never be forced upon you, and the licensing fee will never increase. Open-source tools tend to utilize open standards and exchange data much more fluidly than proprietary tools.

If necessary, your developers can fix a bug themselves. A bug might be critical to your application, but that doesn’t mean it’s a high priority for the application vendor.

Open-Source Tools

Programming Python for rapid application development Numpy and Scipy for scientific omputing C, C++ and Fortran for optimized code Parallel Computing MPI PETSC pypar and mpi4py (Python bindings for MPI) CUDA and PyCUDA for massively parallel execution on NVIDIA graphics processors (GPUs) Simulation OpenFOAM: Computational Fluid Dynamics libmesh: finite element library Visualization Matplotlib (2D plotting) VTK (3D visualization) Paraview (3D visualization) VMD (Molecular dynamics visualization) Documentation LyX LaTeX Asymptote (technical vector graphics) Office and OpenOffice Version control Subversion Git OpenFOAM

Using Python to generate XML files for visualization in Paraview

Thu, 13 Nov 2008 19:00:23 +0000

VTK is an open-source software system for “3D computer graphics, image processing, and visualization” developed by by Kitware. VTK is the foundation of Paraview, an industrial-strength CFD visualization tool that I have found to be very useful. I generate “second generation” XML-based files from my Python code and import them into Paraview for visualization. I am in the process of creating some Python classes to do, and I hope to publish them soon. Until then, I want to share some useful resources. The VTK file formats are specified in this document. It’s a pretty good specification, but it lacks some examples. Soon I will post an example of a valid unstructured, serial .vtu file. Each VTK file includes data from only one time step, so you have to keep track of time yourself (the filename is an easy solution). Paraview can read in data from multiple time steps, but you have to specify them in a .pvd file. This is also an XML file, with the following format: (reference)

Unexpected integer/float math behavior in Python

Fri, 07 Nov 2008 00:21:16 +0000

I wasted some time today tracking down a bug in one of my programs. It turned out to be “unexpected behavior” rather than a bug. I was aware of this aspect of the language, but I made an assumption and got bit. Read on for a valuable lesson. Python handles integer math differently than floating point math. If you type a number without a decimal point, Python treats it as an integer. All math performed only with integers results in integers.For example, 1/2 evaluates to 0 while 1./2. evaluates to 0.5. If you mix integers and floats, Python will produce a floating point result (1/2.=0.5), but you must be very careful. For example, you might expect the expression 4/3*3.14159 to yield a floating point result. It does yield a floating point number, butnotthe one you were expecting! 4/3*3.14159 yields 3.14159. What happened? Python works from left to right. 4/3 evaluates to the integer “1”. 1*3.14159 evaluates to 3.14159. For comparison, 4./3.*3.14159 evaluates to 4.1887866. Here’s the problem with this particular aspect of Python: according to the rules of math, 4/3*3.14159 is exactly the same expression as 4*3.14159/3, but in Python they yield different results if you forget the decimal points! 4*3.14159 evaluates to a floating point, so (4*3.14159)/3 yields the “correct” floating point value. Lesson Learned: be explicit about specifyingall floats if you are doing floating-point math! Sometimes I get lazy and leave a trailing decimal point off of a number when doing a floating point calculation, knowing that the results are “upcast” into floats. Not any more! Note: this unexpected behavior goes away in Python 3.0

What is a shock solution?

Fri, 31 Oct 2008 22:22:11 +0000

Definition of a shock solution

Definition: “shock solution” is an informal mathematical term used to describe a type of weak solution in the field of partial differential equations. When a nonlinear or quasilinear equation does not have a strict solution, a weak solution may be found that allows useful information to be obtained even though a strict solution cannot be obtained. Reference: J. Kevorkian, Partial Differential Equations, Springer-Verlag, 2000.

Why did I choose the domain shocksolution.com?

The shock solution is a clever method of obtaining useful information from a problem that could be considered unsolvable. It is a reminder that even the most complex and difficult problem can be analyzed and understood using the correct tools. My objective is to develop these tools so that better decisions can be made when dealing with the complex problems found in the world.

Profiling Python code

Fri, 24 Oct 2008 21:37:24 +0000

“Speed” is a complicated term when used in the context of software. Does it mean raw speed of execution, or reducing the amount of time until a correct result is obtained? Python is not the first language that comes to mind when people think of “fast software.” It is true that pure Python will usually not execute as quickly as the same algorithm directly coded in C or Fortran. However, when you define speed as “least amount of time until you get the right answer,” then Python is pretty fast. It is so easy to develop correct code in Python, when compared to low-level compiled languages, that Python is often the fastet route to a correct answer, even if the execution time is longer. Having said that, there aretimes when code has to execute quickly, and that’s why I will introduce you to profiling Python code. Like all things Python, profiling is easier than you think. Python 2.4 has the profile module, and Python 2.5 has both profile and cProfile. cProfile is written in C for lower overhead, and it’s the recommended version. I am stuck with profile, because the cluster that I am working with still uses Python 2.4. You can read the docs for more details, but I will quickly outline what I find to be the most helpful usage. For example, say I want to profile the filerun_sim.py. I use the following command line:

Updated: building 64-bit Numpy with Intel compilers (icc)

Fri, 17 Oct 2008 19:36:38 +0000

I had to re-build Numpy because our cluster was upgraded and the Intel compilers and libraries were moved to a different directory. This turned out to be a half-day affair of trial-and-error. I learned a few important things, which I will try to list here:

*Delete the numpy-1.0.4/build directory after every build attempt. Doing “python setup.py clean” isnoteffective. I kept getting errors about undefined symbols when I tried to “import numpy” on the Python command line. It was looking for symbols in the old locations, even though I had just rebuilt the code using the new library locations. It turned out that I needed to delete the build directory in order to force a complete bottom-up rebuild.*The use of “setup.py” from distutils is not well documented online. The best thing to do is run “python setup.py –help-commands” to get a list of available commands. Then run “python setup.py <cmd> –help” to get help for that specific command. You can string commands together on the command line, as I will show in the example below.When you test the new numpy, make sure you arenotin the numpy-1.0.4 directory! If you are in the numpy source directory, when you import numpy, you will get the message “Running from numpy source directory.” and you will not be able to load any symbols from numpy. On 64-bit architectures, you need to compile position-independent library code. For some reason, distutils does not do this automatically, and the compilation will fail with an error similar to the following:

Even faster collision detection in Python using Numpy

Thu, 16 Oct 2008 19:44:15 +0000

Last night, in the shower, I realized that my collision detection routine could be even faster. Here is a representative snippet of code from my previous post:

d2 = (x-self.x[0:i])*(x-self.x[0:i]) + (y-self.y[0:i])*(y-self.y[0:i]) + (z-self.z[0:i])*(z-self.z[0:i])

For some reason, I used the code (x-self.x)*(x-self.x) instead of (x-self.x)**2. Upon further reflection, I realized that (x-self.x)*(x-self.x) computes the difference between array elements twice, and then multiplies the results. Using a “power function” should enable the interpreter to compute the difference only once, and then multiply each element times itself. Here is the updated code, using Python’s power operator:

Haze damages LCD projectors

Tue, 14 Oct 2008 17:24:28 +0000

We recently learned the hard way that the use of haze can lead to reliability problems for some LCD projectors. We have two rather large LCD projectors (don’t know the specs offhand) permanently mounted to the ceiling, projecting onto the front of screens located on both sides of the stage. A third LCD projector hits a large rear-projection screen that basically forms the back wall of the stage. We also have an oil-based hazer mounted above the stage. We had some old, tired projectors, so we assumed that they were just old and unreliable. However, after replacing them with new, more powerful models, the reliability issues continued. After sending the new projectors out for service for the second time in less than a year, the service center informed us that a film of oil had been deposited on the LCD. You can buy expensive sealed-optics projectors, which should work reliably in dirty environments, but we saved money and bought standard models. We don’t have the budget to replace the “new” projectors, so we’ve had to stop using haze altogether. The stage doesn’t look nearly as good without it, and our moving lights are much less useful. On a side note: you should be aware that haze can also damage moving light fixtures that use a fan to cool the power supply or motors. The fan sucks in haze and coats the internal components with a film of oil. Eventually, something overheats and the fixture can actually catch fire. Fortunately, we have older High End Studiospots and Trackspots that apparently don’t use forced-air cooling, so apparently the haze doesn’t really get inside. Be warned!

Speeding up Python math with Numpy: collision detection example

Sun, 12 Oct 2008 18:10:20 +0000

Python is a very-high-level language. That makes it easy to write code quickly, but the program may not be as fast as a program compiled from a lower-level language. For this reason, many scientific programs are written in Fortran or C++. However, it has always been my experience that the majority of time on a project is spent in writing, modifiying, and debugging code, rather than executing. Fortunately, if written correctly, the time-critical parts of Python code can execute almost as fast as compiled software. Here is an example of a collision-detection algorithm which achieved almost a ten-fold increase in speed when written to use Numpy.

The price of cheap Chauvet LED fixtures

Thu, 09 Oct 2008 02:34:49 +0000

In my previous post about cheap LED fixtures, I didn’t mention our Chauvet Q-Wash intelligent moving light LED fixtures. We bought four at a very discounted price to supplement our conventional movers. They weren’t bright enough to stand out in the stage wash, but they made nice little beams in the haze, and they could be used to “fill in” small dark spots on the stage or set pieces. As wash lights, they lack gobos or beam control, but they do have smooth color RGB mixing (most “real” wash lights do have some kind of beam width control). So far, you might be thinking that you should pick up a couple, so I should tell you now that they all stopped working within a year or so of installation. One died within a month or two, and after about six months of operation, we lost one every couple of months until they were all gone. The problem wasn’t the LEDs–it was the motors or control circuitry. We were using haze, which is known to shorten the lifespan of moving lights, but that’s the standard operating environment for concert lights! The fixtures were also permanently mounted above our stage, so they weren’t damaged during handling. They just quietly died on their own–well, except for the one that went insane and just started moving around on its own until we got up in the lift and unplugged it! Fortunately this didn’t happen during a performance, as the motors got very hot and might have started smoking or something. The moral of the story is that a low initial price doesn’t always save you money in the long run.

Installing numpy with the Intel Math Kernel Library (mkl)

Fri, 19 Sep 2008 21:24:58 +0000

Today I installed numpy on a cluster. Normally, as a Gentoo admin, I just install things with emerge, and all the details are taken care of automagically. However, this cluster runs Red Hat Enterprise, and I don’t have admin privileges, so I had to install numpy in my home directory. I installed 1.0.4, to match the version used on another system. You may not need to do this for more recent versions of numpy, which may have an improved setup script. The overall process is:

Python Pickle: Painless binary storage for Python objects

Mon, 15 Sep 2008 22:23:18 +0000

The pickle module provided with Python is so useful that I’m surprised I haven’t used it before. Pickle allows you to save an entire data structure (such as an object) to disk as a binary file in a effortless (and fairly efficient) manner. For example, in my latest project I have created a Monte Carlo simulation that can take quite a bit of time to run. I also need to make multiple runs to get statistics on the results. At the end of each run, I need to dump the resulting data to disk so that it can be read in later by an analysis program. If I had to write data in a format that could be interchanged with other scientific software, I’d use the hdf5 format with the pytables package. However, right now I just need to get something working, and the pickle module is perfect. Here is how I save an object called box:

My first Spore creation

Tue, 09 Sep 2008 02:10:22 +0000

http://www.spore.com/flash/dsDemo/sporeds_widget.swf

Python threads are easy (with example)

Tue, 19 Aug 2008 22:50:07 +0000

It’s remarkably easy to spawn a Python thread. However, before doing so, I caution you that a Python thread is not the same thing as an OS thread. Python threads run within the Python interpreter, but the Python interpreter always executes in a single process. The reasons why have already been explained elsewhere, so I refer you to the thread module documentation to learn about the Global Interpreter Lock. You probably have objections to this state of affairs, and I assure you they have already been voiced by Juergen Brendel and responded to by Guido van Rossum (creator of Python). Anyway, the upshot is that Python can only utilize one core of a multi-core CPU. This isn’t such a big deal for me because I’m a scientific programmer, and if I really need to write parallel code it’s going to have to run on a cluster or a grid. Threads don’t help with that. Having said all that, threads in Python are still useful. I will detail one example in which I spawn a thread to load a large binary file. While this doesn’t spread the work across multiple CPU cores, it does enable the GUI to remain interactive while the file loads. All you have to do to create a Python thread is create a class that is derived from Thread. In the example below, I derived a class called Loader, which “wraps” the function that actually reads the binary files. The __init__ method accepts the filename and other options as arguments. The run() method is required. Don’t call run() directly–instead, call the start() method (inherited from the base class) to start the thread.

Redirecting text output from Python functions

Sat, 16 Aug 2008 20:47:05 +0000

Two posts ago, I described how I wrote a function in Python that reads in a binary file from Labview. In my last post, I described using wxPython to write a GUI to process the data from those binary files. Naturally, I called the binary-file-reader function from the GUI. The problem is that the file reader prints a lot of information to the terminal, using Python printstatements. None of this goes to the GUI, requiring the user to run the GUI from a terminal and keep an eye on the text output, which is inconvenient. However, I don’t want to modify the file reader to include GUI-specific code, because that would be less modular and less re-usable. Instead, I learned that Python has a very easy facility to redirectstdout, the default destination of the print statement. I modified the file reader as follows:

Fun with threads in Python and wxPython

Mon, 11 Aug 2008 22:12:07 +0000

I have finally gotten back to programming in the last couple of days. Our project has finally started to generate a lot of data, so I’ve been refactoring and improving my code that reads data stored in LabView binaries. Today I spent a lot of time creating a GUI for browsing data. Arguably, this wasn’t the best use of my time, but I learned a lot about multi-threaded Python GUI programming with wxPython. You can find a gold mine of information on the multi-threaded wx programming at the wxPython wiki. Because the LabView binary data has to be read sequentially, and the files are rather large, it takes a long time to read in a file. I spawn a thread to handle the file reading, while allowing the GUI to remain responsive. The thread posts messages to the GUI window, which are used to update the user on the status of the file reading operation. When the file is read, a final message containing the data is posted to the window. It’s really pretty slick now that I’ve figured out how to do it. I will soon post a clever scheme to capture text output from the file-reading function, and display it in the GUI, without making substantial changes to the file-reading function.

Reading Labview binary files with Python

Wed, 25 Jun 2008 16:04:46 +0000

My research group uses Labview 7.1 to write custom data acquisition (DAQ) software. I code everything else in Python, so I need to get data from Labview into Python for processing. Our DAQ program produces Labview binary files, so I had to find a way to read them with Python. Binary files are nice because they are a compact way to store numerical data as compared to ASCI or (heaven forbid) XML, but they are much harder to read. The binary format used by Labview is documented only indirectly, so I had to hack a little. The first thing to realize is that the Labview binary file is a direct dump of the data that was stored in RAM. How Labview stores data in memory is documented here. Indirectly, this documents how binary files are stored on disk. Our DAQ program writes a rather complex “cluster” (Labview’s version of a C structure) to disk. The elements of the cluster are stored contiguously as a sequence of bytes, and there’s no way to know which byte goes with which element, unless you know the size of each element and the order in which they are stored in the cluster. So, the first step is to document the cluster that’s being written to disk. You can use the context help in Labview to view the data type of the wire that leads to the VI that writes the file. With this in hand, you are ready to write Python code. First, make sure you open the file in binary mode:

Server move completed and general update

Sun, 22 Jun 2008 19:37:44 +0000

The site is back online after a server move. Actually I can’t blame the server move for the downtime, because I went out of town and then had a lot of catching up to do, and as a result I didn’t switch the domain to point to the new server. I haven’t been blogging much because I haven’t done much software development, Linux admin, or lighting design lately. I have been busy in the lab at work and I’ve been supervising a trainee lighting operator instead of running my own shows. We haven’t been moving forward with plans to purchase a new lighting control console, so no update on that, either. Hopefully, I’ll have something interesting to post soon.

Presentation at BarCamp Orlando 2008

Mon, 07 Apr 2008 03:14:32 +0000

I gave a presentation at BarCamp Orlando on 5 April 2008. Here is a link to my presentation, Data Analysis with Python.

Connecting to a Windows file server from a Linux box

Wed, 23 Jan 2008 18:26:00 +0000

Updated 18 Jan 2013

Gentoo

There are two basic prerequisites for mounting a Windows share on a Gentoo Linux client. First, your kernel must be compiled with cifs support. Second, you must emerge the appropriate packages. The package used to be called mount-cifs, but this package is obsolete. You only need to install net-fs/samba, which will pull in net-fs/cifs-utils and a few other dependencies. The Samba page on the Gentoo Wiki is pretty good. Since you probably need a user name and password to log in to the Windows share, you should create a credentials file. You don’t want to specify your password on the command line because it will be saved in your .bash_history file and unauthorized users may be able to see it. I created a file called /etc/credentials.txt and set permissions so that only root can see it. The format is:

Found a bunch of templates

Tue, 22 Jan 2008 21:02:05 +0000

I use XFCE as my window manager, with Thunar as my file manager. Thunar has always had this “Create Document -> from template” entry in the File menu, but it didn’t come with any templates. I finally went looking for some templates, and found a good collection at stalefries. Just unpack the archive in a directory called “Templates” in your home directory. BTW, isn’t it about time I created a “linux” category instead of trying to stuff my Linux entries under the “software development” category?

Cheap LED fixtures

Fri, 11 Jan 2008 05:23:47 +0000

Normally, we stay far away from anything marketed to DJ’s, but we’ve been experimenting with cheap LED fixtures as an inexpensive way to put more colors on stage. LED fixtures have a few advantages: they efficiently produce light without much heat, they don’t require a dimmer, and they can produce multiple colors without gel changes. They also have some disadvantages. Color mixing is produced by combining LED’s of different colors (RGB) at different intensities, so the color is only “mixed” at the surface it is illuminating. It can be difficult to produce white light that matches light from other fixtures. LED’s cannot be dimmed by lowering the voltage, like a conventional light. Instead the LED is strobed (pulsed) at a rate too fast for your eye to discern. Unfortunately, the Chauvet ColorPalette fixtures shown below use a pulse rate that is slow enough that it can be picked up on a video camera. When the fixtures illuminate a set piece like this one, which is in the background of many IMAG shots, the flicker is very distracting if the intensity is run at anything less than 100% Practically speaking, when a Color Palette is visible on camera,we can only get seven full-intensity colors: red, blue, green, white, purple, blue-green, and yellow.   We’ve since moved them out into the auditorium where they are used to illuminate the walls. We did find one really cool application for these lights. We have a number of deep-purple, “blacklight” Color Palettes mounted backstage to enable people to move in the dark. They stay on all the time, draw very little power, and are totally invisible to the audience, even during a dead blackout. This is a Chauvet COLORado 1, which is meant to replace a small par can. It’s mounted on a pipe above and behind the musicians and lights them from behind. This is the view you get from the front row in the audience. See the red, blue, and green LEDs? That’s producing white light, but you wouldn’t know it from this angle. That’s what happens when the colors don’t mix at the light. We don’t have the strobing problem on video with the COLORado’s, but I don’t know if that’s due to their positioning or their design. For more information, see the LED lighting reviews by onstagelighting.co.uk. There are also pro-grade LED fixtures that are much better…but I haven’t used them.

Cool set design: Part II

Fri, 04 Jan 2008 02:07:09 +0000

Here’s another cool set design. What’s interesting about these two pictures is that the set is illuminated using the same lights at two different intensity levels, but the color seems different. That’s a neat feature of incandescent lights–at low intensity you can get deep, saturated colors. Here is a look in another color. I didn’t think to take close-up pictures of this set. It turns out that our set designer likes to have these pictures, so now I’m trying to do a more thorough job.   This one is constructed of a wooden frame with a light colored wrinkly fabric stapled to it. Like I said, I don’t design or build these things, but if you want to know the details, I’m sure I can find out for you.

Cool set design: part 1

Wed, 02 Jan 2008 03:14:57 +0000

After a longer-than-expected holiday break, I’m back with a slightly different topic: set design and construction. Someone else designs and builds our sets, but I get to light them, and I’m usually tasked with documenting them. Our lighting console evaluations have been on hold for the holidays, but I plan to get back to it later this month. The set I’m writing about today is one of the more exotic ones we’ve tried. Here’s how it looked when lit Pieces were cut from white Styrofoam and stitched together to make a wall of crazy curves: The set was lit from below with cheap LED panels. The black boxes in front of the fixtures are cinder blocks that are basically sitting there to keep people from stepping on or kicking the fixtures as they enter and exit the stage. These LED fixtures are handy, but they have their drawbacks. That will be the subject of another post.

ChamSys MagicQ console review::part I

Fri, 07 Dec 2007 03:07:47 +0000

I don’t know how to pronounce it, but I like it! This review is based on my impressions gathered from a few minutes of hands-on experience at LDI 2007, and the MagicQ PC software that I downloaded from ChamSys. I plan to review a lot of consoles by downloading the offline editors, and hopefully we will get demo units of our favorites. Installation: Download from the link above. I’m thrilled that it runs natively on Linux. I made a directory /opt/magicq/ and uncompressed the tarball. I had to install libftdi as indicated in the instructions (version 0.7 works fine). In order to run the software as a non-root user, I had to give write permission on the usb device, even though I’m not using any wings. That was the only hitch. Programming: The MagicQ has a programmer and a playback section–the playback section can be playing cues from lists, while the user is manually setting values in the programmer. This is similar to the Hog, but different from the ETC Congo. Like the Hog, there are multiple playback faders, each with a cue list, and the console mixes the outputs using HTP or LTP (highest- or latest-takes-precedence). The MagicQ assigns HTP or LTP per DMX channel. Conventional lights are usually run HTP, while movers are run LTP. This overcomes one of the worst features of the Hog–you can only choose HTP or LTP for an entire cue list, meaning that it’s hard to mix conventional lights and moving lights in the same cue list. Neat feature: the manual is available on the console by pressing the Help button. Next time, I will try patching some fixtures and running the demo show.

LDI Report::Part 3::Midrange Lighting Control Consoles

Thu, 29 Nov 2007 01:40:37 +0000

My real mission at LDI was to review lighting consoles–management is considering replacing our Hog 1000 in the next year or so. The Hog is serving our needs well, but we’ve already used up both DMX outputs, and Hog programming is not the easiest thing to teach to new lighting operators. So, we’re looking at consoles that match or exceed the capabilities of the Hog 1000, with 4 DMX outputs and hopefully with a more user-friendly design. We’re keeping the price below $30,000–hopefully well below! So, without further ado, here are the consoles I got to see in person. Please keep in mind that these are first impressions, and I really need to download the offline emulators before I say anything more. The Avolites Pearl Expert and Pearl 2008. The main thing that caught my eye with this console is the fader paging system. There’s basically a big drum embedded in the board above the playback faders. You use console tape on the drum to label the faders. To get to a new bank of faders, roll the drum one click and bring up a new set of labels. It’s intuitively obvious when you play with it. The MagicQ is a module console from a new company called ChamSys. It’s being distributed in the US by PRG. Some of the programmers previously worked on the Hog 2 and 3, so this board runs much like a Hog 2. It might be described as “what the Hog 3 should have been.” Very programmable–and Linux-based! These are Vector consoles from Compulite. They seem to be reasonably priced and extremely programmable, but retain some of the simplicity of a theater-style console. I got to talk to a software developer on the project, and although the consoles run a Windows XP operating system, they have taken great pains to maintain realtime hardware response. The Congo and Congo Jr. were originally made by Avab, and somehow acquired by ETC. Compared to other brands, I was most impressed with the user interface of the Congo. It’s designed like an ETC theater console, but integrates moving light support seamlessly. There is no separate programmer–you are always working within a cue. It’s probably too early to say this, but I think this would be the easiest one to teach to a beginner. Unfortunately, it seems to be one of the more expensive consoles in this lineup. Here is the LeapFrog from zero88. I didn’t get much chance to use it, but with only 2 built-in DMX outputs, I’m not sure this is going to have the horsepower we need. The Frog 2 seems to be very much like the Hog 1000 or Hog 2 in terms of its programming style. I don’t think it adds much value when compared to the Hog 1000. Overall, I was impressed with the consoles I saw at LDI. My impression is that there are many powerful consoles to choose from, but many of them are too programmable for their own good. I love programming features, but that’s because I’m a professional programmer who happened to learn lighting. What about the theater lighting person who needs to run moving lights? Is it really a good thing that bump button #7 can be programmed to shut off the console? On another note, a lot of people were very helpful when showing me their consoles. However, they would have been a lot more effective if they had just shut up and listened to my needs for a minute before going off and describing all kinds of features I don’t care about. Stay tuned for more in-depth reviews!

LDI Report::Part 2::Cool Products

Fri, 23 Nov 2007 04:26:17 +0000

These are a few interesting products that I saw at LDI. I spent most of my time at the show looking at and trying out various lighting control consoles. I didn’t have time to prowl around the smaller booths where the most interesting products are sometimes hidden. These are just the things that caught my eye on the way from one place to the next.  Over the next month, I will go deep into detail on the lighting consoles I saw, so I’m excluding that product segment from this post. Here is a giant video ball from Skyview. It’s inflatable, it will float if filled with helium, and it has a video screen. This is an innovative product for the film industry from Airstar. It’s an inflatable flood light. The large, inflatable reflector housing seem easy to set up, and compact to store when deflated. This is Airstar’s booth. I was really impressed with these illuminated spheres, which can also be filled with helium to make them float in midair. Check out their gallery page for cool applications. I want to rent some of these! Last but not least, the Fogscreen! It uses plain water and ultrasonics to produce a very fine fog of water–the drops are so fine that it doesn’t even feel like water, and you don’t get wet at all from standing in it. The fog is forced into a sheet that can be used as a screen for projection.

LDI Report::Part 1::Eye Candy

Mon, 19 Nov 2007 04:27:19 +0000

LDI is where lighting designers go to dream. Companies spend big bucks to bring their products and show them off in the most spectacular fashion. Ironically, some of the less-reputable names have some of the largest booths. High End was completely absent–a source told me that they don’t spend money on trade shows because they introduce new products directly to key designers (not because they are having financial difficulties). In this post I’m going to highlight a few photos of the show in general. For more, check out my Flickr photostream. In future posts, I plan to focus on reviewing some lighting consoles. First, some spectacular booths from Chauvet and Robe--it appears they are trying hard to be taken seriously by lighting professionals. At the Chauvet booth, note the array of 96 mini-movers! I was also impressed with the booth from XST: I’m not a rigger, but it sure looks like they know what they’re doing with truss. Cyberhoist also had a really sweet booth. The trusses moved up and down at changing angles in various coordinated patterns.

Lighting design review: Mutemath in Orlando, FL

Fri, 16 Nov 2007 18:46:38 +0000

I don’t go to many concerts. I don’t want to spend the money to hear a band play a poor version of their CD. But I heard from several people that Mutemath was a great live act, and it turns out that the rumors were true. I saw them on November 2 at House of Blues in Orlando, FL. Not only were the music and showmanship excellent, the lighting design was first-class. Too many rock bands have a cloud of haze, a bunch of colored par cans, and some moving beams.   Mutemath is not your average rock band, and I have the poor-quality cell-phone-camera pictures to prove it. First, we have an overall shot of the stage. Nothing too special about the lighting in this view. Note that there was almost no front lighting. Here is a creative look. Notice the beams coming from below the video screens, illuminating everything within three feet of the floor. The LD told me they are MAC 250’s that tour with the band. These lights were only used for floor-level illumination. Most of the lights on the truss overhead were wash lights and par cans belonging to HOB. In this picture you can just barely make out that the drummer has taped his headphones to his head with gaffers’ tape. A blinding, cold white look. Notice the vertical white stripes visible below the video screen panels. Those are fluorescent light bulbs, rigged to turn on and off instantly in chase sequences. The LD told me there is one brand of bulb in one specific fixture that will do that. I wish I had gotten a good pic showing how effectively the fluorescents provide cold, dynamic light.   Another interesting thing about this photo is that the bass player is sitting behind the drums, the lead singer/keyboard player is also playing drums, and the drummer is doing some improvised percussion on top of the keyboard. This is part of what makes Mutemath so much fun to watch. I didn’t get a picture of one of my favorite looks. They managed to find some huge, clear incandescent bulbs and hang a couple of them on either side of the video screen. With the stage nearly dark and filled with haze, the bare bulbs were turned on to a low intensity, allowing the glowing filaments to illuminate the haze with an eerie warm glow. This was easily the most creative lighting I’ve ever seen in my limited concert experiences. I did stop by after the show to compliment the LD, but I didn’t get his name, and there are so many things I want to ask him!

My current Gentoo Linux setup

Mon, 12 Nov 2007 22:15:03 +0000

This post is a reminder to myself more than it is intended for anyone else. Well, that’s most of this blog. Gentoo is a great distribution of Linux, but you have to install everything yourself. This is a good thing, because you get a very lean system, but the downside is that you are re-inventing the wheel somewhat with every install. So, without further ado, here is my current favorite configuration: Global USE flags:X cups dbus hal jpeg png gs dvi gif tiff mpeg bzip2 gtk lapack tetex -gnome -kdePackage list (basic software):* dbus (message bus system)* hal (Hardware Abstraction Layer) gentoolkit vim/gvim x.org xfce4 thunar mozilla-firefox acroreadPackage list (technical software): blas-atlas* lapack-reference numpy (unstable 0.90.1) matplotlib (Matlab-like plotting package for Python) texlive (experimental, hard-masked) lyx evince (dvi, ps, pdf viewer) paraview (3D visualization of datasets, unstable)

Blog infrastructure update

Mon, 08 Oct 2007 01:48:30 +0000

During the past week, the blog engine was updated and a few things were tweaked “under the hood.” You may have already noticed that the theme has been totally changed. It’s a modified version of the Blue Box theme by Bob–the header photo is mine.

Hog Programming: Cuelists vs. Palettes

Wed, 03 Oct 2007 17:40:00 +0000

I’ve always programmed the Hog 1000 lighting console using cue lists. Why? My prior lighting experience involved HogPC, which is essentially the same software running on a standard Windows PC. The HogPC software emulates the hardware of the Hog console with graphical buttons and wheels on the computer monitor, which can be manipulated with the mouse. This works, but it is much more awkward than having a physical console with real buttons. Using HogPC, it makes sense to pre-program as much as possible, because doing complicated things “live” is difficult using only a mouse (especially on a smaller monitor.) When I moved to the Hog 1000, I continued to rely on cue lists, mostly because I didn’t feel comfortable enough with the console to do things “live.” As I’ve grown more familiar with the console, I have begun to experiment with running moving lights off of palettes, rather than pre-programming cue lists. Cue lists have a couple of problems. On the Hog, a cue list can have either “latest takes precedence” (LTP) or “highest takes precedence” (HTP) as the priority scheme. HTP makes sense for conventional dimming-onlyfixtures (desk channels) and LTP makes sense for movers. Things get tricky when you mix desk channels and movers on one cue list. Another obvious problem is flexibility–when you have a pre-programmed list of cues and something changes at the last minute, you have to do some fast editing and hope you didn’t screw up something later in the cue list. Palettes offer the flexibility of running “on-the-fly”: choose fixture(s), color, position, and hit ENTER. That’s great when you don’t know what to expect. One disadantage is complexity–you have to hit a lot of buttons quickly, and do it again for every transition. It’s easy to mess up. Another difficulty is adjusting the rate of the transition–it’s hard to make two movers do a slow sweep across the stage while eight others snap into position. This is also a problem for smooth color fades on LED fixtures. So far, the best strategy I’ve found is to mix the methods. Program cue lists for rehearsed shows with known, fixed programs. For a more fluid situation, program cue lists with only position and fade time for moving lights. Use the cue list(s) for position (and maybe focus) and manually set color and gobo on the fly using palettes. Keep movers and desk channels on separate cue lists. That’s my best practice so far…but I’m always learning and open to ideas.

ODEPACK re-released

Wed, 26 Sep 2007 03:44:58 +0000

Minutes ago I uploaded a tarball containing my re-release of ODEPACK, a standard numerical tool for the solution of systems of ordinary differential equations. The algorithms and numerical code are identical to the distribution found on Netlib, but everything has been placed into an organized “package” with Makefiles, examples, documentation, etc. The code is set up to be compiled as a shared library that you can link to your code. Download and enjoy! And tell me if you’re using it!

Direct access to C++ containers from Python

Thu, 13 Sep 2007 16:31:01 +0000

In previous examples, I’ve shown how to pass Python lists into C++ using Boost.python. Because Python lists can contain a mixture of objects of different types, C++ has to use extract<type> to determine what kind of data to get from a list item. This approach works well, but it has its faults. In the C++ code, you might need to have duplicate data structures (a boost.python list to interchange data with Python and a C++ container such as a vector to process the data) and the code to convert between them. If you want to use your C++ library with languages other than Python, you want a non-Python-specific interface. Wouldn’t it be great to define your core data structures using a standard C++ container (like vector) and then access and modify them in Python? You can, but it’s non-obvious and it’s not well explained in the Boost.python docs. Read the docs for the indexing suite and take a look at my demonstration code for exposing C++ vectors to Python, and you should be able to figure it out. Once you get it working, it’s a very clean way of doing things. Just remember, because you’re working with C++ containers that accept only the specified type of object Python will throw a TypeError if you try to append the wrong kind of object.

Mixing objects of different types in a Boost.python list

Wed, 08 Aug 2007 15:35:26 +0000

Demonstration Code–mixing objects of different types in a Boost.python list In Python, a list is allowed to contain objects of various types. Using Python’s introspection capabilities, it’s easy to process a mixed list because you can just test each list member to see what kind it is, using isinstance(objectA, TypeB). This creates a problem when the list is passed into C++, where arrays and containers are designed to hold objects of one type. I encountered this problem when writing a small module to parse mathematical expressions. The expression is entered as a string in Python, which is parsed and “compiled” into a stack of objects. The compiled stack is passed to C++ using boost.python, where it is processed when needed. The stack consists of two or more fundamentally different objects: numerical constants and operators. The evaluation routine pops an object. If it’s a numerical constant, the value is placed on the operand stack. If it’s an operator, the appropriate operation is performed on the values at the top of the operand stack. With help from Alex Mohr and Christopher Woods on the C++-sig mailing list, I developed a method that is illustrated in the example. In C++, define a polymorphic base class (has at least one virtual method) and several classes that are derived from the base class. These classes are exposed to Python using Boost.python so that objects can be created in Python. A list of mixed objects is then passed back to C++ for processing. When trying to access a member of the list, C++ doesn’t know what kind of object to expect, so we have to use the Boost.python function extract<>() (see previous post for an introduction to extract). Now we extract a pointer to the base class using: object = extract <BaseClass*> (list[index]); This works because every object in the list is derived from the base class. As shown in the example, I first tried to do this with references (to avoid the whole pointer mess) but it did not work, so it seems that pointers are unavoidable here. It seems that this use of introspection is relatively rare in OOP, and is sometimes actively discouraged. However, for this type of problem it seems like the most logical design. Is there a different design pattern that would work better?

USPTO tries peer review for software patents

Wed, 04 Jul 2007 21:55:06 +0000

The US Patent and Trademark Office (USPTO) has launched a trial program to subject software patents to peer review online. Naturally, the big software companies aren’t exactly publicizing this. We all know that a lot of stupid patents get issued. Well, stop complaining and start reviewing! Let’s make this trial successful!

Truss warmers

Sun, 03 Jun 2007 22:39:29 +0000

Truss warmers are lights used to illuminate truss. Truss is fun to light because it’s semi-reflective and has a neutral color, so it easily reflects whatever color is used to light it. We used a little par light on one end of a section of truss to turn it blue, with a “cap” taped on the other end to keep the light from spilling out onto other stage objects. We also had some Trackspot lights sitting on the floor. Because the hazer wasn’t working they were pretty much useless, so I positioned two of them to point into the ends of truss that didn’t have blue lights. The Trackspots have a dozen or so colors built in, so it was possible to change the color of the center truss. Rumor has it we have some small LED color-changing fixtures on order…those should be great fun as truss warmers!

April 22, 2007: The importance of haze

Sun, 27 May 2007 16:34:20 +0000

If you are not “into” lighting, then you may not have paid any attention to the beams of light that seem to descend from the lighting fixtures towards the stage. There’s nothing special about the lights –the beams are produced when the light is reflected from millions of tiny drops of haze. Haze is produced by a machine that’s located above the stage behind the teaser (upper curtain). Haze is distinct from “fog.” A fog machine produces clouds of visible, dense, rolling fog, while haze is supposed to hang unobtrusively in the air until it’s hit with a light beam. Ideally, you would never see the haze being produced, but because of the location of our hazer, it sometimes looks more like fog when it gets blown around by the air handlers. The reason I’m writing about haze this week is because on April 22 our haze machine was out of commission. Look at the photos from this week and compare them to previous weeks. The haze adds color and depth to the stage. I’m excited that we’re finally adding color to the front of the stage. For Easter, we added three pars near the front on each side of the stage, with blue gels. They were a critical part of achieving this look:

Why the people are blurry

Mon, 21 May 2007 03:50:00 +0000

This is a simple post to explain why the people are usually blurry inmy lighting photographs. As an illustration, I took three photos of a speaker on stage, using different exposure times. An exposure time of 1/40 second produces a sharp speaker, but an overally ugly picture because of the noise in the image. Look at the graininess that is visible in the full-size image. Exposure time 1/15 second: Exposure time 1/5 second: At 1/5 second the noise is gone, but now the exposure is so long that the speaker moves during the exposure, producing the “halo” effect. Since my goal in taking these picture is to showcase the lighting, I use a longer exposure to capture the darker parts of the image, and I live with the fuzzy people. Another option is to get a more expensive camera with a more sensitive sensor and a broader range of apertures (my camera only supports two).

April 1, 2007 Lighting for solo vocalists

Sun, 13 May 2007 21:21:23 +0000

This weekend, two songs started with a solo vocalist. I used the front lights to highlight the solos and provide a transition when the rest of the singers and the band joined in. In the following song, the female vocalist at stage left leads the song, so I highlighted her for the first look by dimming the front lights on the rest of the band and singers. I then brought up front lights on the other singers and band members. I did the same thing for this song, but with a different lead vocalist. And three more songs… We did one cool thing that I didn’t photograph. We have six pars on the ceiling throughout the auditorium that light the walls. With blue gels, we used them as “house lights” while communion was being served.

Example: using an extractor class in Boost.Python

Mon, 16 Apr 2007 03:12:23 +0000

The documentation for the Boost.Python extractor class is not terribly enlightening for a new user. The examples given in the tutorial are code fragments, and it’s never clear exactly what the context is or what they are supposed to accomplish. That’s a shame, because the extractor classes are really cool and easy to use, so people should use them! Here is a Boost Python extractor example that should make things clear.

How do you say "bye bye Ubuntu" in Zulu?

Wed, 11 Apr 2007 18:48:46 +0000

I finally had enough and wiped Ubuntu off of my computer at work, and now Gentoo is happily running. Maybe I could have gotten Ubuntu to do what I wanted if I had spent more time with it, but I was sick of playing with it instead of getting work done. To be fair to Ubuntu, maybe I’m just more familiar with how to do strange things with Gentoo. Things like installing and using my open-source toolchain:

Climate change catastrophe: filler for a slow news day

Sat, 07 Apr 2007 19:41:40 +0000

When there’s a slow news day, and no interesting disaster has taken place to make a good headline, you can always count on global climate change to fill in the gaps. Yesterday we learned about “natural wonders” that are being destroyed by climate change. Of course, this is a press release from the World Wildlife Fund, not a peer-reviewed scientific paper and in fact not even issued by the IPCC. I think the most interesting aspect of this article is the implied viewpoint that climate change can somehow be prevented. WWF’s chief scientist said, “…more attention will have to be paid to adapting to change, not only trying to prevent it…” In the controversy over whether or not humans are causing global warming, environmentalists seem to have forgotten that the global climate has been changing, all on its own, for millions of years. Probably millions of species have been wiped out, and natural wonders destroyed, by past climate change long before humans came along. The “natural wonders” that are familiar to us exist because they happen to thrive in the particular climate conditions that occured in the past several thousand years–geologically speaking, a very short time. When conditions change, nature will change. Even if we can prevent one component of climate change that is caused by humans, the world will change, species will die out, and people will suffer. Welcome to life on Earth–adapt or go extinct. On a side note: “…diplomats and scientists were negotiating the text of a 21-page summary of the full 1,572-page scientific report.” Now that sounds like formula guaranteed to achieve accuracy and exclude bias. Also, now available for download: the executive summary of the 4th Report, 1st Working Group, from the International Panel on Climate Change. Unfortunately, we’ll have to wait for May to get the full report and see what it really says.

Science by press release

Sat, 31 Mar 2007 15:44:22 +0000

There’s a reason that the scientific community uses a peer-review process to validate publications. Unfortunately, the “global warming” branch of climate science doesn’t operate this way.   Recently, excerpts of a report from the UN International Panel on Climate Change (IPCC) were leaked to the press ahead of the release of the full report. As usual, they released the most scandalous and exciting bits and left out all the qualifiers and complexity that make for good science but poor headlines. In this case, “Australia will suffer more droughts, fires, floods and storms” and the “Great Barrier Reef will be devastated by 2030.” We’re not supposed to ask how they came up with these predictions. Further, with a report prepared by more than 2000 scientists and beaurocrats, nobody actually has to face consequences if their predictions don’t hold true. I wish I had time to get into the details of climate models and explain how these things work, and expose their strengths and weaknesses….

Discovery Lights, 17-18 March 2007

Mon, 26 Mar 2007 02:38:45 +0000

Challenges for this week: The set has been the same for over a month, so I had to come up with something to distinguish it from previous weeks. In addition, a couple of weeks ago our four Trackspots were transferred to another venue, leaving us with two Studiospots as the only intelligent lights. We have two more that have been backstage awaiting repairs for months. I’d like to see them back up and running, but we are a church, and if we’re using our budget to help poor people instead of fixing our lights, I’m okay with that. The first look is the one I used before the band went onstage. The next look was used when the band started playing an instrumental intro to the first song. The center screen showed an eye-candy video while the side screen showed announcements. Both Studiospots were set to white, with open gobo and a random effect so that their beams moved around the front of stage. Here they are shown hitting the drummer and guitar player. The first song was a continuation of the intro, so I used the same colors but brought up the front lights. I used the red lights on the backdrop, and blue lights on the truss, to provide transitions in this song. I was saving the movers for a later song… I used a totally different color to distinguish the next song, with two different looks. It was a cool day outside, so the air conditioner wasn’t running much and the haze hung in the air beautifully. I used the Lava gobo with a very slow rotation in the two Studiospots for this song. To provide transitions within the song, I moved the lights from the lead singer, to the backdrop, to the guitarist/drummer for a solo, and back to the singer. It doesn’t show up that well in the photos, but it was a nice effect. For the finale, I lit up the truss in the back. I used two basic looks (with the fixed lights) for the final song. The band continued to play an instrumental version while the audience greeted one another, so I dimmed the front lights for the third look. It doesn’t show up in the pictures, but I used the studiospots to hit the white draped fabric with a subtle rotating gobo effect. I like using the gobos on irregular surfaces where you can’t tell that it has a cheesy shape. Announcements and message. For the announcements, I pointed the studiospots across the stage to illuminate the backdrop with a deep blue/purple gobo. During most of the service, the stage is too brightly lit to achieve these highly saturated colors.

Who got fired for this?

Mon, 19 Mar 2007 02:34:32 +0000

What an unusual pine tree! Who would have guessed that this is actually a giant antenna, cleverly disguised as a tree? I just hope that my taxpayer dollars weren’t used on this monstrosity.

How the E815 dies

Mon, 19 Mar 2007 00:21:30 +0000

I have a Motorola E815 cell phone. This morning, I was reminded why it hasn’t been an entirely satisfying experience. The E815 has excellent reception (on Verizon’s network) and lots of software features that work really well. Unfortunately the mechanical quality leaves a lot to be desired. The E815’s weakness is the socket where the wall charger attaches. My first E815 crashed as I was unplugging the charger. I unplugged it and the phone rebooted. This had happened several times before, but this time the phone got to a certain point in its boot cycle and rebooted again. I couldn’t even turn it off using the end/power key–I had to remove the battery. Fortunately there were about three weeks left on the warranty, and Verizon eventually got me a new one that worked (on the third try). Last night, I plugged in my phone before bed and set the alarm. Early this morning the alarm went off, so I opened the phone and hit the “end” key to shut it up. The phone rebooted and came back to the alarm screen. I unplugged it and hit “end” again, and it worked. Moral of the story–be VERY careful when plugging/unplugging your E815 charger, and try not to do anything with the phone when it’s plugged in. Also, sign up for Verizon’s free backup service to save your contacts in case the phone dies, and transfer your photos and ringtones to a micro-SD card.

"Ubuntu" actually means "eats your config files"

Wed, 14 Mar 2007 20:08:52 +0000

The Dapper-to-Edgy upgrade was surprisingly smooth, once I figured out how to do it. Unfortunately, the new version overwrote my “/boot/grub/menu.lst” so I could not boot Windows. I have enough experience with Linux that this was a five-minute fix, but it could be pretty scary for an inexperienced person. Who knows what else got over-written and I just didn’t realize it. I was further disappointed to find out that the version of PIDA in Edgy is broken.

Ubuntu is an African word that means "devours your soul"

Mon, 12 Mar 2007 22:57:45 +0000

When it comes to Linux, I’m a Gentoo guy. Go ahead, make fun of me and call me a ricer. Then go try to do something difficult on Redhat or Ubuntu, and come back and tell me what you think. I needed a Linux development environment at work, so I took my Windows machine and added a Linux partition. Since Ubuntu has been getting a lot of good press, and I wanted something that would install fast, I tried Ubuntu. Specifically, I installed Kubuntu, because Linus Torvalds doesn’t use Gnome. It really did install quickly and everything worked great out of the box. I installed a number of packages, and they worked too! Too good to be true? Absolutely! It all started when I wanted to compile something. Silly me, I thought gcc was a basic part of a basic Linux installation…but no, that must be a Gentoo bias, so I installed gcc. Then I decided to install numpy, the latest numerical toolkit for Python. It’s been around for a couple of years as a beta, so I figured there must be an “unstable” build out there. Nope. To get numpy, you have to upgrade from Ubuntu 6.06 “Dapper” to 6.10 “Edgy”. <sarcasm>Because it makes sense to upgrade gcc, KDE, Firefox, etc just to get a new Python package.</sarcasm> Now, go find instructions on how to upgrade the Ubuntu distribution. Then try those instructrions and realize that they won’t work at all for Kubuntu because the package managers are totally different. While you’re there, read the horror stories on the forums from people who followed the “official” upgrade instructions and ended up with less-than-functioning systems. FYI, the Kubuntu upgrade instructions are actually easy to find once you know where to look. Kubuntu 6.10 is installing right now, so tomorrow we’ll see how it went.

Python calling C++ calling Fortran

Fri, 09 Mar 2007 04:40:03 +0000

I got it working a while ago, but I haven’t had a chance to update my eager audience. I built a simple simple simulation tool using this crazy combination of languages. It’s a library written in C++ that uses the finite volume method to simulate diffusion in one dimenstion. What’s cool is that it’s designed to be called from Python using Boost. Furthermore, it then uses Boost Bindings (not officially part of Boost) to call LAPACK (a Fortran library) to solve linear equations (as shown in a previous post). It’s pretty cool that all this stuff actually works together. Why did I do this? Python is my favorite very-high-level language for doing scientific programming. Compared to C++ or Fortran, it’s just so easy to do things in Python, like making plots or reading in an xml file. I chose C++ for the numerical codes because it’s faster than Python and the Boost library connects the two. I could have used Fortran, but the Python interfaces doesn’t seem to be as clean, probably because Fortran isn’t object-oriented. Finally, since the reference implementation of LAPACK is still in Fortran, I just used the Boost bindings to call it. I might switch to a native C++ linear algebra library if a clear leader emerges from the confusing alternatives that are available today. More complaints on that later…

Theme construction in progress

Tue, 06 Mar 2007 04:11:06 +0000

I’m working on a new and improved theme for the lighting page. It’s still a bit rough because I don’t have time to finish it today, but I think it looks promising!

Subtle aspects of using Boost and LAPACK

Tue, 27 Feb 2007 23:43:59 +0000

I spent part of the afternoon wrestling with LAPACK integration in a C++ program, using the unofficial Boost bindings. I learned a couple of interesting things. test.C is a file that demonstrates them. 1. You need to create column-major matrices to pass to LAPACK. The subtle part about this is that the matrices are internally stored as column-major (like Fortran) but are accessed like row-major matrices (like C). In other words, even when you declare A as column-major, A(3,5) refers to the third row and fifth column of A. This was not obvious to me. 2. The LAPACK routines may modify the matrices that are passed to them. For example, when using gesv(A,B) to solve a system of linear equations, the matrix B is modified to contain the solution values–this is obvious. It is not obvious, at first, that the matrix A is also modified (and I don’t know what useful information it contains). 3. There does not seem to be any binding to the specialized LAPACK solvers for banded and tri-diagonal matrices. I would like to know why, since my matrix is tri-diagonal and it can probably be solved considerable more quickly.

The need for 3D technical drawing software

Mon, 26 Feb 2007 21:42:17 +0000

Today I was reminded again of the need for a new kind of technical drawing software. Here is the problem I am facing: I want to create a vector drawing of a three dimensional structure. Right now, I can either create it in CAD and export a bitmap, or I can manually draw a 3-D representation using 2-D vector software such as Dia or Xfig. For publication-quality graphics, the first approach is usually out. The second approach works, but what if I spend an hour drawing, only to realize that I should have chosen a different perspective or viewpoint to show a critical feature of the drawing? I have to modify the entire drawing just to change the viewpoint. Or, what if I need to show two views to capture all the details? I have to create both from scratch. I envion a new type of tool, or CATD (Computer Aided Technical Drawing). My vision starts with a 3-D CAD tool, which could be built around lines, points, and surfaces (like AutoCAD) or a solid modeling kernel (like SolidWorks). The user should have the ability to choose surfaces to be transparent or solid, and create cut planes to show internal features. However, the 3-D model is only the first step. Once the user has created the model and chosen a viewpoint that shows its important features, the 3-D model is projected onto a 2D surface to create a vector drawing that can be further manipulated. Ideally, the 2-D vectors created from each “object” in the CAD model would remain grouped together for easy manipulation. The final output would be a publication-quality PostScript, PDF, or SVG file. I don’t know if this is commercially available, but it certainly isn’t available as open source. As a starting point, I propose creating a plugin for Blender that fulfills this purpose. It would take the 3-D model, as displayed on the screen, and create a vector 2-D representation that can be imported into other software for manipulation. Ultimately, the tool should allow users to import different types of CAD models. The text should be somehow integrated with LaTeX for seamless typesetting of equations and numbers.

LISP is cool but annoying

Mon, 26 Feb 2007 05:09:14 +0000

I spent today organizing my personal finances in GnuCash. I also spent a few hours customizing a report, which unfortunately means hacking in Scheme, a “dialect” of LISP. Don’t get me wrong, I actually like LISP/Scheme quite a bit. What I don’t like is writing code in “normal” languages, like C++ and Python, all week long and then switching to Scheme. Scheme has a totally different syntax and requires a different way of thinking about a problem. Normal language: result = myfunction (argument1, argument2) Scheme code: (myfunction (argument1 argument2)) If you have never seen a LISP language, find one and check it out. Here are the links I always use:

Lighting for February 11/12

Sat, 17 Feb 2007 22:25:15 +0000

Nothing too exciting this weekend in the lighting department. In a change from last time, two Studiospots were flown from a pipe just in front of the stage. I don’t think they’re real useful in that position because they’re too close to the stage to really light a person, and we normally don’t use enough haze to make the beams really visible. I used them to create patterns on the white backdrop, but you can’t see them very well in this picture. We did manage to get some good coordination between the lighting colors and the the background graphics. Here’s another example of how the camera changes the look of the colors. In this case, the camera has increased its sensitivity to capture the darker portions of the image, which caused the blue and amber lights on the backdrop to appear much more intense than they actually were. In reality, they were light pastel colors. I need to experiment with my camera to find out if one of the metering modes will improve this.

The beginning of the lighting blog

Sat, 17 Feb 2007 05:02:40 +0000

I’m of the lighting operators/designers at Discovery Church in Orlando, FL. I’ve been doing lighting, off and on, since 2002. Unfortunately I didn’t get a digital camera until November 2006 so I don’t have much of a record of the early years. Now I’m trying to photograph everything I work on. It’s partly to help myself improve as a lighting designer, partly for fun, and hopefully somebody else benfits too. I do try the occasional “artistic” shot, but primarily I’m concerned with documenting the look rather than taking the perfect photo. Thus, the performers tend to be blurry because I’m using a long exposure (and no flash, of course) to capture the whole stage.

About Craig Finch

Fri, 16 Feb 2007 03:27:36 +0000

I am a Senior Pre-Sales Cloud Engineer/Solutions Architect with SADA, an Insight Company. I help customers take full advantage of Google Cloud (Workspace, Maps, and Google Cloud Platform) by providing consultative advice and scoping professional services engagements. I work with mid-sized corporations to large enterprises across a variety of industry verticals, but primarily in high-compliance areas such as fintech and information security.

Previously, I was a Principal Consultant and co-founder at Rootwork InfoTech LLC, a professional services business meeting IT infrastructure needs of small to mid-sized organizations. I sold my share of the business to my business parter and co-founder Mike Soule, and he continues to operate the business. If your organization is “too small” to engage with the major IT services companies, but you need a high level of enterprise-grade infrastructure expertise, please reach out to Mike at Rootwork.

shocksolution.com

Accessing a Mulesoft API mock with curl

Contact Craig

Google Apps Script: Spreadsheet to Document

Google Cloud Build for Cloud Functions

Global vs Regional

Terraform: for_each on a list of resources

Example: assign a unique role on each resource

Using SSL Certificates with the Apache Tomcat Web Server

Creating PKCS12 Files

Auto-Create Multiple Blocks in a Terraform Resource

Creating Multiple Resources

The Best Mental Model for Writing Terraform Code

Define a Google Load Balancer and Cloud Storage bucket with Terraform

How to Enable Multi Factor Authentication (MFA) on your IEEE Email Account

Multi-Page Blueprints for Confluence

Background on Confluence

Node.js: Connecting to MySQL on a socket with Sequelize

Enable Port Statistics on a Netgear M4200/M4300 Switch

Use an SSL/TLS Certificate with ESET Security Management Center Appliance

Alert Policies for Log Metrics on Google Stackdriver Monitoring

Create a Logs-Based Metric

Testing Access to Google Bigtable

Dynamic node groups with Salt

How to use SSL/TLS certificates on a Netgear M4200/4300 switch

curl or libcurl: SSL certificate problem: unable to get local issuer certificate

Troubleshooting Strategy

2020 Ubiquiti Wireless Access Point (WAP) Roundup

First Generation Access Points

Monitoring a switch or router port via SNMP

Use Case

openssl unable to read/load/import SSL private key from GoDaddy

Nginx default user changed in RedHat/CentOS package nginx-1.16.1-1.el7

mount.cifs error 13 after update (CentOS/RedHat Linux 7.6)

Configuration that triggered the problem

Secure, Automated Network Device Backups

Another Reason to Enable Private IP Addresses on Google Cloud SQL

QNAP NAS: Backup & Sync to Google Cloud Storage

Disk space utilization monitoring in Google StackDriver

File System Type Filter

Managing remote servers with salt-ssh

Find Last Login Time for Wordpress Users in the SQL Database

Question: What are the timestamps for the last login for every Wordpress user in this Wordpress site?

MySQL replication fails with WSREP error after restart

Scenario

Updating to Wordpress 5 on a Linux Host: Requesting FTP Credentials

Troubleshooting Wordpress permissions errors on Linux hosts

Netgear GS728TS, GS728TPS, GS752TS, and GS752TPS Switch Configuration Guide

Overview of the Netgear GS Switch Series

Creating Kubernetes Secrets Using TLS/SSL as an Example

Prerequisites

Yum/RPM Package Signing Key Fingerprints

The Importance of Checking Keys

Blog

Home

Optimize the Wordfence Firewall: Needs FTP Credentials

How to run Shadowrun: Dragonfall on Ubuntu Linux

Installation Procedure

New Relic APM on Fedora/CentOS/RedHat with SELinux Enforcing: Solved

Internet Service Comparison: Spectrum Cable Modem vs AT&T Fiber

Spectrum Cable Internet

Running Unity games via Steam in Linux

Prerequisites

Audit terminology: SAS 70, SSAE 16, SSAE 18, SOC1, SOC2, Type 1, Type 2

Configuring Laravel/Lumen applications to connect to SQL database sockets

Configure CentOS/RedHat VMs with Kickstart files on Virtualbox

Service account credentials with the Python client for the Google Drive API (v3)

General error: 2006 MySQL server has gone away

Viewing logs for a cluster of instances on Google Stackdriver Logging

Protecting a RESTful JSON API from a CSRF attack

Upgrade Ubiquiti UniFi Access Points (WAP) now to avoid KrackAttack

Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

Symptoms

Diagnosis

Managing persistent disks on Google Compute Engine

Migration from GoDaddy to Wordpress.com hosting

About

Contact

Is there a "Primary Domain Controller" in Active Directory?

The Historical Answer