Linux on shocksolution.com

Using SSL Certificates with the Apache Tomcat Web Server

Tue, 30 Nov 2021 03:20:26 +0000

Creating PKCS12 Files

PKCS #12 is a format for storing multiple cryptography objects in a single archive file. You can store arbitrarily complex objects within a PKCS #12 archive, but the most common use is to store a single private key and its certificate chain. Create a PKCS12 file from PEM files:

openssl pkcs12 -export -in ssl_cert.pem -inkey key.pem -certfile bundle.crt -name "*.example.com" -out example.com.p12

Depending on the product you’re working with, the documentation may call for a .pfx file instead of a .p12 file. PFX is an older format that was a predecessor to PKCS #12. In most modern systems, A PFX FILE AND A P12 FILE ARE EXACTLY THE SAME THING! You can just change the extension if needed and that will generally work. For example, the docs for ESET Security Management Center call for a pfx file, but a PKCS12 file will work just fine.

Node.js: Connecting to MySQL on a socket with Sequelize

Thu, 22 Apr 2021 18:54:25 +0000

According to its official description, “Sequelize is a promise-based Node.js ORM for Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server.” Sequelize is widely used in Node.js web applications to abstract the database layer. I recently found a “corner case” for Sequelize which is possible to accomplish, but is not well documented anywhere on the web. I need to use Sequelize to connect to a MySQL server via a UNIX socket. I’m developing some plugins for Confluence that use the atlassian-connect-express toolkit, which has a datastore that wraps Sequelize. The way Atlassian has chosen to wrap Sequelize is rather unfortunate, probably because they are trying to maintain backwards compatibility with JugglingDB. Essentially, the Atlassian Connect Express only looks at the URL, and ignores any option passed to Sequelize. Therefore, you have to pass everything you need via the URL, and this is where it’s tricky. Here’s the form of the URL that you need to connect to MySQL via a UNIX socket with Sequelize:

Use an SSL/TLS Certificate with ESET Security Management Center Appliance

Wed, 11 Nov 2020 21:19:29 +0000

ESET Security Management Center (ESMC) is the replacement for the ESET Remote Administrator (ERA) Server. You can install ESMC as a “virtual appliance” which will run on hosts such as VMWare or Linux/KVM/QEMU. Under the hood, this appliance is a CentOS 7 server running the Apache Tomcat web server. ESMC is implemented as a Java application that runs as a systemd service. In my experience, ESET is a solid security product, and the free ERA/ESMC provides some rudimentary endpoint management capabilities. Unfortunately, everything about ESET is confusing, from the naming of their products to their documentation. The documentation is thorough…it’s just hard to find what you’re looking for, which is usually in multiple overlapping documents that each contain part of the puzzle.

Testing Access to Google Bigtable

Fri, 05 Jun 2020 20:14:15 +0000

This is a minimal PHP script that’s useful for verifying connectivity and permissions to read data from a Google Bigtable instance. Unlike some of the sparse examples in Google’s PHP SDK, this script will work for any table, and you don’t need to know anything about the structure of the table. This script tests the critical elements that need to work together to enable Bigtable access:

Credential file GRPC extension for PHP Protobuf extension for PHP Scopes enabled on Compute Engine instances

curl or libcurl: SSL certificate problem: unable to get local issuer certificate

Tue, 10 Mar 2020 17:00:17 +0000

curl, or an application that uses libcurl, may have a problem with an SSL certificate that works fine when using a web browser to access the same URL. Typical error output from curl looks like this:

$ curl -v https://my-subdomain.mysecuresite.com

 Trying xxx.xxx.xxx.xxx:443…
 TCP_NODELAY set
 Connected to my-subdomain.mysecuresite.com (xxx.xxx.xxx.xxx) port 443 (#0)
 ALPN, offering h2
 ALPN, offering http/1.1
 successfully set certificate verify locations:
 CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: none
 TLSv1.3 (OUT), TLS handshake, Client hello (1):
 TLSv1.3 (IN), TLS handshake, Server hello (2):
 TLSv1.2 (IN), TLS handshake, Certificate (11):
 TLSv1.2 (OUT), TLS alert, unknown CA (560):
 SSL certificate problem: unable to get local issuer certificate
 Closing connection 0
 curl: (60) SSL certificate problem: unable to get local issuer certificate
 More details here: https://curl.haxx.se/docs/sslcerts.html 
 curl failed to verify the legitimacy of the server and therefore could not
 establish a secure connection to it. To learn more about this situation and
 how to fix it, please visit the web page mentioned above.

Troubleshooting Strategy

A good starting point for any SSL error on a public-facing URL is to analyze the URL at SSL Labs.
Does the error happen for all certificates issued by a specific Certificate Authority (CA)? If so, the system running curl may need to have a root certificate for that CA added or updated in its certificate repository. That’s a relatively rare problem, but might occur if the system running curl is very old. There’s a comprehensive thread about this issue on Stack Overflow.
If the error only happens for one specific site, it’s likely that the site is missing an intermediate certificate. The command to diagnose this issue is also found in that Stack Overflow thread:

openssl s_client -connect myhost.com:443 -servername myhost.com -showcerts

The output should show a series of certificates, starting with the site certificate, and ending with the root certificate for the Certification Authority. If this chain only shows the site certificate, that’s the problem.

Monitoring a switch or router port via SNMP

Mon, 16 Dec 2019 20:19:52 +0000

Most business-class switches and routers support monitoring via SNMP. The main barrier to implementing SNMP monitoring is that so many parameters can be monitored, and it can be very hard to isolate the most important parameters. This example shows how to monitor the status of a specific port. Note that the term “port” includes link aggregation groups (LAGs) or other logical groups of ports like VLANs.

Use Case

A top-of-rack switch has several important link aggregation groups (LAGs). For example, one LAG has connections to two identical routers in a VRRP cluster. Other LAGs provide redundant network paths to independent NICs on the same server. I want to be alerted if the port status changes on any critical LAG. The ISP delivers Internet to a single port on the switch, which I also want to monitor.

openssl unable to read/load/import SSL private key from GoDaddy

Mon, 02 Dec 2019 13:24:00 +0000

openssl is the standard open-source, command-line tool for manipulating SSL/TLS certificates on Linux, MacOS, and other UNIX-like systems. I recently ran into an interesting problem using openssl to convert a private key obtained from GoDaddy. Someone else used GoDaddy’s “wizard” interface to generate a certificate signing request (CSR) and private key, and saved the files on their Windows workstation. They purchased an SSL cert from GoDaddy, and shared all the files with me for installation on servers. GoDaddy saved the private key in the newer PKCS #8 format (pkcs8), and one system required the key in the older PKCS #1 (pkcs1) format. It’s easy to tell the difference.

Nginx default user changed in RedHat/CentOS package nginx-1.16.1-1.el7

Tue, 12 Nov 2019 02:58:34 +0000

The latest Nginx rpm (nginx-1.16.1-1.el7) from EPEL for CentOS/Redhat is a fairly major update, from version 1.12 to 1.16, and includes some very important fixes to address multiple CVEs. There is another small but critical change in the latest RPM that isn’t mentioned in the bug report. RedHat had configured nginx to run as user “apache” in past releases, but in the EPEL release on 19 October 2019, the configuration was changed to run as user “nginx”. If you ran “yum upgrade” recently (which you really need to do on a regular basis) on any system with nginx installed, you probably have a system that is affected.

mount.cifs error 13 after update (CentOS/RedHat Linux 7.6)

Wed, 23 Oct 2019 16:23:48 +0000

PERMISSION DENIED

PERMISSION DENIED (Photo by Hidde van Esch on Unsplash)

If you recently updated your RedHat or CentOS 7.6 system, you may suddenly start getting “Permission Denied” errors when attempting to mount SMB shares via CIFS. Typical error messages in syslog look like this:

kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE
kernel: CIFS VFS: Send error in SessSetup = -13
kernel: CIFS VFS: cifs_mount failed w/return code = -13

Configuration that triggered the problem

Synology NAS with latest operating system shares a volume via SMBCentOS 7 Linux server mounts SMB share using a local username and password (NOT domain credentials)cifs-utils version 6.2 installed in May of 2019 (this by itself worked fine)libsmbclient just updated to 4.9.1-6libmount just updated to 2.23.2-61CIFS mount options: vers=3.0,credentials=/root/credentials.txt,sec=ntlmsspi* File /root/credentials.txt contained a username and password that are LOCAL to the SMB server

Secure, Automated Network Device Backups

Tue, 03 Sep 2019 16:38:30 +0000

You probably have at least a few business-class switches and routers silently do their job, year after year. Network devices can be so reliable that we configure them once, and then forget that they exist. However, forgetting them is a really bad idea, since they require regular OS or firmware updates to patch vulnerabilities, and at some point, they will become unreliable and fail. If your device fails, do you have the configuration available to apply to its replacement? Are you sure the configuration that’s in your archive or documentation is up to date? This post deals with automating backups from Netgear switches and Ubiquiti EdgeRouters (and possibly switches, if they run EdgeOS). Please check out the accompanying repo at https://github.com/craigafinch/network-backup-scripts

Disk space utilization monitoring in Google StackDriver

Tue, 28 May 2019 18:22:43 +0000

Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).

File System Type Filter

Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.

Managing remote servers with salt-ssh

Mon, 13 May 2019 21:20:37 +0000

I love using Salt to configure and manage servers at scale. Typically, a Salt master server manages an army of servers which are on the same private network. A salt-minion daemon runs on each client server and communicates with the master. Sometimes you need to manage a server which is outside of your internal network, such as a marketing web server running on a service like AWS LightSail, Linode, or Digital Ocean. That’s when salt-ssh comes in handy.

Find Last Login Time for Wordpress Users in the SQL Database

Tue, 16 Apr 2019 16:49:21 +0000

Most answers say to install some plugin, and there are dozens that will show the date and time when each Wordpress user last logged in. However, plugins are a problem, because they always need to be kept up to date, and many plugins get abandoned. Now, you have a potential security problem. Also, this approach assumes that your Wordpress front-end is up and running; if your site has been hacked or accidentally broken, you need to go into the database to do some forensics.

MySQL replication fails with WSREP error after restart

Mon, 11 Mar 2019 12:16:20 +0000

Scenario

A single Percona MySQL 5.7 server replicates asynchronously with a cluster of Percona MySQL 5.7 servers. After rebooting the single server, replication failed with the following symptoms:

mysql> show slave status \G**************************** 1. row ************** Slave_IO_State: Waiting to reconnect after a failed registration on master Master_Host: my-replication-partner Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000120 Read_Master_Log_Pos: 3848512 Relay_Log_File: my-host-hame-relay-bin.000012 Relay_Log_Pos: 4 Relay_Master_Log_File: mysql-bin.000120 Slave_IO_Running: Connecting Slave_SQL_Running: Yes

I checked my MySQL text log (/var/log/mysql.log in my case, but the path may vary, or messages may go to syslog) and found the following error:

Updating to Wordpress 5 on a Linux Host: Requesting FTP Credentials

Mon, 14 Jan 2019 17:10:43 +0000

Wordpress 5 didn’t install automatically on sites hosted on my CentOS 7 Wordpress hosting server. That’s by design, because I’m selfish and I’d rather wait for a few bug-fix releases before making a major upgrade. When I chose to upgrade manually on a test site, Wordpress asked for my FTP credentials:

Connection Information: FTP credentials

If you’re hosting Wordpress on your own server, it does not need FTP credentials. This page is a symptom of a permissions issue with the underlying server. The web server (typically Apache) is unable to write the updated files to the Wordpress directory. From a security standpoint, this is actually a good thing. You generally don’t want to allow a public-facing service to make changes on your server. This problem occurs in other situations, such as configuring the WordFence plugin, so I’ve written a page devoted to troubleshooting Wordpress permissions on Linux hosts. That page describes how to fix the permissions issue without compromising the general security of your Wordpress host.

Optimize the Wordfence Firewall: Needs FTP Credentials

Sun, 26 Aug 2018 20:17:21 +0000

I use Wordfence as a Wordpress security solution. Even the free version has a comprehensive set of features, and the premium version is even better. It’s very easy to install and configure Wordfence as a regular plugin. However, the step to “Optimize the Wordfence firewall” fails for many people. This is not the fault of Wordfence, since different steps may be required depending on how your web server is configured. My solution is for those who are hosting their own Wordpress, but may be helpful to other situations. Please note that these instructions affect the security of your system, and are aimed at users with intermediate to advanced Linux knowledge.

How to run Shadowrun: Dragonfall on Ubuntu Linux

Sat, 16 Jun 2018 15:55:16 +0000

I’m very happy that Harebrained Schemes chose to release its Shadowrun game series (Shadowrun Returns, Shadowrun: Dragonfall, and Shadowrun: Hong Kong) for Linux. It can be a little difficult to get this stuff running on Linux. In this post, I will explain the errors you may see when trying to run Shadowrun on Linux.

Installation Procedure

I purchased and downloaded Shadowrun:Dragonfall on sale from Humble Bundle. 1. Verify file integrity. The md5 checksum is provided on Humble’s download page-check the output from this command against what Humble provides. This is how you make sure that you aren’t installing a corrupted or infected application on your system: [code language=“bash”] md5sum shadowrun-dragonfall-linux.tar.gz_2.0.9.zip [/code] 2. Unzip the downloaded file: [code language=“bash”] unzip shadowrun-dragonfall-linux.tar.gz_2.0.9.zip [/code] 3. Move the unzipped file (shadowrun-dragonfall-linux.tar.gz) to wherever you want to install it. I put it in a subdirectory called “Games” in my home directory: [code language=“bash”] mv ~/Downloads/shadowrun-dragonfall-linux.tar.gz ~/Games/ [/code] 4. Unpack the TAR archive. Note that there is a minor error on the part of the developers or Humble; the file you get after unzipping has the wrong file extension, so you have to use a different command than you normally would. The file should be called “shadowrun-dragonfall-linux.tar” because it’s not compressed with gzip. [code language=“bash”] tar xf shadowrun-dragonfall-linux.tar.gz [/code] 5. Run the game. This probably won’t work the first time; see the appropriate section below to install required packages on your system. [code language=“bash”] ~/Games/Shadowrun\ Dragonfall/Dragonfall Dragonfall: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory It seems Shadowrun Dragonfall has exited unsuccessfully (error = 127). Please verify that you have the latest drivers installed before filing a bug report here: http://harebrained-schemes.com/shadowrun/bug-reports [/code] 6. Install required packages on Ubuntu 16.04 (Xenial) 64-bit (x86_64). Shadowrun is a 32-bit game, but it runs on 64-bit Ubuntu because of Ubuntu/Debian’s multilib feature. You just need to install two specific 32-bit libraries that probably aren’t on your system by default: [code language=“bash”] sudo apt-get install libxcursor1:i386 libglu1-mesa:i386 [/code] You also need the appropriate graphics card drivers for your hardware. I have an Nvidia card and I use the proprietary drivers available directly from Nvidia (not the open-source version, which has caused problems for me with other games). If you had to do something differently on another version of Ubuntu, please leave a comment below.

New Relic APM on Fedora/CentOS/RedHat with SELinux Enforcing: Solved

Thu, 14 Jun 2018 03:27:28 +0000

I found the solution for getting the New Relic APM Agent to run on a CentOS 7 system with SELinux Enforcing. The only SELinux modification you have to make is to change the context of the log file directory /var/log/newrelic. You can temporarily make these changes with the chcon command: [code lang=“bash”] chcon -R -t httpd_log_t /var/log/newrelic/ [/code] Kill any running instances of newrelic-daemon. If you’re using New Relic’s recommended default setup, make sure that the file /etc/newrelic/newrelic.cfg does not exist. Verify that all required values in /etc/php.d/newrelic.ini are correct (especially the license key and the application name). Finally, restart httpd: [code lang=“bash”] systemctl restart httpd [/code] If that works, you should see output in both /var/log/newrelic/newrelic-daemon.log and /var/log/newrelic/php_agent.log. Now, make this file context change persistent, and verify the result: [code lang=“bash”] semanage fcontext –add –type httpd_log_t “/var/log/newrelic(/.*)?” restorecon -RF /var/log/newrelic/ ls -laZ /var/log/newrelic/ [/code] When you restart Apache, it spawns the newrelic-daemon processes, which communicate with APM via the socket file /tmp/.newrelic.sock. You really don’t want to run the newrelic-daemon processes via systemd if you can avoid it. The resulting process tree should look like this:

Running Unity games via Steam in Linux

Mon, 28 May 2018 19:15:33 +0000

There seems to be a lot of confusion about how to run games built on the Unity engine distributed by Steam on a Linux system. I recently solved this problem after backing the promising survival sandbox game Planet Nomads on Kickstarter. Here is how to configure an Ubuntu Linux system and launch Planet Nomads successfully.

Prerequisites

If you have an Nvidia card, you must download and install the official proprietary drivers provided by Nvidia. I could not get Unity to launch with the open-source drivers. Once I installed the proprietary (closed source) drivers, I was able to launch Planet Nomads from the Steam application.

Configure CentOS/RedHat VMs with Kickstart files on Virtualbox

Sat, 05 May 2018 18:54:50 +0000

Kickstart is a type of file that’s used to automatically install RedHat or CentOS Linux on a physical or virtual server. If you are managing more than a few servers, it’s a good idea to configure the servers via kickstart files instead of logging in and configuring each one manually. However, some unfortunate choices were made when defining the Kickstart file format, and it’s not the easiest thing to use. When creating a new Kickstart or making major changes, it’s good to have way to quickly iterate and test your changes. Iterating on a Kickstart file is slow and inconvenient in an enterprise environment in which a freshly booted server obtains an image and Kickstarter file via PXEboot. To speed up Kickstart testing and debugging, I’ve developed a simple way to use Kickstart files with CentOS/RedHat guest virtual machines that run in VirtualBox on my Mac. This process should also work with Windows or Linux hosts. When the RedHat installer is booting, it looks for a file called ks.cfg on a volume with label OEMDRV. First, we will create this volume as a small virtual disk image. On a Mac, you can create this image with Disk Utility as shown below. Note that the Name needs to be OEMDRV (though you can make the filename anything you want).

Create OEMDRV disk image OSX Disk Utility

General error: 2006 MySQL server has gone away

Tue, 10 Apr 2018 03:19:57 +0000

“MySQL Server has gone away” is a cryptic error that can be hard to troubleshoot (look at all the various responses on Stack Overflow!) Many problems can cause this error; I would like to document one specific case. In this example, the client is a PHP app using the Phalcon framework:

[Mon, 09 Apr 18 03:34:08 -0400][ERROR] SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /path/to/ModelBase.php:
Stack trace:
#0 [internal function]: PDOStatement->execute()
...
#17 {main}

This error can be tricky to troubleshoot. I looked in three logs:

Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

Sun, 01 Oct 2017 17:24:01 +0000

When you run yum upgrade on a CentOS/RedHat 7 instance, you will be upgraded to 7.4. If you have a FreeRADIUS server, you will be upgraded from version 2 to 3, and your server will likely stop authenticating! Good times! Fortunately, the solution was not complicated for us, because we had good documentation.

Symptoms

Your radius server suddenly starts denying logins.

Diagnosis

The configuration file /etc/raddb/radiusd.conf has changed enough between versions 2 and 3 that you cannot re-use the old version. To get radius debugging output, stop the radiusd service, and run (as root):

Managing persistent disks on Google Compute Engine

Tue, 19 Sep 2017 12:19:24 +0000

It can confusing when you have multiple persistent disks on an instance running on Google Compute Engine. For example, a server may have separate disks for the filesystem root, MySQL data, logs, and /tmp. Once you’ve created the Compute Engine disks and attached each one to the instance, how do you know which Compute Engine disk maps to each volume on the instance? First, describe the instance to see a list of the attached disks:

MySQL/MariaDB, logrotate, and SELinux

Tue, 17 Jan 2017 03:30:14 +0000

Assumption: You have SELinux Enforcing on your database server. If you’re still solving problems by permanently setting SELinux to Permissive, I don’t think you can really call yourself an IT professional. Here are the commands to set the SELinux context of all text MySQL log files to var_log_t so that they can be rotated by logrotate:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/.*.log(-[0-9]+(.gz)*)*)+"
restorecon -R -v -F /var/lib/mysql

NOTE: This expression will also match the file /var/lib/mysql/tc.log and change its context, which will prevent MySQL from starting. If your version of MySQL uses tc.log, here is one possibility:

Synology DiskStation DS1515+ Review

Sun, 18 Dec 2016 20:19:48 +0000

Summary: The Synology DS 1515+ is a capable little NAS with a large feature set, but it has some software reliability issues and limited technical support.

Hardware

The DS1515+ is a compact unit that feels sturdy. It holds five 2.5" or 3.5" drives, and 3.5" drives can be installed without any tools. You will need a Phillips-head screwdriver to install an additional RAM module, but that’s also a very simple process. The fans are also easily replaceable.

anacron run-parts generates invalid or malformed syslog messages

Sun, 06 Nov 2016 02:43:43 +0000

On RedHat and CentOS 6 and 7, anacron generates syslog messages that are mangled when they are forwarded by rsyslog. I found the cause and a solution in a comment by Tomas Heinrich on this old Fedora bug. Sadly, that bug was closed instead of getting fixed, probably because Fedora is oriented toward desktop users who are not generally forwarding syslog messages! Syslog uses the following default template for messages: "%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" The syslogtag field is truncated to 32 characters, but cronie-anacron includes a lot of data in its tag. The tag gets truncated to 32 characters, resulting in invalid syslog. If you try to send them to Logstash, you’ll get errors like this: {"message":"Nov 4 19:01:01 my-web-03 run-parts(/etc/cron.hourly)[1858 starting 0anacron","@version":"1","@timestamp":"2016-11-05T00:01:01.146Z","port":55456,"type":"syslog","tags":["_grokparsefailure"],"host_ip":"192.168.17.23"} Unfortunately, the problem is with the sending host, not Logstash, so you have to update the rsyslog config on every host that generates anacron messages. I prefer to do this in /etc/rsyslog.d to keep the configs more manageable. Add these two lines, which add a template that has no restriction on the length of the tag field, and activate the template when forwarding: $Template fwd,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%" $ActionForwardDefaultTemplate fwd I placed these lines in /etc/rsyslog.d/logstash.conf along with my other Logstash-specific rsyslog configs.

Making Windows work for Linux and OS X admins

Wed, 25 May 2016 04:10:17 +0000

If you are a Linux or OS X power user, then you’re used to having all the necessary tools built into your OS. When you log into a Windows system (What! No command line?) you may feel lost. These tools and shortcuts will help you be productive on Windows systems.

Windows Shortcuts

Alt-x is a magic shortcut key on Windows 8 and Server 2012. It pops up a little menu in the lower right corner of the screen which contains just the items that an admin needs. Try it!

Windows Programs

If you want Windows to work a little bit like Linux or OS X, you need to install a few of these free programs. Some are gratis, and some are libre.

Troubleshooting SSSD, realm, kerberos, and SSH

Thu, 10 Mar 2016 19:51:20 +0000

SSSD (System Security Services Daemon) allows Linux systems (specifically, Red Hat, CentOS, and Fedora) to verify identity and authenticate against remote resources. If you have a CentOS or Red Hat enterprise system, and you need to authenticate against a domain controller such as FreeIPA or Active Directory, SSSD is the way to go. I use SSSD on CentOS 7 systems, but it is now available on CentOS 6 as well. A few years ago, adclient (an open-source project from Centrify) was your only option to make a CentOS 6 server authenticate against Active Directory. adclient seems to have reached end of life, so SSSD is definitely the path forward. I won’t repeat the procedure for using Active Directory as an identity provider on a Red Hat 7 system. Instead, I want to provide a few troubleshooting tips, since limited information is available on SSSD and related tools.

LVM device-mapper: reload ioctl failed: Invalid argument

Fri, 19 Dec 2014 17:23:19 +0000

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection that LVM is relatively new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

Monitoring with SNMP, Part 3: Automate active monitoring with Nagios

Mon, 12 May 2014 12:29:08 +0000

My last post showed how to monitor networked devices with SNMP. You could try to remember to manually check the status of things periodically, but that would be missing the point of computers. Instead, automate your monitoring with Nagios, a web-based monitoring tool for Linux that automates the process of actively querying devices and doing something with the information. Nagios is available as free open source software (Nagios Core), and the company offers additional non-free products with premium features. The open-source version is fine for getting started and setting up basic monitoring. Nagios does a lot more than just SNMP monitoring. I’ll refer you to the Nagios Core documentation to get Nagios up and running, and I’ll focus on how to set up Nagios to actively monitor devices with SNMP.

Monitoring with SNMP, Part 2: Command-line tools for active SNMP

Fri, 02 May 2014 18:27:02 +0000

In Part 1, I summarized the basic concepts of SNMP and defined the terms and acronyms used in this post. Now, I will show how to use SNMP to monitor actual devices. As an example, I will monitor an enterprise-grade uninterruptible power supply (UPS) and power distribution unit (PDUs) from Tripp-Lite. These devices have an SNMPWEBCARD installed to support communication over Ethernet. Command-line tools for SNMP communication should be available for any Linux distribution (or any other UNIX-derived OS). Documentation for the basic SNMP tools is available online. The challenge with SNMP is figuring out what parameters are supported by a particular device. Most devices support a set of standard OIDs that return basic information such as device name, uptime, etc.

Monitoring with SNMP, Part 1: Fundamentals of SNMP

Wed, 05 Mar 2014 16:14:49 +0000

SNMP is a protocol for conveying information and controlling devices over a network. SNMP can be used in two ways:

Active: a device sends a command to set a parameter or request information for another device Passive: a device sends an alert (called a trap) to another device, which is configured to receive traps and do something with the information.

The “payload” of an SNMP message is called an Object Identifier, or OID. An OID is an ordered list of non-negative numbers, such as:

Running network experiments on the GENI project

Fri, 17 Jan 2014 20:50:04 +0000

The GENI Project is a networking testbed that is used by researchers studying novel networking technologies. While the technology is fascinating, the web site is, unfortunately, a confusing mess. Here are some pointers to get you started (or refresh your memory). This post will be updated as I learn more.

Key GENI Links

GENI Portal

This is where you log into the GENI Project. Your institution must have Shibboleth enabled and be part of the InCommon Federation. Click on the “Use GENI” button, enter the name of your institution into the search box, and you will be redirected to your institution’s login page.

Collaborative Git workflow: Shared Repository on a File Server

Fri, 10 Jan 2014 20:55:26 +0000

GitHub is a great tool for collaborating on projects. However, sometimes it is necessary to mimic the “GitHub workflow” using a shared repository on a local Linux server. The following example shows how I shared an example repository with multiple users. We are also using the Git flow model for branching, aided by the handy git flow plugin.

On my workstation

I started by creating a repo on my local workstation and setting it up to use the git flow plugin.

Linux configuration management roundup

Mon, 09 Dec 2013 19:52:20 +0000

Our high performance compute cluster (HPCC) has fairly primitive tools for managing the deployment of the operating system on the compute nodes. Our current tools are “aspencopy,” which takes an “image” of a the filesystem of a running server and saves it as a .tar.gz file (NOT a disk image). “aspenrestore” is its counterpart, which deploys an “image” to another server. The utility is smart enough to update things like the host name, IP address, host SSH keys, etc. However, the images are essentially “black boxes,” in the sense that there is no system for keeping track of which configuration changes have been applied to which image, and no way to know which image is running on each server. The next cluster that I am responsible for purchasing must include a configuration management/data center automation system, such as:

Configuring GRUB2 on Ubuntu to boot from another Linux partition

Mon, 02 Dec 2013 13:28:30 +0000

My recent Ubuntu installation was my first experience with the new GRUB 2.x series of bootloaders. Unforunately, the process of manually configuring GRUB2 on Ubuntu is not well documented in the case that everything doesn’t work “automagically.” I had to solve two problems: the blank screen at boot, and getting GRUB to boot to an existing partition with CentOS 5 installed.

Blank screen at boot

I got a blank screen after selecting an OS from the GRUB menu. GRUB seemed to work fine: it presented a menu of OS selections and allowed me to select one. Some text would then scroll by quickly, and then the monitor would then go blank and go into power-saving mode. This PC has an old NVIDIA card (I did not install any NVIDIA drivers). I found that I need to edit the file /etc/default/grub and set the nomodesetoption: GRUB_CMDLINE_LINUX="nomodeset" I then ran the commandupdate-grub to create the file /boot/grub/grub.cfg. You don’t want to edit this file by hand, since it gets over-written by configuration scripts.

Configuring XFCE4 and LightDM on a minimal Ubuntu installation

Mon, 25 Nov 2013 03:07:43 +0000

I recently installed Ubuntu on an older PC with 1GB of RAM and 80GB of hard drive space, so I wanted a lightweight desktop interface. I chose XFCE, since it is both lightweight and usable, and I have used it extensively. You can get Ubuntu pre-made with XFCE (xubuntu), but there are some disadvantages. The ISO is slightly too large to fit on a CD, and it comes with a lot of applications that I don’t need. Instead, I installed Ubuntu from a minimal CD and then used apt-get to install XFCE and the LightDM display manager. I learned that a couple of extra steps must be taken to get XFCE to play with lightdm. I got the error “can’t find session ubuntu”. Don’t bother changing the .dmrc file in your home directory; this file is overwritten every time lightdm starts! I configured lightdm to use an XFCE session with the command:

Preventing "soft" failures due to memory fragmentation in Linux

Wed, 20 Nov 2013 13:27:41 +0000

A previous post documented that a Linux server running a pre-2.6.24 kernel can fail to allocate large chunks of memory after its memory has been fragmented by a “thrashing” incident. In this post, I will point out some ways to prevent this problem.

Use a Newer Kernel

We have some servers running RHEL 5.9 with the kernel updated to 2.6.34.14. After a thrashing incident, these servers do not experience the same problem with allocating large blocks of memory. I think the fix is documented in the release notes for kernel 2.6.24. Section 2.4 talks about “anti-fragmentation patches” and includes a link to this article about Linux memory management, which links to this thorough documentation of the anti-fragmentation patches.(BTW, here is the full list of 2.6 kernel changelogs) My plan is to deploy RHEL 5.9 with the updated kernel to all the compute nodes in our cluster. However, this still doesn’t solve the problem of a user who requests some portion of the RAM on a node and then proceeds to consume more memory than requested. This is unfair to another user whose job is running on the same node.

Memory fragmentation degrades performance in Linux kernels < 2.6.24 (RHEL 5.4)

Fri, 15 Nov 2013 21:30:38 +0000

I have data from the STOKES High Performance Compute Cluster which definitively shows that kernel versions prior to 2.6.24 can suffer significant performance degradation due to memory fragmentation. I noticed the problem on servers running Red Hat Enterprise Linux (RHEL) 5.4 with kernel version 2.6.18-164.el5. This post will document my findings. The graphs were taken from our Ganglia monitoring system.

This node has 24GB of RAM. As long as processes do not request more than 23GB of RAM, the node operates normally. Processes can use 23GB of RAM all day long:

Insert an EPS file into an Asymptote Vector Graphics document

Wed, 23 Oct 2013 13:08:06 +0000

It’s surprisingly difficult to find out whether it is possible to include an image from an Encapsulated PostScript (EPS) file into an Asymptote vector graphics document. It turns out that it is easy, but difficult to find in the Asymptote docs (I finally found the answer, via Google, in the FAQ). It turns out that you use the label function to insert an image into an Asymptote document. Here is a snippet of code that I used to assemble a multi-part figure from several EPS documents:

How to fix your IEEE email alias so that it delivers all messages

Wed, 16 Oct 2013 18:18:11 +0000

I and others who use ___@ieee.org email addresses have a serious problem in which we occasionally fail to receive certain email messages that are sent our ieee.org addresses. This is a serious problem, and I’ve finally found a solution.

First, some background. Before the summer of 2013, the IEEE email address was simply an alias, which forwarded email to an actual email account. Some “spam” filtering was provided by IEEE, which could be modified or disabled through the IEEE web portal. In the summer of 2013, IEEE moved to Google Apps. Your ieee.org email now points to a Google Apps account. If you were previously using the alias service, the Google Apps account is configured to simply forward all email to the other account. However, the Google Apps account is also performing spam filtering on your email! When I finally logged in to my IEEE Google Apps account, I found several non-spam messages which had never been delivered because they were mistakenly sent to the spam folder in the Google Apps account (which I never check). Here is how to disable this dangerous and irritating “feature” that may be causing you to lose valuable email messages.

Using the MATLAB Distributed/Parallel Computing Toolbox without modifying your cluster

Wed, 02 Oct 2013 19:44:53 +0000

This post shows how I set up MATLAB to perform parallel calculations on a multi-core server which is part of an existing cluster that is managed using Torque and Moab. I did this without making any changes to the cluster submission system. The installation documentation for the MATLAB Parallel Computing Toolbox and Distributed Computing Server is poor. The instructions make a lot of assumptions that don’t fit into an existing “production” cluster environment. For example, they assume that a GUI is available to configure the cluster (this step is not required for my method!) Further, the instructions give the impression that MATLAB components must be “installed” on both the head node and compute nodes by an administrator. THIS IS FALSE! Ordinary users can run concurrent (multicore) MATLAB jobs without administrative privileges!

PHPUnit on CentOS 6

Wed, 28 Aug 2013 13:09:16 +0000

In my last post, I showed how to install PHPUnit with PHP 5.3 on a CentOS 5.9 system. Today, I will tackle the much simpler task of installing PHPUnit on a CentOS 6.4 system. The task is easier since PHP5.3 is standard on CentOS 6. On my system, PEAR was already installed. Steps: 1. Install the PHPUnit channel for PEAR:

yum install php-channel-phpunit

2. Set channel to auto-discover:

Installing PHPUnit on CentOS 5.9

Tue, 20 Aug 2013 02:13:21 +0000

I recently had to install PHPUnit on a CentOS 5.9 system. This turned out to be a little more complicated than I had anticipated, so I wrote down the procedure in case I had to do it again. Here are my notes: 1. Install php53 packages, not php (php packages install the old 4.x series) 2. Install PEAR, PHP’s component management system. Here is the complete Yum log for a PEAR installation: Jun 03 22:49:56 Installed: php53-common-5.3.3-13.el5_8.x86_64 Jun 03 22:49:57 Installed: php53-cli-5.3.3-13.el5_8.x86_64 Jun 03 22:49:58 Installed: php53-5.3.3-13.el5_8.x86_64 Jun 03 22:51:24 Installed: php53-devel-5.3.3-13.el5_8.x86_64 Jun 03 22:51:25 Installed: 1:php-pear-1.4.9-8.el5.noarch 3. Install php53-xml: Jun 03 23:30:40 Installed: php53-xml-5.3.3-13.el5_8.x86_64 4. Install php54-xml and restart httpd to get the DOM extension for PHP 5. Install PHPUnit with PEAR:

Very helpful PDF manipulation tools on Linux

Fri, 14 Jun 2013 21:25:15 +0000

You can do some really cool stuff with Ghostscript on the command line on a computer running Linux or OS X. However, it’s hard to remember all that stuff when you don’t use it very often. I recently learned about a set of command-line PDF manipulation tools called PDFjam. If you are a geek like me, you probably already have pdfLaTeX installed. I can’t summarize the capabilities any better than the description on the PDFjam site:

When root cannot run X applications from a local terminal

Sun, 02 Jun 2013 03:08:56 +0000

I run XWindows on a CentOS 5.9 box as a non-root user. To perform administrative tasks, I occasionally log in as root using “su” in a terminal window. However, I found that I was unable to start any GUI applications as root. I got the following error:

Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified
E233: cannot open display Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

This can happen when xhost isn’t configured to allow local, non-network connections. As the user who started the xwindows session (not root) run:

The cause of init: Id "co" respawning too fast, and how to fix it

Fri, 05 Apr 2013 20:26:03 +0000

I recently found the following message in the system logs for one of the compute nodes in the STOKES cluster:

init: Id "co" respawning too fast: disabled for 5 minutes

This caught my attention, because this OS image on this particular node should be identical to the image that is deployed on the rest of the nodes in the cluster. Why was it the only one producing this strange warning message? I searched the web and learned that the following line in /etc/inittab is causing the warning:

How to plot .xvg files from GROMACS on Windows

Tue, 12 Mar 2013 18:00:51 +0000

GROMACS produces graphical output in the form of .xvg files. These are designed to be viewed with a classic UNIX/Linux plotting program called Grace. If you happen to be using Linux and you have Grace installed, it is very easy to plot the data with the command

xmgrace my_file.xvg

If you aren’t using Linux, plotting .xvg files is quite a bit more difficult. Gnuplot is the only free plotting program that I have found that can handle .xvg files. Gnuplot is available for Windows (there is a direct download link near the top of the download page), but I will caution you that Gnuplot is not easy to learn. It has a command-line interface, and there are no point-and-click options to do basic operations such as labeling the axes of the plot. If you are using Gnuplot, you can plot the contents of an .xvg file with the command:

Installing Lumerical FDTD on a linux cluster

Mon, 25 Feb 2013 20:25:43 +0000

Most of the time, RPM (especially in conjunction with yum) is a decent package management solution. However, I can think of two common circumstances when you don’t want to let RPM install a package:

You don’t have root permissions on a system such as a shared cluster You are an administrator on a shared cluster and you can’t risk having a package over-write system-critical files

One approach is to extract the files from the RPM package and install them manually. I used this method to install FDTD Solutions from Lumerical on the STOKES Linux cluster. The application is distributed as an RPM package. I never install a third-party RPM as root, because a badly constructed package might over-write a system-critical file that some user is depending on. I downloaded the appropriate TAR file from Lumerical, uncompressed it, and looked at the contents. There is a simple script called install.sh which checks to make sure the user is root and then tries to install a hardware key driver and an RPM that contains the FDTD software. Here are my recommendations for installing Lumerical products on a cluster:

OpenMPI, Intel Compilers and RedHat 5: cannot find -lnuma

Wed, 20 Feb 2013 23:21:42 +0000

I found an interesting quirk when trying to build an OpenMPI application on a visualization node with a “stock” version of Red Hat Enterprise Linux 5.8. I used mpicc to compile the application and got the following error:

$ mpicc hello_world_mpi.c -o hello_world
/usr/bin/ld: cannot find -lnuma

This was rather surprising, since this node mounts a directory via NFS that contains OpenMPI and Intel Composer 2013, and these applications are known to work on other nodes. To find the source of the problem, I used the showme option to see the command that is actually run by mpicc:

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Wed, 16 Jan 2013 22:29:19 +0000

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

A list of system and application profiling tools

Mon, 07 Jan 2013 14:21:04 +0000

This article from Admin Magazine has a handy list of profiling tools.

Installing and configuring Infiniband on a Red Hat system

Wed, 12 Dec 2012 23:15:54 +0000

This post will take you through the installation and configuration of an Infiniband card on a server running Red Hat Enterprise Linux 5.4. These steps are applicable to any version of Red Hat 5, and will probably work with version 6 as well. It has been surprisingly hard to find all of these steps in one document.

Required packages

openib-1.4.1-6.el5.noarch libibverbs-1.1.3-2.el5.x86_64 libnes-0.9.0-2.el5.x86_64 libibumad-1.3.3-1.el5.x86_64 opensm-libs-3.3.3-2.el5.x86_64 swig-1.3.29-2.el5.x86_64 ibutils-libs-1.2-11.1.el5.x86_64 ibutils-1.2-11.1.el5.x86_64 (provides ibdiagnet and others) opensm-3.3.3-2.el5.x86_64 libibmad-1.3.3-1.el5.x86_64 infiniband-diags-1.5.3-1.el5.x86_64 (provides handy tools like ibstat and ibstatus) libibverbs-utils-1.1.3-2.el5.x86_64 (provides handy tools ibv_devinfo and ibv_devices) libibverbs-devel-1.1.3-2.el5.x86_64

The Infiniband troubleshooting quick reference

Tue, 04 Dec 2012 16:59:34 +0000

Glossary of Infiniband Terminology

GID: Global Identifier GUID: Global Unique Identifiers (also known as Direct Address) HCA: Host Channel Adapter LID: Local Identifier TCA: Target Channel Adapter SM: Subnet Manager

Infiniband: The Host Perspective

Every host on an Infiniband fabric has three identifiers: GUID, GID, and LID. A GUID is similar in concept to a MAC address because it consists of a 24-bit manufacturer’s prefix and a 40-bit device identifier (64 bits total). The Global Identifier (GID) is a 128-bit identifier similar to an IPv6 address (technically, a GID is a valid IPv6 identifier with restrictions). The GID consists of the 64-bit GUID plus an additional 64-bit EUI-64 identifier, for a total of 128 bits. The GID is used for routing between subnets. The default GID prefix is 0xfe80::0. Finally, there is the local identifier (LID), which is assigned by the subnet manager. The LID is a 16-bit identifier that is unique within a subnet. Hosts have an LID between 0 and 48,000, usually expressed in hexadecimal notation (such as 0xb1). Routing within a subnet is managed by LID. The GUID, GID, and LID for a Linux server are stored in text files. The exact path to this text file will depend upon which Infiniband driver is used on the system. For a system with the MLX4 driver (such as RedHat/CentOS 5.x), the commands are:

Managing an HPC cluster or cloud infrastructure: alternatives to xCAT

Sun, 04 Nov 2012 19:01:15 +0000

xCAT is the eXtreme Cloud Administration Toolkit from IBM. It’s a suite of tools that IBM has developed to manage large groups of servers, such as a cloud infrastructure or a high-performance computing cluster (HPCC). I have only used xCAT to administer a mid-sized compute cluster (about 140 compute nodes totaling about 1400 cores running RHEL 5). Overall, I have not found xCAT to be particularly effective for managing a mid-sized cluster. In many ways, xCAT is a brilliant piece of software, but like many “brilliant” solutions, it’s just too complex for its own good. There might be a cluster that is so large and complex that only a tool like xCAT can effectively manage it (especially if you have an administrative staff and you can pay someone to be a full-time xCAT guru). If you have a smaller cluster with limited administrative resources, you’re better off finding a simpler management solution. In contrast, I will briefly outline the administrative tools provided by Aspen Systems. We are in the process of expanding our IBM x1350 cluster with about 1000 compute cores from Aspen. Aspen has developed their own suite of cluster administration tools that take a very different approach. For a system such as ours (2400 cores total), the Aspen approach makes a lot more sense. The Aspen tools have also been used to manage much larger clusters for customers such as NOAA, NREL and NIST, so I’m not sure if there is a cluster that is “too big” for Aspen’s tools. I’ll list some of the key differences between the IBM and Aspen systems:

Mac OS X hack: download multiple files with curl

Mon, 29 Oct 2012 11:03:26 +0000

Updated 2017-08-01 to fix typo and use new demo URL. Also note that the Homebrew package manager has matured since this post was first published in 2012. Once you’ve configured Homebrew, installing a tool like wget is as simple as running brew install wget wget is an incredibly useful GNU tool on Linux. Unfortunately, it doesn’t come with OS X (as of Mountain Lion). OS X includes curl, which is a very handy tool but lacks at least one important feature of wget: the ability to use wildcards to get multiple files at the same time. For example, let’s say you want to download a subset of files from an FTP server. With wget, you could type:

An Improved Ebuild for OpenFOAM on Gentoo Linux

Mon, 22 Oct 2012 09:38:00 +0000

OpenFOAM is a notoriously difficult piece of software to compile, install and run. OpenCFD (the authors of OpenFOAM) have chosen to require the use of recent versions of gcc that are not available on most stable enterprise-class systems (ie Red Hat Enterprise Linux). To make things worse, until recently, OpenCFD also bundled a large number of libraries and helper applications (like VTK and ParaView) with the OpenFOAM source instead of using libraries and tools that are already on the system. Fortunately, OpenCFD has now moved the extra tools to a separate tarball, and the wizards at Gentoo have managed to create an ebuild for OpenFOAM. This is why I run Gentoo on my desktop workstation!

Firefox in CentOS/Fedora/Red Hat Enterprise Linux can only access Google

Mon, 10 Sep 2012 03:54:24 +0000

The Problem

I ran into what was probably the strangest error I have ever encountered as a Linux admin. I created a fresh CentoOS 6 virtual machine (VM) using VirtualBox on my OS X Mountain Lion system. To my great surprise, I was only able to access Google and a handful of other web sites from Firefox in the VM. I assumed that there was an issue with the networking of the VM, but after several frustrating hours of checking my setup I realized that the network in the VM was working fine. I was able to ping any web site by name on the command line, so clearly DNS was working. I could navigate to <<www.google.com&gt>; and perform searches, but clicking on most links resulted in the browser waiting forever. I could navigate to any site by IP, although the sites didn’t always load correctly (such as centos.org).

Hack of the Day: running CFD-ACE+ on Gentoo Linux

Tue, 24 Jul 2012 18:50:49 +0000

CFD-ACE+ is a multiphysics and computational fluid dynamics (CFD) simulation tool that was originally developed by CFD Research Corp. and is now distributed by ESI Software. The only platforms officially supported by CFD-ACE+ are Red Hat Enterprise Linux, SUSE Linux and Windows. Fortunately, it seems that ACE+ runs on other Linux distributions with only a little hacking. I just installed and tested CFD-ACE+ successfully (albeit not very thoroughly) on an up-to-date Gentoo Linux system. The process will require hacking some config files to build an external library from source. I started by downloading the files intended for Red Hat 6. Each package is a strange self-extracting executable file that unpacks itself when run. They also provide a C-shell script called install.com that automates the install process. The install script is very well behaved in that it doesn’t try to spew libraries and binaries throughout your filesystem, and you can run it as an ordinary user and install the file in your home directory. The tricky part about the installation is that the unpacking software (which is bundled into the self-extracting executable) needs the library selinux.so.1 in order to run. That library isn’t normally installed on a Gentoo system unless you’ve specifically set up SELinux. In order to avoid the potential problems of installing this library in my system paths, I created srcandincludedirectories in my home directory. I downloaded source tarballs for libsepol and libselinux from the download page of the selinux project. First, build libsepol usingmake. Then, copy the libsepol.so.1 binary from the src directory to the lib directory you just created in your home directory. Copy the include/sepol directory with all of its contents to the include directory in your home directory. When you are done, you should have:

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

Mon, 02 Jul 2012 17:14:54 +0000

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy). First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPICtells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the**-openmpflag to be consistent with the flags I used for building NumPy. The option--fcompiler=intelem** forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output. Once you have built SciPy successfully, install it with the following command:

How to dump/print/display/show UEFI configuration for IBM hardware

Thu, 07 Jun 2012 22:00:48 +0000

Newer enterprise-class hardware is starting to use UEFI instead of BIOS. UEFI is a big improvement, but there isn’t much documentation on the subject. Today I had to configure the UEFI for a new node in our cluster. I wanted to configure it exactly like an existing node that works. It was challenging just to find out how to display the configuration of the old node without rebooting the node and paging through dozens of configuration screens with the keyboard. Then I discovered IBM’s Advanced Settings Utility software, which fortunately was already installed on our cluster. It appears to be free for download, but with a restricted license. To display the complete UEFI configuration for a server, ssh to the server and run

Benchmarking BLAS Part 1: Building BLASbench with gcc

Fri, 25 May 2012 22:37:22 +0000

I recently installed ATLAS on my new workstation and I’ve been wondering what kind of performance gain I would get compared to the reference BLAS. After some searching, I finally found a BLAS benchmarking tool called BLASbench, which is part of the LLCbench suite. This tool has a rather strange build process, so follow the instructions on the web page. You have to create a file called sys.def which contains build instructions for your system. Here’s the sys.def file I used for my Gentoo system:

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

Tue, 24 Apr 2012 16:40:40 +0000

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL). This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5). First, copy the file site.cfg.exampletosite.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

Installing PETSc and libMesh on CentOS 5

Fri, 20 Jan 2012 20:06:18 +0000

Prerequisites: boost-devel, openmpi-devel, valgrind-devel I installed PETSc and libMesh in my user directory, since I have a single-user workstation.

Installing PETSc for use with libMesh

First, ensure that OpenMPI is installed and the system paths have been configured correctly. You will need to add the directory containing OpenMPI binaries to your $PATH, and the directory containing OpenMPI libraries to $LD_LIBRARY_PATH. CentOS does not do this automatically! Type mpicc on the command line to make sure the shell can find the binary. If mpicc and mpif77 run, then you should not need to use the comnand-line flags to the configure script for PETSc. If you are going to use libMesh with PETSc, you need to enable C++ with a command-line option to the configure script. Here is the process I used (it might be a good idea to set PETSC_ARCH and PETSC_DIR in your .bashrc so they are consistent for all builds).

Paraview 3.12 on 64-bit CentOS 5.7

Thu, 19 Jan 2012 19:23:15 +0000

I finally got around to installing Paraview on my CentOS 5 box. There are no official RPMS for CentOS 5, so I was expecting a painful build process. To my great surprise, I was able to download the Paraview binary for Linux (64-bit) from Kitware, unpack the archive, and run it! As root, I placed the entire ParaView-3.12.0 directory under /opt/Paraview. You could also keep it in your home directory if you are running a single-user system. I added /opt/Paraview/ParaView-3.12.0/bin to the $PATH in the .bashrc file in my home directory. Now I can type paraview at the command line and it runs! What if you want Paraview to appear in the Applications menu in your desktop environment? Most modern desktops (I use XFCE4) construct the Applications menu “on the fly” based upon the files in a standard directory (/usr/share/applications on CentOS). The Free Desktop Project has created a standard for desktop entry files. You may also find this summary of the standard to be helpful. To add Paraview to the menu, you simply need to create a new file in the standard location. If you installed Paraview in your user directory, you may want to place the desktop file in $HOME/.local/applications. Here are the contents of a file I called paraview.desktop:

Installing Sage 4.7 on CentOS 5

Mon, 07 Nov 2011 23:07:59 +0000

I recently upgraded my desktop workstation from an old 32-bit version of Gentoo to 64-bit CentOS 5. I downloaded and installed the latest version of Sage, and the process went smoothly. If you find this post helpful, please check out the Sage Beginner’s Guide at Packt Publishing. Since I don’t use Sage every day, I actually refer to my own book on a regular basis!

Download

Since CentOS is designed to be binary compatible with Red Hat Enterprise Linux, the correct binary to download is sage-4.7.2-linux-64bit-red_hat_enterprise_linux_server_release_5.6_tikanga-x86_64-Linux.tar.gz

Linux kernel 3.1 config for Gentoo guest on VMWare Fusion

Mon, 24 Oct 2011 03:23:48 +0000

The following kernel configuration was posted by Alessandro Di Marco as a comment on a previous post in which I provided a Linux kernel config for an earlier version of the kernel. In the interest of sharing his contribution, I’m placing it in its own post. I don’t know the author personally and I have not tried this config. Like any other free software, you are using it at your own risk and neither Alessandro nor I provide any warranty. Download the file from the link below, change the file extension from .txt. to .config and load the config into one of the standard kernel configuration tools (such as make menuconfig) and look over all the options before using it. Gentoo Kernel 3.1 Config

How to build ScipPy with Python 2.7.2 on CentOS5

Thu, 25 Aug 2011 00:17:57 +0000

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path. This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib). EDIT 2: you should use the –prefix=/home/yourusername instead of –user. The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages. The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up! I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5. I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system. Here is how I did it. First, with superuser privileges, use yum to install the following packages. You may need to set up epel as an alternate repository:

Replacing text in place with GNU sed

Mon, 11 Apr 2011 17:39:38 +0000

sed is a stream editor, which means that it accepts a stream of text, processes it, and spits out another stream of text. sed can process files that are too large to load into memory, and it is a completely command-line tool that can easily be integrated into shell scripts. This excellent sed tutorial was written for an old version of sed provided by Sun Microsystems, and it doesn’t cover one of the most useful features of GNU sed. GNU sed accepts a -icommand line argument that tells sed to replace the text file in place, rather than writing the output stream to another file. Another nice feature is thatsed -i will create a backup file before processing if you provide a backup suffix:

Building and linking to a shared Fortran library

Mon, 26 Oct 2009 22:34:02 +0000

I’m using GNU Fortran (gfortran) to build several shared libraries, and then dynamically linking to them from a Fortran program. The process is a little different than what I’m used to for C libraries, so I thought I’d explain it. Unlike C, there is no need to #include header files when compiling code that relies on functions defined in an external library. Likewise, there is no need to use -l or -L linking flags to tell the linker about s hared libraries (at least when they’re in the same directory). In fact, the whole process requires a lot less command-line options than I had expected.

X.org configuration for a Gentoo guest on VMWare Fusion

Thu, 22 Oct 2009 03:25:37 +0000

Here is an xorg.conf for a Gentoo guest running in VMWare Fusion on MacOS 10.6 (Snow Leopard). It’s surprisingly simple to set up X when the hardware is fake ;) Gentoo runs great on a Macbook Pro 13". I was concerned at first about the lack of 3D acceleration, but I tried a few 3D applications and found the performance to be acceptable. Blender ran really smoothly with a simple model (once I found out that command-click is equivalent to middle click). Rotating a field of several hundred spheres in Paraview was a little slow, but still very acceptable. I also bought a Logitech wireless mouse, and found that the middle button works just as you’d expect (paste) in Gentoo running in Fusion. I didn’t have to install the Logitech drivers in OS X.

Kernel configuration for a Gentoo guest in VMWare Fusion

Fri, 09 Oct 2009 22:48:23 +0000

I recently installed Gentoo Linux (amd64) as a guest on my Mac (OS 10.6 Snow Leopard) using VMWare Fusion. I thought I’d post the kernel config that I am using, since I didn’t find any out there that I trusted. If you can use this as a starting point, then it should save you some time and trouble. It’s a pretty minimal configuration–I think I removed all the extra drivers and stuff. You could lean it out a little more by removing the audio and a few other extras that I thought I might use. Let me know if you have any trouble with it. Gentoo amd64 kernel config for VMWare Fusion So far I am very pleased with its performance. My only disappointment is that VMWare doesn’t support Linux graphics hardware acceleration.

Profiling memory usage of Python code

Fri, 17 Apr 2009 22:08:52 +0000

In a previous post, I explained how to use the Python profiler. The profile is great for finding out which parts of the code run the slowest, or are called most often. However, the profiler doesn’t give any information about how much RAM is being consumed, or where it’s being consumed. If your program needs so much memory that it starts swapping to disk, its speed can be reduced by orders of magnitude. On the positive side, your code may run much faster if it fits entirely in the processor cache. In this post, I will introduce two tools that can help you understand the RAM usage of your Python code.

How to make two mice work with xwindows (x.org)

Sat, 11 Apr 2009 04:01:45 +0000

It’s a real pain to surf the Web when the batteries die in your wireless mouse or trackball. I use some old rechargeables that are no longer fit for digital camera service, so this happens to me fairly often. My batteries just died again, so I had to figure out how to configure X.org to use a backup mouse. It’s actually not hard to have an old 3-button mouse plugged into the PS/2 port and tell X.org to use it as a backup. First, define two input devices:

The Python configparser: a way to read simple data files

Tue, 03 Feb 2009 23:42:21 +0000

My simulation library, which is written mostly in Python, needs a lot of data and parameters in order to run. In some cases, I just hard-code the values in the script that calls the library, and in other cases I load a pickle file containing a Python objext. What if I want to read in data or configuration parameters from a human-editable text file? If the information is extensive or complex, XML might be a good choice, but XML is overkill for simple configuration or data files. Fortunately, a standard Python library called ConfigParser has already defined a configuration file format, and provides methods to interact with such files. Here is a sample of the format used in a config file:

Deploying Python applications on Windows

Thu, 15 Jan 2009 16:38:56 +0000

Writing applications in Python on a Linux system is almost too easy. Deploying Python apps on other Linux systems is not hard, because most Linux systems already have Python, with its core libraries and tools, installed. Most Linux systems also have package managers that make it easy to find and install required components. But, what happens when your co-workers who use Windows need to use your app? When you tell them to “go to the command line and…” you’ve pretty much lost them at “command line.” How do you package Python in a way that’s easy for a Windows user to install? Here is a process that worked for me:

Scipy.integrate ODEPACK import error solved!

Wed, 14 Jan 2009 16:46:16 +0000

I recently found a solution to a problem that had been vexing me for about a year. In order to successfully import anything from scipy.integrate, I had edit the file scipy/integrate/__init__.py and comment out the line

from odepack import *

If not, I would get various import errors such as

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python2.5/site-packages/scipy/integrate/__init__.py", line 10,
in <module>
 from odepack import *
 File "/usr/lib/python2.5/site-packages/scipy/integrate/odepack.py", line 7,
in <module>
 import _odepack
ImportError: /usr/lib/python2.5/site-packages/scipy/integrate/_odepack.so:
undefined symbol: daxpy_

I found the solution in Gentoo bug 251165. The problem only occurs when Scipy is built with non-reference versions of BLAS and CBLAS. The solution is to install the reference implementation of BLAS and CBLAS, rebuild Scipy, and then use whichever implementation of BLAS or CBLAS you want. I don’t know why this happens, and I don’t know if it affects distributions other than Gentoo. By the way, Gentoo has a really useful system tool called eselect, which has various modules that are used to choose between different versions or implementations of tools on your system (BLAS, Java virtual machine, kernel sources, OpenGL, etc.)

Tools for Python software development

Tue, 06 Jan 2009 18:52:45 +0000

I have found a few tools over the years that I find extremely useful for developing software. Python is my language of choice at the moment, but I’m sure these tools will be handy for any language.

Subversion is an open-source version-control system. Version control was designed to allow multiple programmers to work on the same project at the same time without stepping all over one another. However, even though I am a solo developer, I find version control to be extremely helpful.
1. When I commit changes to the repository, I can document what I’ve changed and why. This is a great help when I introduce a bug and have to go back and find it.
2. The repository is stored on a remote server that is backed up nightly.
3. It’s easy to make an “unstable” branch for implementing new features. When I make changes that don’t work, it’s easy to revert to a previous version that works.
4. It is easy to deploy my code to the Linux cluster and make sure that the cluster is running the latest version of my software.

Go read the documentation on the Subversion web site to find out what it can do for you.RapidSVN is a GUI client for a Subversion server. By default, Subversion comes with a command-line client that does everything you need. However, sometimes it’s easier to stay organized when everything is presented visually. Here is a screenshot of RapidSVN:

RapidSVN screenshot

Update 2: building 64-bit Numpy with Intel compilers and MKL

Tue, 09 Dec 2008 19:33:12 +0000

NOTE: these instructions are obsolete. Please see Building NumPy on a 64-bit Red Hat Cluster with Intel MKL. In a previous post I described how I built Numpy with Intel compilers and the Math Kernel Library on a 64-bit cluster. Today I upgraded to Numpy-1.2.1 and I made a few improvements to my install process. Please read the previous post, since I will not duplicate some important information, and then read on. This time, I made use of a site.cfg file. Copy the file site.cfg.exampletosite.cfg and edit. At the end of the file, uncomment the [mkl] section and set the path to your library. Mine looks like:

Using Python to generate XML files for visualization in Paraview

Thu, 13 Nov 2008 19:00:23 +0000

VTK is an open-source software system for “3D computer graphics, image processing, and visualization” developed by by Kitware. VTK is the foundation of Paraview, an industrial-strength CFD visualization tool that I have found to be very useful. I generate “second generation” XML-based files from my Python code and import them into Paraview for visualization. I am in the process of creating some Python classes to do, and I hope to publish them soon. Until then, I want to share some useful resources. The VTK file formats are specified in this document. It’s a pretty good specification, but it lacks some examples. Soon I will post an example of a valid unstructured, serial .vtu file. Each VTK file includes data from only one time step, so you have to keep track of time yourself (the filename is an easy solution). Paraview can read in data from multiple time steps, but you have to specify them in a .pvd file. This is also an XML file, with the following format: (reference)

Updated: building 64-bit Numpy with Intel compilers (icc)

Fri, 17 Oct 2008 19:36:38 +0000

I had to re-build Numpy because our cluster was upgraded and the Intel compilers and libraries were moved to a different directory. This turned out to be a half-day affair of trial-and-error. I learned a few important things, which I will try to list here:

*Delete the numpy-1.0.4/build directory after every build attempt. Doing “python setup.py clean” isnoteffective. I kept getting errors about undefined symbols when I tried to “import numpy” on the Python command line. It was looking for symbols in the old locations, even though I had just rebuilt the code using the new library locations. It turned out that I needed to delete the build directory in order to force a complete bottom-up rebuild.*The use of “setup.py” from distutils is not well documented online. The best thing to do is run “python setup.py –help-commands” to get a list of available commands. Then run “python setup.py <cmd> –help” to get help for that specific command. You can string commands together on the command line, as I will show in the example below.When you test the new numpy, make sure you arenotin the numpy-1.0.4 directory! If you are in the numpy source directory, when you import numpy, you will get the message “Running from numpy source directory.” and you will not be able to load any symbols from numpy. On 64-bit architectures, you need to compile position-independent library code. For some reason, distutils does not do this automatically, and the compilation will fail with an error similar to the following:

Installing numpy with the Intel Math Kernel Library (mkl)

Fri, 19 Sep 2008 21:24:58 +0000

Today I installed numpy on a cluster. Normally, as a Gentoo admin, I just install things with emerge, and all the details are taken care of automagically. However, this cluster runs Red Hat Enterprise, and I don’t have admin privileges, so I had to install numpy in my home directory. I installed 1.0.4, to match the version used on another system. You may not need to do this for more recent versions of numpy, which may have an improved setup script. The overall process is:

Server move completed and general update

Sun, 22 Jun 2008 19:37:44 +0000

The site is back online after a server move. Actually I can’t blame the server move for the downtime, because I went out of town and then had a lot of catching up to do, and as a result I didn’t switch the domain to point to the new server. I haven’t been blogging much because I haven’t done much software development, Linux admin, or lighting design lately. I have been busy in the lab at work and I’ve been supervising a trainee lighting operator instead of running my own shows. We haven’t been moving forward with plans to purchase a new lighting control console, so no update on that, either. Hopefully, I’ll have something interesting to post soon.

Connecting to a Windows file server from a Linux box

Wed, 23 Jan 2008 18:26:00 +0000

Updated 18 Jan 2013

Gentoo

There are two basic prerequisites for mounting a Windows share on a Gentoo Linux client. First, your kernel must be compiled with cifs support. Second, you must emerge the appropriate packages. The package used to be called mount-cifs, but this package is obsolete. You only need to install net-fs/samba, which will pull in net-fs/cifs-utils and a few other dependencies. The Samba page on the Gentoo Wiki is pretty good. Since you probably need a user name and password to log in to the Windows share, you should create a credentials file. You don’t want to specify your password on the command line because it will be saved in your .bash_history file and unauthorized users may be able to see it. I created a file called /etc/credentials.txt and set permissions so that only root can see it. The format is:

Found a bunch of templates

Tue, 22 Jan 2008 21:02:05 +0000

I use XFCE as my window manager, with Thunar as my file manager. Thunar has always had this “Create Document -> from template” entry in the File menu, but it didn’t come with any templates. I finally went looking for some templates, and found a good collection at stalefries. Just unpack the archive in a directory called “Templates” in your home directory. BTW, isn’t it about time I created a “linux” category instead of trying to stuff my Linux entries under the “software development” category?