Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

When you run yum upgrade on a CentOS/RedHat 7 instance, you will be upgraded to 7.4. If you have a FreeRADIUS server, you will be upgraded from version 2 to 3, and your server will likely stop authenticating! Good times! Fortunately, the solution was not complicated for us, because we had good documentation.


Your radius server suddenly starts denying logins.

Continue reading Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes


Managing persistent disks on Google Compute Engine

It can confusing when you have multiple persistent disks on an instance running on Google Compute Engine. For example, a server may have separate disks for the filesystem root, MySQL data, logs, and /tmp. Once you’ve created the Compute Engine disks and attached each one to the instance, how do you know which Compute Engine disk maps to each volume on the instance?

Continue reading Managing persistent disks on Google Compute Engine

MySQL/MariaDB, logrotate, and SELinux

Assumption: You have SELinux Enforcing on your database server. If you’re still solving problems by permanently setting SELinux to Permissive, I don’t think you can really call yourself an IT professional.

Here are the commands to set the SELinux context of all text MySQL log files to var_log_t so that they can be rotated by logrotate:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/.*.log(-[0-9]+(.gz)*)*)+"
restorecon -R -v -F /var/lib/mysql

NOTE: This expression will also match the file /var/lib/mysql/tc.log and change its context, which will prevent MySQL from starting. If your version of MySQL uses tc.log, here is one possibility:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/db-server-01.log(-[0-9]+(.gz)*)*)+"

This expression is server-specific, because MySQL uses the hostname to name the log file.

Continue reading MySQL/MariaDB, logrotate, and SELinux

Synology DiskStation DS1515+ Review

Summary: The Synology DS 1515+ is a capable little NAS with a large feature set, but it has some software reliability issues and limited technical support.


The DS1515+ is a compact unit that feels sturdy. It holds five 2.5″ or 3.5″ drives, and 3.5″ drives can be installed without any tools. You will need a Phillips-head screwdriver to install an additional RAM module, but that’s also a very simple process. The fans are also easily replaceable.


You will need to connect the DiskStation to the Internet to install the operating system, which is called DiskStation Manager (DSM). The installation process went smoothly. From a security standpoint, this process is slightly scary, and I hope that the automated installation process used some method to verify the integrity of the operating system.

Once the operating system is installed, there is a nice graphical user interface to manage all aspects of the system. It is possible to enable SSH to allow access to the unit by command line, but the CLI is officially undocumented, and Synology support won’t answer questions about it. The documentation on the Synology support site is pretty thorough, so I won’t go through the process to create a RAID or volumes.

Continue reading Synology DiskStation DS1515+ Review

anacron run-parts generates invalid or malformed syslog messages

On RedHat and CentOS 6 and 7, anacron generates syslog messages that are mangled when they are forwarded by rsyslog. I found the cause and a solution in a comment by Tomas Heinrich on this old Fedora bug. Sadly, that bug was closed instead of getting fixed, probably because Fedora is oriented toward desktop users who are not generally forwarding syslog messages! Syslog uses the following default template for messages:

"%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"

The syslogtag field is truncated to 32 characters, but cronie-anacron includes a lot of data in its tag. The tag gets truncated to 32 characters, resulting in invalid syslog. If you try to send them to Logstash, you’ll get errors like this:

{"message":"Nov 4 19:01:01 my-web-03 run-parts(/etc/cron.hourly)[1858 starting 0anacron","@version":"1","@timestamp":"2016-11-05T00:01:01.146Z","port":55456,"type":"syslog","tags":["_grokparsefailure"],"host_ip":""}

Continue reading anacron run-parts generates invalid or malformed syslog messages

Making Windows work for Linux and OS X admins

If you are a Linux or OS X power user, then you’re used to having all the necessary tools built into your OS. When you log into a Windows system (What! No command line?) you may feel lost. These tools and shortcuts will help you be productive on Windows systems.

Windows Shortcuts

  • Alt-x is a magic shortcut key on Windows 8 and Server 2012. It pops up a little menu in the lower right corner of the screen which contains just the items that an admin needs. Try it!

Continue reading Making Windows work for Linux and OS X admins

Troubleshooting SSSD, realm, kerberos, and SSH

SSSD (System Security Services Daemon) allows Linux systems (specifically, Red Hat, CentOS, and Fedora) to verify identity and authenticate against remote resources. If you have a CentOS or Red Hat enterprise system, and you need to authenticate against a domain controller such as FreeIPA or Active Directory, SSSD is the way to go. I use SSSD on CentOS 7 systems, but it is now available on CentOS 6 as well. A few years ago, adclient (an open-source project from Centrify) was your only option to make a CentOS 6 server authenticate against Active Directory. adclient seems to have reached end of life, so SSSD is definitely the path forward.

I won’t repeat the procedure for using Active Directory as an identity provider on a Red Hat 7 system. Instead, I want to provide a few troubleshooting tips, since limited information is available on SSSD and related tools.
Continue reading Troubleshooting SSSD, realm, kerberos, and SSH

LVM device-mapper: reload ioctl failed: Invalid argument

LVM2 (Logical Volume Management) is pretty amazing, but when something goes wrong, it’s not easy to troubleshoot. This is not the fault of the tools, but a reflection¬†that LVM is relatively¬†new in Linux, and not widely understood.

What I Tried to Do

I tried to increase the size of a logical volume with the lvextend command:

lvextend --extents 100%FREE /dev/VolumeGroup1/var

This form of the command is supposed to use all of the free space in the volume group.

What Failed

The command responded with an error message that contained this text:

device-mapper: reload ioctl failed: Invalid argument

Running lvdisplay showed the following status for the volume:

LV Status    suspended

Continue reading LVM device-mapper: reload ioctl failed: Invalid argument

Monitoring with SNMP, Part 3: Automate active monitoring with Nagios

My last post showed how to monitor networked devices with SNMP. You could try to remember to manually check the status of things periodically, but that would be missing the point of computers. Instead, automate your monitoring with Nagios, a web-based monitoring tool for Linux that automates the process of actively querying devices and doing something with the information. Nagios is available as free open source software (Nagios Core), and the company offers additional non-free products with premium features. The open-source version is fine for getting started and setting up basic monitoring. Nagios does a lot more than just SNMP monitoring. I’ll refer you to the Nagios Core documentation to get Nagios up and running, and I’ll focus on how to set up Nagios to actively monitor devices with SNMP.
Continue reading Monitoring with SNMP, Part 3: Automate active monitoring with Nagios

Monitoring with SNMP, Part 2: Command-line tools for active SNMP

In Part 1, I summarized the basic concepts of SNMP and defined the terms and acronyms used in this post. Now, I will show how to use SNMP to monitor actual devices. As an example, I will monitor an enterprise-grade uninterruptible power supply (UPS) and power distribution unit (PDUs) from Tripp-Lite. These devices have an SNMPWEBCARD installed to support communication over Ethernet.

Command-line tools for SNMP communication should be available for any Linux distribution (or any other UNIX-derived OS). Documentation for the basic SNMP tools is available online. The challenge with SNMP is figuring out what parameters are supported by a particular device. Most devices support a set of standard OIDs that return basic information such as device name, uptime, etc.
Continue reading Monitoring with SNMP, Part 2: Command-line tools for active SNMP