General error: 2006 MySQL server has gone away

“MySQL Server has gone away” is a cryptic error that can be hard to troubleshoot (look at all the various responses on Stack Overflow!) Many problems can cause this error; I would like to document one specific case. In this example, the client is a PHP app using the Phalcon framework:

[Mon, 09 Apr 18 03:34:08 -0400][ERROR]  SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /path/to/ModelBase.php:
Stack trace:
#0 [internal function]: PDOStatement->execute()
...
#17 {main}

Continue reading General error: 2006 MySQL server has gone away

Advertisements

Viewing logs for a cluster of instances on Google Stackdriver Logging

StackDriver Logging is a great feature of Google Compute Engine (GCE). You pretty much need a centralized logging solution if you are taking maximum advantage of the features offered by GCE. For example, most production applications will run on a cluster of web servers. If you set up the cluster as a managed instance group on GCE, Google can auto-scale the size of the cluster based on traffic. The challenge is that it’s much  harder to troubleshoot errors across a cluster. The requests that caused the error could be spread across any number of servers, with randomly assigned names. If load drops and the server pool contracts, you will entirely lose any log data on a server that’s auto-deleted. StackDriver Logging is the answer to this problem. Configure all servers to send all logs to StackDriver, and you can view all of your web server logs in one interface, with the entries in chronological order.

View StackDriver Advanced Filter as a Gist on GitHub

Continue reading Viewing logs for a cluster of instances on Google Stackdriver Logging

Upgrade Ubiquiti UniFi Access Points (WAP) now to avoid KrackAttack

On October 15, 2017, security researcher Mathy Vanhoef announced the discovery of KrackAttacks, a serious flaw in the WPA2 encryption protocol that encrypts most WiFi connections. Using this method, an attacker can decrypt traffic from almost any wireless access point (WAP) and clients. Every WiFi access point will need to be upgraded with patch that prevents this attack.

Ubiquiti has already released a patch for UniFi access points that addresses this vulnerability. However, the upgrade process may not be straightforward, depending on the age of your access points. The following process will ensure that you are running a safe version of the firmware (3.9.3.7537 or later). Note that any upgrade will result in downtime for each access point as it is upgraded, so you will want to perform a rolling upgrade if you have multiple devices that will be upgraded while people are using them. Continue reading Upgrade Ubiquiti UniFi Access Points (WAP) now to avoid KrackAttack

Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

When you run yum upgrade on a CentOS/RedHat 7 instance, you will be upgraded to 7.4. If you have a FreeRADIUS server, you will be upgraded from version 2 to 3, and your server will likely stop authenticating! Good times! Fortunately, the solution was not complicated for us, because we had good documentation.

Symptoms

Your radius server suddenly starts denying logins.

Continue reading Warning: CentOS/RedHat 7.4 installs FreeRADIUS 3 with breaking changes

Managing persistent disks on Google Compute Engine

It can confusing when you have multiple persistent disks on an instance running on Google Compute Engine. For example, a server may have separate disks for the filesystem root, MySQL data, logs, and /tmp. Once you’ve created the Compute Engine disks and attached each one to the instance, how do you know which Compute Engine disk maps to each volume on the instance?

Continue reading Managing persistent disks on Google Compute Engine

Migration from GoDaddy to WordPress.com hosting

I’ve migrated my blog from GoDaddy to a paid plan with WordPress.com. I decided to trade flexibility for simplicity, since I don’t want to spend time fiddling around with the administration of this site. When I spend all day on IT, doing IT for a blog is no longer on my list of hobbies. With GoDaddy, you are responsible for all aspects of security, including updating WordPress, updating themes and plugins, and installing a security plugin to block the thousands of known attacks against WordPress. I have two major security concerns with GoDaddy. I had to allow insecure ciphers in order to connect to the shell on my site via SSH (or SCP). Why on earth haven’t they updated to a recent version of SSH?!? GoDaddy also requires you to purchase an SSL certificate from them to enable TLS on your domain. There’s no way to install a free certificate from Let’s Encrypt. I looked into enabling two-factor auth, but you have to manage it yourself with a plugin.

With a Premium plan from WordPress.com, SSL is enabled automatically with a certificate from Let’s Encrypt. I can use my domain name, and security is mostly handled for me. I can enable two-factor authentication. The downside is that I’ve lost some flexility; I can’t use my own theme or install my own plugins unless I upgrade to a Business plan. Also, I have to set up ads through Wordads.com to try to cover the cost of the site. I was fairly successful in covering the cost of hosting with Google AdSense; I’m not sure how effective WordAds will be.

Is there a “Primary Domain Controller” in Active Directory?

The Historical Answer

With Windows NT, prior to the advent of Active Directory, there was one Primary Domain Controller (DC) per domain, and every other DC was a Backup.

The Modern Answer

When Active Directory was introduced with Windows 2000, domain controllers became fully multi-master. There is no primary domain controller. However, there are two caveats that may confuse you.

Continue reading Is there a “Primary Domain Controller” in Active Directory?

MySQL/MariaDB, logrotate, and SELinux

Assumption: You have SELinux Enforcing on your database server. If you’re still solving problems by permanently setting SELinux to Permissive, I don’t think you can really call yourself an IT professional.

Here are the commands to set the SELinux context of all text MySQL log files to var_log_t so that they can be rotated by logrotate:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/.*.log(-[0-9]+(.gz)*)*)+"
restorecon -R -v -F /var/lib/mysql

NOTE: This expression will also match the file /var/lib/mysql/tc.log and change its context, which will prevent MySQL from starting. If your version of MySQL uses tc.log, here is one possibility:

semanage fcontext -a -t var_log_t "/var/lib/mysql(/db-server-01.log(-[0-9]+(.gz)*)*)+"

This expression is server-specific, because MySQL uses the hostname to name the log file.

Continue reading MySQL/MariaDB, logrotate, and SELinux

Synology DiskStation DS1515+ Review

Summary: The Synology DS 1515+ is a capable little NAS with a large feature set, but it has some software reliability issues and limited technical support.

Hardware

The DS1515+ is a compact unit that feels sturdy. It holds five 2.5″ or 3.5″ drives, and 3.5″ drives can be installed without any tools. You will need a Phillips-head screwdriver to install an additional RAM module, but that’s also a very simple process. The fans are also easily replaceable.

Configuration

You will need to connect the DiskStation to the Internet to install the operating system, which is called DiskStation Manager (DSM). The installation process went smoothly. From a security standpoint, this process is slightly scary, and I hope that the automated installation process used some method to verify the integrity of the operating system.

Once the operating system is installed, there is a nice graphical user interface to manage all aspects of the system. It is possible to enable SSH to allow access to the unit by command line, but the CLI is officially undocumented, and Synology support won’t answer questions about it. The documentation on the Synology support site is pretty thorough, so I won’t go through the process to create a RAID or volumes.

Continue reading Synology DiskStation DS1515+ Review

anacron run-parts generates invalid or malformed syslog messages

On RedHat and CentOS 6 and 7, anacron generates syslog messages that are mangled when they are forwarded by rsyslog. I found the cause and a solution in a comment by Tomas Heinrich on this old Fedora bug. Sadly, that bug was closed instead of getting fixed, probably because Fedora is oriented toward desktop users who are not generally forwarding syslog messages! Syslog uses the following default template for messages:

"%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"

The syslogtag field is truncated to 32 characters, but cronie-anacron includes a lot of data in its tag. The tag gets truncated to 32 characters, resulting in invalid syslog. If you try to send them to Logstash, you’ll get errors like this:

{"message":"Nov 4 19:01:01 my-web-03 run-parts(/etc/cron.hourly)[1858 starting 0anacron","@version":"1","@timestamp":"2016-11-05T00:01:01.146Z","port":55456,"type":"syslog","tags":["_grokparsefailure"],"host_ip":"192.168.17.23"}

Continue reading anacron run-parts generates invalid or malformed syslog messages