Google Cloud Operations, formerly known as Stackdriver Logging and Monitoring, can be very confusing to set up. It’s easy to monitor something simple, but more complex cases quickly get confusing. One of the more flexible but confusing types of alert policies in Stackdriver Monitoring is a Logs-Based Metrics policy, which gives you the ability to monitor entries in Stackdriver Logging.

Create a Logs-Based Metric

The first step is to go to Stackdriver (Operations) Logging and create a user-defined metric. Go to the Logs Viewer and build a query to return the subset of log entries that you want to monitor (the details of query building are beyond the scope of this article). As of Sept. 2020, I highly recommend enabling the Preview Mode of the Logs Viewer interface, which is much better at guiding you through the process of building the query. If you must use the Classic interface, click the little black arrow at the right side of the filter box, near the top of the Logs Viewer screen. Select “Convert to advanced filter” from the drop-down menu, and create an advanced filter using the query language. Once you have created the right filter (query) that selects the log entries that you want to monitor, proceed to the next step:

Salt is a great choice for a configuration management system. I have found that it scales well, both in terms of number of nodes, and complexity of configurations. However, I have found one aspect of Salt that remains under-developed: Salt doesn’t have great support for custom groupings of minions. Minion groups (which could also be called node groups, host groups, instance groups, etc.) allow you to target a predefined group of minions.

If you operate a Netgear switch in an environment that is subject to compliance requirements such as PCI or HIPAA, you are probably doing vulnerability scans, and the HTTP management interface of your switches will generate medium vulnerabilities (at least from Nessus):

SSL Self-Signed CertificateSSL Certificate Cannot Be Trusted* SSL Certificate Expiry

It IS possible to use signed certificates to secure the HTTPS interface on a Netgear switch, but the process to do so is insanely obscure. I do not know how someone had time to figure this out! Apparently, even Netgear tech support didn’t have any idea.

Ubiquiti publishes detailed data sheets for all of their products, but it’s hard to find all the information you need in one place for comparison purposes. That’s the purpose of this article.

First Generation Access Points

It can be confusing to distinguish between the generations of UniFi access points; you have to look at the access point label, or access it via the UniFi software to see the version. The performance difference between generations are significant. The datasheet for first generation Ubiquiti UAP wireless access points is still available.

Most business-class switches and routers support monitoring via SNMP. The main barrier to implementing SNMP monitoring is that so many parameters can be monitored, and it can be very hard to isolate the most important parameters. This example shows how to monitor the status of a specific port. Note that the term “port” includes link aggregation groups (LAGs) or other logical groups of ports like VLANs.

Use Case

A top-of-rack switch has several important link aggregation groups (LAGs). For example, one LAG has connections to two identical routers in a VRRP cluster. Other LAGs provide redundant network paths to independent NICs on the same server. I want to be alerted if the port status changes on any critical LAG. The ISP delivers Internet to a single port on the switch, which I also want to monitor.

openssl is the standard open-source, command-line tool for manipulating SSL/TLS certificates on Linux, MacOS, and other UNIX-like systems. I recently ran into an interesting problem using openssl to convert a private key obtained from GoDaddy. Someone else used GoDaddy’s “wizard” interface to generate a certificate signing request (CSR) and private key, and saved the files on their Windows workstation. They purchased an SSL cert from GoDaddy, and shared all the files with me for installation on servers. GoDaddy saved the private key in the newer PKCS #8 format (pkcs8), and one system required the key in the older PKCS #1 (pkcs1) format. It’s easy to tell the difference.

The latest Nginx rpm (nginx-1.16.1-1.el7) from EPEL for CentOS/Redhat is a fairly major update, from version 1.12 to 1.16, and includes some very important fixes to address multiple CVEs. There is another small but critical change in the latest RPM that isn’t mentioned in the bug report. RedHat had configured nginx to run as user “apache” in past releases, but in the EPEL release on 19 October 2019, the configuration was changed to run as user “nginx”. If you ran “yum upgrade” recently (which you really need to do on a regular basis) on any system with nginx installed, you probably have a system that is affected.

PERMISSION DENIED (Photo by Hidde van Esch on Unsplash)

If you recently updated your RedHat or CentOS 7.6 system, you may suddenly start getting “Permission Denied” errors when attempting to mount SMB shares via CIFS. Typical error messages in syslog look like this:

kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE
kernel: CIFS VFS: Send error in SessSetup = -13
kernel: CIFS VFS: cifs_mount failed w/return code = -13

Configuration that triggered the problem

Synology NAS with latest operating system shares a volume via SMBCentOS 7 Linux server mounts SMB share using a local username and password (NOT domain credentials)cifs-utils version 6.2 installed in May of 2019 (this by itself worked fine)libsmbclient just updated to 4.9.1-6libmount just updated to 2.23.2-61CIFS mount options: vers=3.0,credentials=/root/credentials.txt,sec=ntlmsspi* File /root/credentials.txt contained a username and password that are LOCAL to the SMB server

You probably have at least a few business-class switches and routers silently do their job, year after year. Network devices can be so reliable that we configure them once, and then forget that they exist. However, forgetting them is a really bad idea, since they require regular OS or firmware updates to patch vulnerabilities, and at some point, they will become unreliable and fail. If your device fails, do you have the configuration available to apply to its replacement? Are you sure the configuration that’s in your archive or documentation is up to date? This post deals with automating backups from Netgear switches and Ubiquiti EdgeRouters (and possibly switches, if they run EdgeOS). Please check out the accompanying repo at https://github.com/craigafinch/network-backup-scripts