Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).

File System Type Filter

Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.

Creating Kubernetes secrets isn’t intuitive the first time you do it. A common reason to use a secret is to add a SSL/TLS certificate to a cluster. Kubernetes provides two ways to add a secret: directly on the command line, and from a YAML source file. First, let’s generate a test certificate to work with and select our cluster.

Prerequisites

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com"

This command produces two files: tls.key and tls.cert. In production, you’d generate a key file and use it to obtain a certificate from a certificate authority.

StackDriver Logging is a great feature of Google Compute Engine (GCE). You pretty much need a centralized logging solution if you are taking maximum advantage of the features offered by GCE. For example, most production applications will run on a cluster of web servers. If you set up the cluster as a managed instance group on GCE, Google can auto-scale the size of the cluster based on traffic. The challenge is that it’s much  harder to troubleshoot errors across a cluster. The requests that caused the error could be spread across any number of servers, with randomly assigned names. If load drops and the server pool contracts, you will entirely lose any log data on a server that’s auto-deleted. StackDriver Logging is the answer to this problem. Configure all servers to send all logs to StackDriver, and you can view all of your web server logs in one interface, with the entries in chronological order. View StackDriver Advanced Filter as a Gist on GitHub The problem is that the Stackdriver docs only explain how to select an instance by the instance ID, which is a random-looking string that is not visible in the Console. Further, if you have a large cluster of disposable servers, you would have to look up the ID for every instance. If an instance was deleted when the cluster was auto-scaled, you won’t even know what to look up. Therefore, it’s much better to search for the logs based on a human-readable server name. For example, assume that your servers are Google Compute Engine instances that are named [region]-[zone]-web-[id]. Here’s a filter to match on that naming convention: [sourcecode language=“plain”] resource.type=“gce_instance” logName=“projects/my-project-name/logs/apache-error” labels.“compute.googleapis.com/resource_name”:"-web-" textPayload:“execution time” [/sourcecode]