Alert Policies for Log Metrics on Google Stackdriver Monitoring
Google Cloud Operations, formerly known as Stackdriver Logging and Monitoring, can be very confusing to set up. It’s easy to monitor something simple, but more complex cases quickly get confusing. One of the more flexible but confusing types of alert policies in Stackdriver Monitoring is a Logs-Based Metrics policy, which gives you the ability to monitor entries in Stackdriver Logging.
Create a Logs-Based Metric
The first step is to go to Stackdriver (Operations) Logging and create a user-defined metric. Go to the Logs Viewer and build a query to return the subset of log entries that you want to monitor (the details of query building are beyond the scope of this article). As of Sept. 2020, I highly recommend enabling the Preview Mode of the Logs Viewer interface, which is much better at guiding you through the process of building the query. If you must use the Classic interface, click the little black arrow at the right side of the filter box, near the top of the Logs Viewer screen. Select “Convert to advanced filter” from the drop-down menu, and create an advanced filter using the query language. Once you have created the right filter (query) that selects the log entries that you want to monitor, proceed to the next step:
Disk space utilization monitoring in Google StackDriver
Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).
File System Type Filter
Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.