Disk space utilization monitoring in Google StackDriver
Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).
File System Type Filter
Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.
Disk Metrics State Filter
Metric “Disk Utilization” is actually a collection of states, only one of which you want to alert on. You can choose from states: free, reserved, or used. If you don’t select, it defaults to “free,” so your condition needs to trigger if “free” falls belowa threshold. Otherwise, you need to add another filter to set “state=used” and set a condition to trigger if “used” goesabove the threshold.
Final Configuration
The screenshot below shows how to set up an alerting trigger when disk used goes above a threshold:
Stackdriver Disk Space Alert Configuration
Stackdriver Disk Space Alert Configuration