Setting up disk space utilization alerts in Google Stackdriver is almost unbelievably tricky. It’s easy to think that you can just choose the metric “Disk Utilization” and add a condition to alert when disk utilization crosses some threshold for some length of time (such as over 55% for 1 hour).
File System Type Filter
Your first problem is that a Linux system has many types of file systems, even if it only has one disk. You probably want to alert when your root disk is running out of space; therefore, you need to filter out non-disk filesystems like procfs or tmpfs. Add a device filter to select exactly which devices you want to alert on; in my case, with only one disk per server, I match only on device rootfs.
Disk Metrics State Filter
Metric “Disk Utilization” is actually a collection of states, only one of which you want to alert on. You can choose from states: free, reserved, or used. If you don’t select, it defaults to “free,” so your condition needs to trigger if “free” falls below a threshold. Otherwise, you need to add another filter to set “state=used” and set a condition to trigger if “used” goes above the threshold.
Final Configuration
The screenshot below shows how to set up an alerting trigger when disk used goes above a threshold:
