A Prometheus disk space alert that doesn't cry wolf
Tested with Prometheus 3.5 and node_exporter 1.9.1.
Don't alert on a percentage alone. Fire when free space is low as a percentage and in gigabytes, add a predict_linear rule that catches a disk filling fast, and alert separately on inodes.
"Disk over 85% full" is the first alert almost everyone writes, and one of the first everyone learns to ignore. Here is why it misfires, and three rules that replace it.
The problem with a percentage on its own
A single percentage treats every disk the same, and disks aren't the same:
- A 10 TB archive volume at 85% still has 1.5 TB free and may take months to fill. The alert fires anyway and stays red, so people learn to look past it.
- A 20 GB root disk at 80% has 4 GB left. A runaway log can eat that in an hour, and this alert hasn't fired yet.
The first case trains on-call to ignore disk alerts. The second means the alert they ignore is the one that mattered.
Rule 1: low as a share and in absolute terms
Fire only when free space is low as a percentage and small in bytes. A big disk at 85% stays quiet; a small one still alerts:
- alert: DiskSpaceLow
expr: |
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_size_bytes < 0.15)
and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} < 100 * 1024^3
and node_filesystem_readonly == 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} {{ $labels.mountpoint }} under 15% and under 100 GiB free"
The fstype filter skips memory-backed and container filesystems, and readonly == 0 skips read-only mounts such as snap packages, which always look 100% full. Tune the 100 GiB to your estate: it is the amount of free space below which you would want to act this week.
Rule 2: will it fill up soon?
The speed matters as much as the level. predict_linear fits a line through the last few hours and asks where it will be in the future. This fires when a disk is on course to be full within 24 hours:
- alert: DiskWillFillIn24h
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
and node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.4
and node_filesystem_readonly == 0
for: 1h
labels:
severity: critical
The < 0.4 stops a mostly empty disk from alerting just because a big file was copied onto it, and for: 1h waits for the trend to hold. This is the rule that wakes you for the runaway log, while Rule 1 handles the slow creep during working hours.
Rule 3: inodes
A disk can run out of inodes (the slots for files) with plenty of gigabytes free: millions of tiny session files, mail queue entries or cache files. Writes then fail with "No space left on device" and df -h looks fine. df -i shows the real problem (more in "No space left on device" but df shows free space):
- alert: DiskInodesLow
expr: |
node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_files < 0.05
and node_filesystem_readonly == 0
for: 10m
labels:
severity: warning
When it fires: find what grew
Before deleting anything, find what changed. Stay on one filesystem with -x so you don't wander into network mounts:
sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -20
Old logs and package caches are usually safe to remove; databases and backups are not. And check for space held by deleted files: a process can keep a deleted log open and its space isn't freed until the process restarts or reopens it. sudo lsof +L1 lists them.
Disk alerts are usually symptoms
A disk rarely fills by itself. Something started writing: a new debug log level, a stuck queue, a backup that failed and kept its temporary files. The disk alert tells you where the smoke is; the cause is whatever changed first.
That is the question Whyfire answers. When an alert fires, it compares every server's signals with the hour before, including log volume, and names the first one that changed, so the card can point at the log that started growing rather than just saying "disk 95%". Rule 1 ships with it as a starter, and the rest of our rule set is in Alert rules that won't burn you out.