Alerts, monitoring and finding the cause
Practical notes for people who look after servers: rules you can copy, commands to run when something breaks, and why the alarm so often goes off in the wrong place. Every snippet is tested before it's published.
Alerting that works
- Prometheus · rule library
Alert rules that won't burn you out
Copy-paste rules for servers, websites, certificates, logs and the monitoring itself, each built to stay quiet until someone needs to act.
- Alertmanager · Microsoft Teams
Prometheus alerts to Microsoft Teams after the connector shutdown
The old webhooks stopped in May 2026. Workflows, the msteamsv2 receiver, and an alert for when delivery breaks.
- Alertmanager
One outage, twenty alerts: taming Alertmanager grouping
group_by, timings and inhibit rules, and the one thing label grouping can't do.
- Prometheus · disks
A Prometheus disk space alert that doesn't cry wolf
Why "85% full" misfires on big and small disks, and three rules that replace it.
When something breaks
- nginx
nginx "upstream timed out (110: Connection timed out)": 502 vs 504
What each error line means, how to find the slow requests, and why the cause is usually behind your app.
- PostgreSQL
PostgreSQL "sorry, too many clients already": find the culprit
The query that shows who is holding the connections, how to get service back, and the alert that warns you first.
- Docker
Docker is filling my disk: logs, images and the build cache
Find which one it is, free the space safely, and cap it for good.
- Linux
"No space left on device" but df shows free space
Inodes, deleted files still open, the root reserve, or the wrong filesystem.
- Let's Encrypt · nginx
Certbot renewed the certificate, but nginx still serves the old one
The deploy hook, the trap when certificates are copied to several servers, and an alert for a server that didn't reload.