HostDown on 2 servers
shop-web-02 is not answering scrapes
What changed, earliest first
| # | Server | What changed | Before → after | Started | Strength |
|---|---|---|---|---|---|
| 1 | shop-db-01 not alarmed | TCP connections | 2 → 302 | 6 min before the alarm | z 24 |
| 2 | shop-db-01 not alarmed | error-like log lines per minute | 2/min → 122/min | 5 min before the alarm | z 96 |
| 3 | shop-db-01 not alarmed | database connection limit hit “FATAL: sorry, too many clients already” |
0/min → 60/min | 5 min before the alarm | z 60 |
| 4 | shop-db-01 not alarmed | log lines per minute | 4/min → 124/min | 5 min before the alarm | z 16 |
| 5 | shop-web-01 | timeouts to another service “ERROR: database timeout after 30000ms in CheckoutService.placeOrder” |
0/min → 32/min | 3 min before the alarm | z 32 |
| 6 | shop-web-02 | timeouts to another service “ERROR: database timeout after 30000ms in CheckoutService.placeOrder” |
0/min → 32/min | 3 min before the alarm | z 32 |
| 7 | shop-web-01 | error-like log lines per minute | 2/min → 34/min | 3 min before the alarm | z 25.6 |
| 8 | shop-web-02 | error-like log lines per minute | 2/min → 34/min | 3 min before the alarm | z 25.6 |
| 9 | shop-lb-01 not alarmed | timeouts to another service “upstream timed out (110: Connection timed out) while reading response header from upstream shop-web-01, 502” |
0/min → 61/min | 2 min before the alarm | z 61 |
| 10 | shop-lb-01 not alarmed | error-like log lines per minute | 2/min → 63/min | 2 min before the alarm | z 48.8 |
| 11 | shop-lb-01 not alarmed | log lines per minute | 32/min → 93/min | 2 min before the alarm | z 8.1 |
| 12 | shop-web-01 | CPU busy | 1.2% → 60.3% | 1 min before the alarm | z 15.7 |
Investigation
Cause: shop-db-01 ran out of database connections at about 20:24 UTC and started refusing queries. The checkout app servers then timed out, the load balancer returned 502s, and both app servers stopped answering. The database is where this began; the app servers are only where it surfaced.
Evidence:
- shop-db-01: TCP connections 2 → 302 from 20:24, then "FATAL: sorry, too many clients already" and lock timeouts, 2 → 122 error lines a minute from 20:25
- shop-web-01 and shop-web-02: "database timeout after 30000ms in CheckoutService.placeOrder" from 20:27; shop-lb-01: upstream timeouts and 502s from 20:28
- both app servers stopped answering at about 20:29; HostDown fired for both at 20:30
Action: On shop-db-01, find what is holding the connections (pg_stat_activity), end it, and cap that client's connections. The app servers should recover once the database accepts connections again; restart them only if they don't.
Origin check: Agree with the pre-pass: shop-db-01's connections climbed 6 minutes before the alarm, ahead of every app-server and load-balancer symptom, on a server that never alarmed. This alert was silenced before against OPS-412 ("raised the app pool size"): that treated the symptom, not this cause.
Written by Claude from the evidence above for this demo. On an appliance with an API key, Whyfire runs this investigation automatically for every incident.
Alerts in this incident
- HostDown on shop-web-02 20:30 UTC
- HostDown on shop-web-01 20:30 UTC
Runbook
Is the server itself up? Try to reach it (ping, SSH, the hypervisor or cloud console).
- If it is up, the agent stopped:
systemctl status node_exporter, thensystemctl restart node_exporter. - If it is not, check the hypervisor or cloud console before rebooting, and look at Whyfire's point of origin: if another server changed first, the outage may have started there.
Whyfire's starter runbook
Happened before
- 2026-10-12: HostDown on demo-web-01
origin demo-db-01 (error-like log lines per minute)
Silenced before
- 2026-10-12: silenced against OPS-412. "OPS-412 checkout app servers restarting under load; raised the app pool size"
Also started nearby
- None