2 servers went down. Why?
shop-web-01 and shop-web-02 stopped responding at 20:30 UTC. 2 alarms, which Whyfire grouped into one incident.
The trouble started 6 minutes earlier on shop-db-01, a server that never alarmed: open connections 2 → 302.
On-call starts at the database instead of the servers that alarmed. Last time, this alert was silenced against OPS-412 (“checkout app servers restarting under load; raised the app pool size”).
How it unfolded
- shop-db-01: open connections 2 → 302Origin
the database, 6 min before the alarm - shop-db-01: database connection limit hit, 60 a minute (none in the hour before)
the database, 5 min before the alarm · “FATAL: sorry, too many clients already” - shop-web-01 and shop-web-02: timeouts to another service, 32 a minute (none in the hour before)
app servers, 3 min before the alarm · “ERROR: database timeout after 30000ms in CheckoutService.placeOrder” - shop-lb-01: timeouts to another service, 61 a minute (none in the hour before)
the load balancer, 2 min before the alarm · “upstream timed out (110: Connection timed out) while reading response header from upstream shop-web-01, 502” - Alarms: shop-web-01 and shop-web-02 stopped respondingAlarm
2 alarms, grouped into one incident - Whyfire's report is ready
point of origin: shop-db-01
The investigation
Cause: shop-db-01 ran out of database connections at about 20:24 UTC and started refusing queries. The checkout app servers then timed out, the load balancer returned 502s, and both app servers stopped answering. The database is where this began; the app servers are only where it surfaced.
Evidence:
- shop-db-01: TCP connections 2 → 302 from 20:24, then "FATAL: sorry, too many clients already" and lock timeouts, 2 → 122 error lines a minute from 20:25
- shop-web-01 and shop-web-02: "database timeout after 30000ms in CheckoutService.placeOrder" from 20:27; shop-lb-01: upstream timeouts and 502s from 20:28
- both app servers stopped answering at about 20:29; HostDown fired for both at 20:30
Action: On shop-db-01, find what is holding the connections (pg_stat_activity), end it, and cap that client's connections. The app servers should recover once the database accepts connections again; restart them only if they don't.
Origin check: Agree with the pre-pass: shop-db-01's connections climbed 6 minutes before the alarm, ahead of every app-server and load-balancer symptom, on a server that never alarmed. This alert was silenced before against OPS-412 ("raised the app pool size"): that treated the symptom, not this cause.
Written by Claude from the evidence below for this demo. On an appliance with an API key, Whyfire runs this investigation automatically for every incident.
The team's checklist for this alert
Your team writes a short "what to check first" once for each alert, and Whyfire puts it on every card for that alert. This is the built-in one for "a server stopped responding". Its last line is why it matters here: it sends on-call to the point of origin, the database, instead of rebooting the app servers.
Is the server itself up? Try to reach it (ping, SSH, the hypervisor or cloud console).
- If it is up, the agent stopped:
systemctl status node_exporter, thensystemctl restart node_exporter. - If it is not, check the hypervisor or cloud console before rebooting, and look at Whyfire's point of origin: if another server changed first, the outage may have started there.
Happened before
- HostDown on demo-web-01, 2026-10-12 19:33 UTC
origin demo-db-01 (error-like log lines per minute)
Earlier incidents with the same alert, servers or origin, with what was found then.
All the evidence, earliest first
Every change Whyfire found on the servers in the 10 minutes around the alarm, compared with the hour before. More dots = a bigger change than normal.
| Server | What changed | Before → after | Started (UTC) | Strength |
|---|---|---|---|---|
| shop-db-01 database | open connections | 2 → 302 | 20:24 6 min before the alarm | ●●●●● |
| shop-db-01 database | errors in the logs | 2 → 122/min | 20:25 5 min before the alarm | ●●●●● |
| shop-db-01 database | database connection limit hit “FATAL: sorry, too many clients already” | 0 → 60/min | 20:25 5 min before the alarm | ●●●●● |
| shop-db-01 database | log volume | 4 → 124/min | 20:25 5 min before the alarm | ●●●●● |
| shop-web-01 app server, alarmed | timeouts to another service “ERROR: database timeout after 30000ms in CheckoutService.placeOrder” | 0 → 32/min | 20:27 3 min before the alarm | ●●●●● |
| shop-web-02 app server, alarmed | timeouts to another service “ERROR: database timeout after 30000ms in CheckoutService.placeOrder” | 0 → 32/min | 20:27 3 min before the alarm | ●●●●● |
| shop-web-01 app server, alarmed | errors in the logs | 2 → 34/min | 20:27 3 min before the alarm | ●●●●● |
| shop-web-02 app server, alarmed | errors in the logs | 2 → 34/min | 20:27 3 min before the alarm | ●●●●● |
| shop-lb-01 load balancer | timeouts to another service “upstream timed out (110: Connection timed out) while reading response header from upstream shop-web-01, 502” | 0 → 61/min | 20:28 2 min before the alarm | ●●●●● |
| shop-lb-01 load balancer | errors in the logs | 2 → 63/min | 20:28 2 min before the alarm | ●●●●● |
| shop-lb-01 load balancer | log volume | 32 → 93/min | 20:28 2 min before the alarm | ●●●●● |
| shop-web-01 app server, alarmed | CPU use | 1.2% → 60.3% | 20:29 1 min before the alarm | ●●●●● |
Lab note: these lab servers are containers sharing one machine, so CPU use moves on several of them at once. On real servers it doesn't.
Also in this demo: HostDown on demo-web-01.