criticalinc_20261012_8bc5d7e7 · 2026-10-12 19:33 UTC

HostDown on demo-web-01

demo-web-01 is not answering scrapes

Point of origin

demo-db-01: error-like log lines per minute up

2/min → 240/min, started 5 min before the alarm · on a different server from the one that alarmed

What changed, earliest first

#	Server	What changed	Before → after	Started	Strength
1	demo-db-01 not alarmed	error-like log lines per minute	2/min → 240/min	5 min before the alarm	z 190.4
2	demo-db-01 not alarmed	database connection limit hit “ERROR: connection pool exhausted, 128 clients waiting”	0/min → 119/min	5 min before the alarm	z 119
3	demo-db-01 not alarmed	timeouts to another service “ERROR: query timed out after 30000ms”	0/min → 119/min	5 min before the alarm	z 119
4	demo-db-01 not alarmed	log lines per minute	17/min → 269/min	5 min before the alarm	z 33.6
5	demo-web-01	CPU busy	3.7% → 36.9%	2 min before the alarm	z 8.8
6	demo-web-01	host reachable (up)	1 → 0	2 min before the alarm	z -4

Investigation

Cause: demo-db-01's database ran out of connections at about 19:27 UTC and started failing queries; demo-web-01 stopped answering five minutes later. The database is where this incident began.

Evidence:

demo-db-01 error-like log lines rose from 2 to 240 per minute from 19:27, all "connection pool exhausted, 128 clients waiting" (postgres) and "query timed out after 30000ms" (app)
demo-web-01's CPU rose from 3.7% to 36.9% at 19:30, then its agent stopped answering at 19:30; HostDown fired at 19:32
no other server or signal moved in the hour before

Action: Check demo-db-01 first: what is holding the 128 connections (long queries, a stuck job), and free them. Then bring demo-web-01's agent back and confirm the web tier recovers.

Origin check: Agree with the pre-pass: demo-db-01's errors started 5 minutes before the alarm, on a different server from the one that alarmed.

Written by Claude from the evidence above for this demo. On an appliance with an API key, Whyfire runs this investigation automatically for every incident.

Alerts in this incident

HostDown on demo-web-01 19:32 UTC

Runbook

Is the server itself up? Try to reach it (ping, SSH, the hypervisor or cloud console).

If it is up, the agent stopped: systemctl status node_exporter, then systemctl restart node_exporter.
If it is not, check the hypervisor or cloud console before rebooting, and look at Whyfire's point of origin: if another server changed first, the outage may have started there.

Whyfire's starter runbook

Happened before

Nothing like this in the last 90 days

Silenced before

Never silenced in the last 6 months

Also started nearby

None