Runbooks

A short "what to check first" for each alert. It goes on the incident card and page, and the AI write-up uses it.

HostDiskLow

starter runbook

Find what grew before deleting anything: du -xh --max-depth=2 <mountpoint> | sort -h | tail.

Old logs and package caches are usually safe to remove; databases and backups are not.
If it grows steadily, plan more disk rather than cleaning every week.

HostDown

starter runbook

Is the server itself up? Try to reach it (ping, SSH, the hypervisor or cloud console).

If it is up, the agent stopped: systemctl status node_exporter, then systemctl restart node_exporter.
If it is not, check the hypervisor or cloud console before rebooting, and look at Whyfire's point of origin: if another server changed first, the outage may have started there.

NoRulesLoaded

starter runbook

A rule file failed to load, so no alerts can fire. On the appliance: whyfire logs prometheus, then fix the file it names.

WhyfireAiFailing

starter runbook

Check ANTHROPIC_API_KEY in /opt/whyfire/appliance/.env (expired key, no credit) and sudo whyfire logs investigator. Cards still go out without the write-up.

WhyfireCardsFailing

starter runbook

Check NOTIFY_URL in /opt/whyfire/appliance/.env still exists (Teams workflow or Slack webhook), then sudo whyfire logs investigator.

WhyfireComponentDown

starter runbook

On the appliance: sudo whyfire status, then sudo whyfire logs <service> for the one that is down.