One outage, twenty alerts: taming Alertmanager grouping
Tested with Alertmanager 0.28.1.
Group by alert name and failure domain (group_by: ['alertname', 'site']), give groups a short group_wait and a sensible repeat_interval, and add a few inhibit rules for known duplicates. What grouping can't do is link alerts that share a cause but no labels.
A switch fails, or a database fills up, and the channel lights up: a card per server, a warning and a critical for the same thing, and the same cards again an hour later. Most of that noise can be configured away. Some of it can't.
Why one fault becomes many alerts
Alert rules are written per symptom and evaluated per series. One fault produces many symptoms on many servers, and each one becomes its own alert:
- the same rule firing on every affected host,
- a warning and a critical version of the same rule,
- downstream symptoms: health checks failing, error rates up, queues backing up.
Alertmanager has three tools for this: grouping, timing and inhibition.
1. Group alerts that belong together
group_by decides which alerts share one notification. Grouping by alert name means "HostDown on twelve servers" arrives as one message instead of twelve:
route: receiver: team-chat group_by: ['alertname', 'site'] group_wait: 30s group_interval: 5m repeat_interval: 4h
Add a label that describes the failure domain, such as site, cluster or environment, so an outage in one data centre doesn't get merged with an unrelated one elsewhere. Avoid instance in group_by unless you really want a message per host.
2. Get the timings right
group_wait(30s above) is how long Alertmanager waits after the first alert in a new group before sending, so alerts that fire a few seconds apart arrive together.group_interval(5m) is the minimum gap before it sends an update to a group that has changed.repeat_interval(4h) is how often it re-sends a group that hasn't changed. Too short and people mute the channel; too long and a forgotten alert goes quiet for a day.
Also use for: on the rules themselves, so a blip of a few seconds never becomes an alert at all.
3. Inhibit the duplicates
Inhibit rules silence one alert while another is firing. The first rule here is the classic one: it drops a warning when its critical twin is already firing for the same thing. The second does the same for a known dependency: if every app alert carries a site label, a "site unreachable" alert mutes everything else from that site:
inhibit_rules:
- source_matchers:
- 'severity="critical"'
target_matchers:
- 'severity="warning"'
equal: ['alertname', 'instance']
- source_matchers:
- 'alertname="SiteUnreachable"'
target_matchers:
- 'severity=~"warning|critical"'
- 'alertname!="SiteUnreachable"'
equal: ['site']
Keep these few and obvious. An inhibit rule that hides the wrong alert is worse than noise, because nothing tells you it happened.
What grouping can't do
All three tools work on labels. They can merge alerts that share a label, and mute alerts whose relationship you wrote down in advance. They can't work out that two unrelated-looking alerts have the same cause.
Take a database that runs out of connections. The alarms go off on the app servers, because they are the ones being health-checked. The database never alarms. No label links "HostDown on web-01" to "connections climbing on db-01", so Alertmanager sends you the web alerts, neatly grouped, and leaves you to find the database yourself.
That step, from the alerts you received to the change that caused them, is what Whyfire does. It sits behind Alertmanager, and when alerts fire it compares every server's signals with the hour before and names the earliest change, even on a server that never alarmed. Alarms that share that cause become one incident with one card. The live demo shows it on exactly this database outage.
Sending these to Teams? The old connectors are gone: see Prometheus alerts to Microsoft Teams. And for rules that don't create the noise in the first place, Alert rules that won't burn you out.