Blog · 13 October 2026 · 12 minute read

Alert rules that won't burn you out

Tested with Prometheus 3.5, Alertmanager 0.28.1, node_exporter 1.9.1 and blackbox_exporter 0.27.

Burn-out rarely comes from big outages. It comes from the alert that fires every night for something nobody needs to fix, until people stop reading any of them. These are the rules we'd put on any Linux estate on day one. Copy the ones you want: each one stays quiet until a person genuinely needs to do something.

On this page: Six rules for the rules · Servers · Why no CPU alert · Websites and certificates · The monitoring itself · Logs · Routing

Six rules for the rules

1. Every alert needs an action. If the honest response is "huh, interesting", it belongs on a dashboard, not in a notification.
2. Wait before firing. A for: clause rides out restarts, deploys and single missed scrapes. Something that fixes itself in two minutes isn't an incident.
3. Two units for anything with a size. 15% of a 10 TB disk is 1.5 TB; 15% of a 20 GB disk is 3 GB. Alert when both the percentage and the absolute amount are low.
4. Severity decides who gets disturbed, and when. critical wakes someone, warning waits for working hours, info never notifies.
5. Events are not states. "A process was killed" happened; there's nothing to resolve. Send those without a "resolved" message.
6. Watch the watcher. A broken alert pipeline looks exactly like a quiet night.

Servers (node_exporter)

Change job="node" to your job's name. Each rule is a list item for the rules: section of a rule group.

Server not answering

- alert: HostDown
  expr: up{job="node"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} has not answered for 2 minutes"

Two minutes rides out a reboot for patching without paging anyone.

Disk space low

- alert: DiskSpaceLow
  expr: |
    (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
      / node_filesystem_size_bytes < 0.15)
    and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} < 100 * 1024^3
    and node_filesystem_readonly == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }}: under 15% and under 100 GiB free"

Low as a share and in gigabytes, so big disks don't nag and small ones still warn. Read-only and memory-backed filesystems are skipped. More in A disk space alert that doesn't cry wolf.

Disk filling fast

- alert: DiskWillFillIn24h
  expr: |
    predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
    and node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.4
    and node_filesystem_readonly == 0
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} is on course to fill within a day"

The rule that catches a runaway log at 2am. The 40% guard stops a nearly empty disk alerting because someone copied a big file onto it.

Out of inodes

- alert: DiskInodesLow
  expr: |
    node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}
      / node_filesystem_files < 0.05
    and node_filesystem_readonly == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} has under 5% of its inodes left"

Millions of tiny files can fill a disk that still has gigabytes free. See "No space left on device" but df shows free space.

Filesystem gone read-only

- alert: FilesystemReadOnly
  expr: node_filesystem_readonly{fstype=~"ext4|xfs|btrfs"} == 1
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} has gone read-only"

Linux remounts a filesystem read-only when it sees disk errors, and from then on every write fails, often without anyone noticing. Data filesystems are normally read-write; leave out any you mount read-only on purpose.

Memory really running out

- alert: MemoryLow
  expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.05
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} has had under 5% memory available for 15 minutes"

"Used" memory is high on every healthy Linux server, because the kernel uses spare memory as cache. MemAvailable already counts cache it can give back, so under 5% for 15 minutes is real pressure.

A process was killed for memory

- alert: OomKill
  expr: increase(node_vmstat_oom_kill[10m]) > 0
  labels:
    severity: warning
    kind: event
  annotations:
    summary: "The kernel killed a process on {{ $labels.instance }} to free memory"

An event, not a state: the kind: event label routes it to a receiver that sends no "resolved" message (see Routing). Needs kernel 4.13 or later.

Clock not synchronised

- alert: ClockNotSynchronised
  expr: node_timex_sync_status == 0 or abs(node_timex_offset_seconds) > 0.05
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }}'s clock is not synchronised"

TLS, Kerberos logins, database replication and every log timeline go wrong when clocks drift. The usual cause is an internal time server nobody watches.

A systemd service failed

- alert: SystemdUnitFailed
  expr: node_systemd_unit_state{state="failed"} == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.name }} has failed on {{ $labels.instance }}"

Needs node_exporter started with --collector.systemd, which is off by default. Catches the backup or sync job that failed at 3am and stayed failed.

A script's metrics went stale

- alert: TextfileMetricsStale
  expr: time() - node_textfile_mtime_seconds > 2 * 3600
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.file }} on {{ $labels.instance }} hasn't been updated for 2 hours"

Scripts that write .prom files for node_exporter's textfile collector (backup ages, queue sizes) stop without telling anyone, and their last values look fine forever. Set the threshold to twice how often the script runs. node_textfile_scrape_error == 1 catches a file it can't read.

Server restarted

- alert: HostRebooted
  expr: time() - node_boot_time_seconds < 600
  labels:
    severity: info
    kind: event
  annotations:
    summary: "{{ $labels.instance }} restarted in the last 10 minutes"

info: never sent, but there on the dashboard when you're asking "did it reboot?" in the middle of something else.

Not on the list: CPU and load

A server at 100% CPU that answers quickly is a server doing its job. High CPU and load are the first thing you look at when something else is wrong, so keep them on the dashboard and alert on what users feel: the site is down, requests are slow, a disk is about to fill.

Websites and certificates (blackbox_exporter)

Website down

- alert: WebsiteDown
  expr: probe_success{job="blackbox"} == 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} has failed its checks for 3 minutes"

Three minutes, because single failed probes happen on the healthiest sites. A login-protected page that answers 401 is up, so probe it with a module that accepts 401, or someone will eventually "fix" it:

modules:
  http_auth:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 401, 403]

Lots of sites failing at once

- alert: ProbesFailingEverywhere
  expr: avg by (job) (probe_success{job="blackbox"}) < 0.5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Over half of all probes are failing: check the prober and its network first"

When dozens of unrelated sites fail in the same minute, the problem is almost always the prober's own network. One alert saying so beats fifty WebsiteDown cards, and the inhibit rule in Routing mutes those while this fires.

Certificate expiring

- alert: CertificateExpiresSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 14 * 86400
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "The certificate for {{ $labels.instance }} expires in under 14 days"

Automatic renewal should have happened long before 14 days, so this means renewal (or the reload after it) is broken. Let's Encrypt certificates shrink to 64 days in February 2027 and 45 in February 2028: keep the threshold below your renewal window. If renewal worked but some servers still serve the old certificate, see Certbot renewed, nginx still serves the old certificate.

The monitoring itself

A whole job disappeared

- alert: NodeJobMissing
  expr: absent(up{job="node"})
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus has no node targets at all: check its configuration"

up == 0 only fires for targets Prometheus knows about. If a config change drops the whole job, nothing is "down", so nothing fires. Add one of these per job.

Alert rules quietly vanished

- alert: AlertRuleGroupsDisappeared
  expr: count(prometheus_rule_group_rules) < count(prometheus_rule_group_rules offset 1h)
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus has fewer alert rule groups than an hour ago"

A rule file can load without a single error and still contain nothing: one stray YAML mistake is enough, and every alert in it is silently gone. Keep this rule in a file of its own, and run promtool check rules before every reload. (It also fires for an hour after you delete rules on purpose.)

Config change didn't load

- alert: PrometheusConfigReloadFailed
  expr: prometheus_config_last_reload_successful == 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus rejected its last configuration change and is running the old one"

Notifications not getting through

- alert: AlertmanagerNotificationsFailing
  expr: sum by (integration) (rate(alertmanager_notifications_failed_total[10m])) > 0
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: "Alertmanager can't deliver {{ $labels.integration }} notifications"

Route this one somewhere other than your chat tool: if Teams delivery is what broke, an alert sent to Teams won't arrive. Details in Prometheus alerts to Microsoft Teams.

The watchdog

- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none
  annotations:
    summary: "Always firing: proves the alerting pipeline works end to end"

This alert always fires. Alertmanager forwards it every minute to an outside heartbeat service (healthchecks.io is one), which emails or texts you when the heartbeats stop. It's the only way to hear about Prometheus or Alertmanager being down, because neither can tell you itself.

Logs (Loki)

A server stopped sending logs

Log shipping fails silently: an agent stops, a rotated file leaves the shipper reading a deleted one, a permission changes. Count each server's log lines with a Loki recording rule (the ruler needs remote_write to Prometheus, and Prometheus needs --web.enable-remote-write-receiver):

groups:
  - name: log-volume
    interval: 1m
    rules:
      - record: host:log_lines:rate5m
        expr: sum by (host) (rate({host=~".+"}[5m]))

Then compare each server with the same time yesterday in Prometheus:

- alert: HostLogsSilent
  expr: |
    (sum by (host) (host:log_lines:rate5m offset 1d) > 0.1)
    unless on (host) (sum by (host) (host:log_lines:rate5m) > 0)
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.host }} sent logs yesterday but none for 30 minutes"

Use whatever label your logs carry for the server name. A server you've just retired will trigger it for a day.

Routing: warnings wait for the morning

This Alertmanager configuration puts the principles above into practice. Critical alerts go out at any hour (point that receiver at whatever wakes your on-call person), warnings and events wait until 08:00 on a working day, info never notifies, the watchdog goes to its heartbeat, and delivery failures go by email:

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'

route:
  receiver: team-chat
  group_by: ['alertname', 'site']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['alertname="Watchdog"']
      receiver: heartbeat
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m
    - matchers: ['alertname="AlertmanagerNotificationsFailing"']
      receiver: email
    - matchers: ['severity="info"']
      receiver: dashboard-only
    - matchers: ['kind="event"']
      receiver: team-chat-events
      mute_time_intervals: ['out-of-hours']
    - matchers: ['severity="warning"']
      receiver: team-chat
      mute_time_intervals: ['out-of-hours']

inhibit_rules:
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: ['alertname', 'instance']
  - source_matchers: ['alertname="ProbesFailingEverywhere"']
    target_matchers: ['alertname="WebsiteDown"']
    equal: ['job']

receivers:
  - name: team-chat
    msteamsv2_configs:
      - webhook_url_file: /etc/alertmanager/teams-url
  - name: team-chat-events
    msteamsv2_configs:
      - webhook_url_file: /etc/alertmanager/teams-url
        send_resolved: false
  - name: heartbeat
    webhook_configs:
      - url: 'https://hc-ping.com/your-check-id'
  - name: email
    email_configs:
      - to: 'oncall@example.com'
  - name: dashboard-only

time_intervals:
  - name: out-of-hours
    time_intervals:
      - weekdays: ['saturday', 'sunday']
        location: 'Europe/London'
      - times:
          - start_time: '00:00'
            end_time: '08:00'
          - start_time: '18:00'
            end_time: '24:00'
        location: 'Europe/London'

A warning that fires at 23:00 and is still firing at 08:00 goes out then; one that cleared itself overnight never disturbs anyone. Using Slack? Swap msteamsv2_configs for slack_configs; the rest stays the same.

Keep it that way

Rules drift back towards noise. Once a week, list what notified and ask of each: did someone act on it? If not, raise its threshold, lengthen its for:, demote it to info, or delete it. Muting a rule by changing its severity keeps it visible; commenting it out loses it for good.

What these rules can't tell you

Good rules tell you what is wrong and where it shows. They can't tell you which of the five alerts that just fired is the cause. That's what Whyfire adds on top of rules like these: when alerts fire, it compares every server's signals with the hour before and names the first thing that changed, even on a server that never alarmed. The live demo shows it on a staged outage.