Blog · 15 October 2026 · 7 minute read

Grafana Alloy vs node_exporter: what changes when the agent pushes

Tested with node_exporter 1.9.1, Grafana Alloy 1.20.1 and Prometheus 3.5.

We switched our own agent from node_exporter to Alloy this week, on seven servers that have faults staged on them around the clock. Here's what's the same, what isn't, and the three things that broke.

They aren't the same kind of tool

node_exporter is an exporter. It answers on port 9100, and Prometheus fetches its metrics every scrape interval. It does nothing else.

Alloy is a collector. Its prometheus.exporter.unix component runs node_exporter's collectors inside Alloy; Alloy scrapes them itself and pushes the samples to Prometheus with remote write. The same agent can read the journal and log files and push those to Loki, which is why Grafana has deprecated Promtail in favour of it.

Same metrics, mostly

Alloy's unix exporter is built on Grafana's fork of node_exporter, which branched in October 2023, before node_exporter 1.7. Metric names are the same, so dashboards and rules mostly carry over. The differences we found against 1.9.1:

• Missing series: node_cpu_online, node_filesystem_mount_info, node_os_support_end_timestamp_seconds, node_pressure_irq_stalled_seconds_total, and the watchdog collector.
• Missing labels: device_error on node_filesystem_* and rotational on node_disk_info.
• Defaults: the processes collector is on (the docs say off); systemd is off, and a systemd { } block alone doesn't turn it on: add it to enable_collectors.
• Labels: series arrive with job="integrations/unix" and an instance of Alloy's choosing. Set them yourself, or every rule written for job="node" goes quiet.

A minimal config that keeps the labels node_exporter dashboards expect:

prometheus.exporter.unix "node" {
  enable_collectors = ["systemd"]
}

discovery.relabel "node" {
  targets = prometheus.exporter.unix.node.targets
  rule {
    target_label = "job"
    replacement  = "node"
  }
  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }
}

prometheus.scrape "node" {
  targets         = discovery.relabel.node.output
  scrape_interval = "15s"
  forward_to      = [prometheus.remote_write.hub.receiver]
}

prometheus.remote_write "hub" {
  endpoint {
    url               = "https://prometheus.example.com/api/v1/write"
    bearer_token_file = "/etc/alloy/token"
  }
}

Prometheus only accepts pushed samples when started with --web.enable-remote-write-receiver. And run Alloy with --disable-reporting (in /etc/default/alloy): anonymous usage reporting to Grafana is on by default. The Debian package's own config collects metrics but sends them nowhere until you replace it.

Footprint

Both side by side on the same machine for five minutes, metrics only, 15-second scrapes:

Alloy's resident memory looks alarming in top, but most of it is pages of its own large binary, which the kernel can drop under pressure; the memory it actually uses is the "in use" row. The CPU figure includes start-up; on our seven 1 GB cloud servers, sending logs as well, each Alloy averages about 1% of a core over an hour. Disk is the bigger difference: about 550 MB installed, and while upgrading it needs room for the download as well.

What push changes

1. A dead server doesn't say it's dead

With node_exporter, Prometheus tries to scrape, fails, and records up == 0. With Alloy, up is produced on the server. A powered-off server sends nothing at all: its series simply stop, and up == 0 never fires. Alert on absence instead:

- alert: HostStoppedSending
  expr: |
    group by (instance) (present_over_time(up{job="node"}[1d]))
    unless
    group by (instance) (present_over_time(up{job="node"}[3m]))
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} sent metrics today but nothing for 3 minutes"

This is Grafana's recommended pattern, and it has a blind spot: a server that has been silent for a whole day drops out of the left-hand side and the alert resolves. The sturdier fix is a list of servers that should be sending, kept wherever you enrol them, and an alert on "listed, but no sample".

2. The server's clock is now the timestamp

Pulled samples are stamped by Prometheus. Pushed samples are stamped by the server. A server whose clock runs three minutes fast sends data that Prometheus treats as three minutes in the future: for three minutes, queries for "now" find nothing (our absence alert fired on a healthy server), and afterwards every change on that server appears three minutes later than it happened. If you work out which server changed first during an incident, that is exactly the wrong way round.

The usual skew check, node_time_seconds - timestamp(node_time_seconds), silently stops working too: both sides now come from the server's clock, so it always reads about zero. Alert on the kernel's own sync state instead, which works either way:

- alert: ClockNotSynchronised
  expr: node_timex_sync_status == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.instance }} clock is not synchronised (NTP)"

That catches a server whose time sync is off, not one that is synchronised to a wrong source. To measure the real offset, have each agent also scrape a small endpoint on the central server that returns its time, and compare.

3. Every server needs a credential

Pull needs a firewall rule; push needs a token and trust. Each server holds a bearer token (Alloy rereads bearer_token_file on every request, so you can rotate it without a restart) and must trust the receiver's certificate. If you use a private CA, note that setting ca_file replaces the system roots rather than adding to them, so moving the receiver to a public certificate later breaks every agent; we put the private CA and the system bundle in one file.

What you get back: no inbound port on any server, which is often what makes the change worth it.

Bonus: hub outages are buffered

Remote write keeps a write-ahead log on the server, so when Prometheus is down the samples wait and are sent afterwards. We restarted our hub's Prometheus mid-test and lost nothing. Two limits: Prometheus rejects samples much older than its newest ones unless you set out_of_order_time_window (it's off by default), and on a full disk Alloy 1.19 could truncate that log on replay; use 1.20.1 or later.

Which to choose

• node_exporter if Prometheus can reach every server, you only need metrics, and disk space on small servers matters.
• Alloy if servers can only connect out (another network, a customer's site, behind NAT), or you want logs from the same agent. Budget for the three changes above before you roll it out, not after the first false alarm.

Why we switched

Whyfire investigates alerts: when one fires, it compares every server's metrics and logs with their own normal and works out which server changed first. We moved its agent to Alloy so servers only ever connect out to the hub, with metrics and logs from one install. Then we added the three fixes: the hub keeps the list of enrolled servers and alerts on any that stop sending, every agent reads the hub's clock and a server more than 30 seconds out raises an alert, and the investigation won't let an off-clock server decide who changed first.