Whyfire

Blog · 13 October 2026 · 6 minute read

Certbot renewed the certificate, but nginx still serves the old one

Tested with nginx 1.28, OpenSSL 3 and blackbox_exporter 0.27.

Quick answer

Renewing only writes new files; nginx keeps the old certificate in memory until it reloads. Add a deploy hook that reloads nginx. If certificates are copied to other servers, every one of those has to reload too, and that's the part most setups miss.

Check what's served against what's on disk

Compare the certificate visitors get with the one certbot wrote:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -serial -enddate
sudo openssl x509 -noout -serial -enddate -in /etc/letsencrypt/live/example.com/cert.pem

Different serial numbers mean nginx is still serving the old certificate. A sudo nginx -s reload fixes it for now; the rest of this page stops it coming back.

One server: a deploy hook

Certbot runs every executable script in /etc/letsencrypt/renewal-hooks/deploy/ after each successful renewal. Create /etc/letsencrypt/renewal-hooks/deploy/reload-nginx:

#!/bin/sh
nginx -t && nginx -s reload

Make it executable, then test the whole chain:

sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx
sudo certbot renew --dry-run --run-deploy-hooks

The flag matters: a plain --dry-run skips deploy hooks, so a successful dry run tells you nothing about the reload. --run-deploy-hooks needs certbot 2.6 or later. Older guides call it --renew-hook, which is deprecated.

Several servers: the gap most setups have

Behind a load balancer, certificates are often renewed on one server and copied to the others: rsync, a shared directory, a configuration management run. The deploy hook reloads the server that renewed. The copy reloads nothing.

Nothing looks wrong for weeks, because the old certificate is still valid. Then it expires on the servers that never reloaded, and visitors see a certificate error some of the time, depending on which server the load balancer sends them to. It's an awkward one to diagnose, because the site works when you test it.

The dependable fix is for each server to check itself. This script reloads nginx when the certificate it serves differs from the one on disk; point it at wherever that server's copy lives:

#!/bin/sh
# Reload nginx when the certificate it serves differs from the one on disk.
name=example.com
disk=$(openssl x509 -noout -serial -in /etc/letsencrypt/live/$name/cert.pem)
live=$(echo | openssl s_client -connect 127.0.0.1:443 -servername $name 2>/dev/null | openssl x509 -noout -serial 2>/dev/null)
if [ -n "$live" ] && [ "$live" != "$disk" ]; then
    nginx -t && nginx -s reload
fi

Save it as /usr/local/sbin/reload-if-new-cert, make it executable, and run it every 15 minutes from /etc/cron.d/reload-if-new-cert:

*/15 * * * * root /usr/local/sbin/reload-if-new-cert

Alert when a server falls behind

Checking the site name only tests whichever server the load balancer picks. Probe each server directly instead, sending the site's name so it presents the right certificate. A blackbox_exporter module:

modules:
  tls_example:
    prober: tcp
    timeout: 10s
    tcp:
      tls: true
      tls_config:
        server_name: example.com

Probe every server's address, with a label saying which site they serve:

scrape_configs:
  - job_name: tls-servers
    metrics_path: /probe
    params:
      module: [tls_example]
    static_configs:
      - targets: ['10.0.0.11:443', '10.0.0.12:443', '10.0.0.13:443']
        labels:
          site: example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Then alert when the servers disagree about when the certificate expires:

- alert: CertificateDiffersBetweenServers
  expr: |
    max by (site) (probe_ssl_earliest_cert_expiry{job="tls-servers"})
      - min by (site) (probe_ssl_earliest_cert_expiry{job="tls-servers"}) > 86400
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Servers behind {{ $labels.site }} are serving different certificates"

A server that missed the reload still has the old certificate, which expires weeks before the new one, so this fires the day after a renewal, long before anyone sees an error. It doesn't depend on certificate lifetimes either, which matters because Let's Encrypt certificates shrink from 90 days to 64 in February 2027 and to 45 in February 2028: renewals, and chances to miss a reload, are about to get more frequent.

The reload you didn't know was missing

This kind of fault shows up as "some users get errors" with nothing obviously broken. Whyfire is built for that gap between where the alarm goes off and where the problem is: when alerts fire, it compares every server's signals with the hour before and names the one that changed first. See it on a staged outage in the live demo.