Certbot renewed the certificate, but nginx still serves the old one
Tested with nginx 1.28, OpenSSL 3 and blackbox_exporter 0.27.
Renewing only writes new files; nginx keeps the old certificate in memory until it reloads. Add a deploy hook that reloads nginx. If certificates are copied to other servers, every one of those has to reload too, and that's the part most setups miss.
Check what's served against what's on disk
Compare the certificate visitors get with the one certbot wrote:
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \ | openssl x509 -noout -serial -enddate sudo openssl x509 -noout -serial -enddate -in /etc/letsencrypt/live/example.com/cert.pem
Different serial numbers mean nginx is still serving the old certificate. A sudo nginx -s reload fixes it for now; the rest of this page stops it coming back.
One server: a deploy hook
Certbot runs every executable script in /etc/letsencrypt/renewal-hooks/deploy/ after each successful renewal. Create /etc/letsencrypt/renewal-hooks/deploy/reload-nginx:
#!/bin/sh nginx -t && nginx -s reload
Make it executable, then test the whole chain:
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx sudo certbot renew --dry-run --run-deploy-hooks
The flag matters: a plain --dry-run skips deploy hooks, so a successful dry run tells you nothing about the reload. --run-deploy-hooks needs certbot 2.6 or later. Older guides call it --renew-hook, which is deprecated.
Several servers: the gap most setups have
Behind a load balancer, certificates are often renewed on one server and copied to the others: rsync, a shared directory, a configuration management run. The deploy hook reloads the server that renewed. The copy reloads nothing.
Nothing looks wrong for weeks, because the old certificate is still valid. Then it expires on the servers that never reloaded, and visitors see a certificate error some of the time, depending on which server the load balancer sends them to. It's an awkward one to diagnose, because the site works when you test it.
The dependable fix is for each server to check itself. This script reloads nginx when the certificate it serves differs from the one on disk; point it at wherever that server's copy lives:
#!/bin/sh
# Reload nginx when the certificate it serves differs from the one on disk.
name=example.com
disk=$(openssl x509 -noout -serial -in /etc/letsencrypt/live/$name/cert.pem)
live=$(echo | openssl s_client -connect 127.0.0.1:443 -servername $name 2>/dev/null | openssl x509 -noout -serial 2>/dev/null)
if [ -n "$live" ] && [ "$live" != "$disk" ]; then
nginx -t && nginx -s reload
fi
Save it as /usr/local/sbin/reload-if-new-cert, make it executable, and run it every 15 minutes from /etc/cron.d/reload-if-new-cert:
*/15 * * * * root /usr/local/sbin/reload-if-new-cert
Alert when a server falls behind
Checking the site name only tests whichever server the load balancer picks. Probe each server directly instead, sending the site's name so it presents the right certificate. A blackbox_exporter module:
modules:
tls_example:
prober: tcp
timeout: 10s
tcp:
tls: true
tls_config:
server_name: example.com
Probe every server's address, with a label saying which site they serve:
scrape_configs:
- job_name: tls-servers
metrics_path: /probe
params:
module: [tls_example]
static_configs:
- targets: ['10.0.0.11:443', '10.0.0.12:443', '10.0.0.13:443']
labels:
site: example.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115
Then alert when the servers disagree about when the certificate expires:
- alert: CertificateDiffersBetweenServers
expr: |
max by (site) (probe_ssl_earliest_cert_expiry{job="tls-servers"})
- min by (site) (probe_ssl_earliest_cert_expiry{job="tls-servers"}) > 86400
for: 1h
labels:
severity: warning
annotations:
summary: "Servers behind {{ $labels.site }} are serving different certificates"
A server that missed the reload still has the old certificate, which expires weeks before the new one, so this fires the day after a renewal, long before anyone sees an error. It doesn't depend on certificate lifetimes either, which matters because Let's Encrypt certificates shrink from 90 days to 64 in February 2027 and to 45 in February 2028: renewals, and chances to miss a reload, are about to get more frequent.
The reload you didn't know was missing
This kind of fault shows up as "some users get errors" with nothing obviously broken. Whyfire is built for that gap between where the alarm goes off and where the problem is: when alerts fire, it compares every server's signals with the hour before and names the one that changed first. See it on a staged outage in the live demo.