Blog · 13 October 2026 · 6 minute read

nginx "upstream timed out (110: Connection timed out)": 502 vs 504

Tested with nginx 1.28 and PostgreSQL 16.

The full line in nginx's error log usually reads:

upstream timed out (110: Connection timed out) while reading response header from upstream

nginx itself is fine. It's telling you the server behind it stopped answering in time.

Read the error line: 502 or 504?

The end of the message tells you which stage failed, and that tells you where to look:

Find the slow requests

The default access log doesn't record how long the app took. Add it, so you can see which requests are slow and whether it's all of them or one endpoint:

log_format timed '$remote_addr [$time_local] "$request" $status '
                 'rt=$request_time urt=$upstream_response_time';
access_log /var/log/nginx/access.log timed;

Then list requests where the app took 10 seconds or more:

grep -E 'urt=[0-9]{2,}\.' /var/log/nginx/access.log | tail -20

One URL that's always slow is an application problem. Everything slow at once, starting at the same minute, means something shared got stuck.

Look behind the app

App servers have a fixed number of workers. When something they all depend on stalls, every worker ends up waiting on it, new requests queue behind them, and nginx times out on requests that have nothing to do with the original problem. In the minutes before the first 504, check:

• The database. Long-running queries and sessions waiting on locks. One blocked session can tie up every app worker. In PostgreSQL, this lists who is waiting on whom:

  SELECT pid, pg_blocking_pids(pid) AS blocked_by, wait_event_type,
         now() - query_start AS waiting_for, left(query, 60) AS query
  FROM pg_stat_activity
  WHERE cardinality(pg_blocking_pids(pid)) > 0
  ORDER BY query_start;

  If connections have run out entirely, see "sorry, too many clients already".
• The app's own pool. Its logs will often say "pool exhausted" or "timed out waiting for a connection" before nginx says anything.
• External APIs the app calls without a timeout of its own.

Raise the timeout only where it's expected

Some requests are legitimately slow: report exports, big uploads. Give those locations a longer timeout and leave the default everywhere else, so a stuck app still shows up quickly:

location /reports/ {
    proxy_pass http://app;
    proxy_read_timeout 300s;
}

The DNS trap

If proxy_pass or an upstream block uses a hostname, nginx looks it up once, when it starts or reloads, and keeps that address. If the backend moves, nginx carries on sending traffic to the old address until someone reloads it, and you get 502s or connect timeouts with a perfectly healthy backend. An automatic package update that restarts nginx at the wrong moment can cause the same thing. Since nginx 1.27.3 the open-source version can re-resolve names itself:

resolver 127.0.0.53 valid=30s;

upstream app {
    zone app 64k;
    server app.internal:8080 resolve;
}

Point resolver at your DNS server. On older versions, reload nginx after a backend's address changes.

Alert on it

If your nginx logs go to Loki, count the timeouts per server. Adjust the labels to match yours:

sum by (host) (count_over_time({job="nginx"} |= "upstream timed out" [5m])) > 20

That tells you nginx is timing out. It doesn't tell you why.

Where the 504s come from

The load balancer returning 504s is where the smoke shows; the fire is usually a server or two further back. Whyfire works that out when the alerts fire: it compares every server's signals with the hour before and names the earliest change. In our live demo, the load balancer's timeouts started at 20:28, but the database's connections had piled up from 20:24.