The 2 AM Postgres connection exhaustion

postgres / operations / 9 min read

It always starts the same way. A deploy goes out Friday afternoon. Nothing in the changelog touches the database. Saturday at 2 AM, the pager fires: request latency is up, health checks are flapping, and the application logs are a wall of one line:

FATAL: sorry, too many clients already

What actually happened

Postgres connections are expensive. Each backend is a forked OS process with its own memory context, file descriptors, and a slot in the shared connection table. The default max_connections is 100, and every deployment that "just works" in staging is silently one autoscaling event away from exceeding it.

The Friday deploy added a new background worker per app instance. Twelve instances times one extra connection each doesn't sound like much - until the connection pooler in front had already been running near its ceiling, and a traffic blip made every worker open its connection at once. Postgres hit the cap, started rejecting, the app retried, the retries each tried to open new connections, and the system entered a retry storm: each recovery attempt made the exhaustion worse.

The arithmetic nobody does

Say you have 20 app pods, each allowed 10 connections, plus cron jobs, plus one developer's open psql session. That is a nominal 212 connections against a 200-slot database. It fits - until a rolling restart causes pods to overlap, or a deploy process forgets to close its pool, and you are at 240. The failure is not gradual. It is a cliff, and the cliff is exactly at max_connections.

Worse, the damage happens before the cap. Postgres throughput degrades as connection count rises, because each backend competes for locks, WAL slots, and CPU. A database at 90% of its connection cap is often already slower than one at 30%, even with identical hardware and identical queries.

Why the obvious fixes don't fix it

The instinctive response is to raise max_connections. This buys you exactly one thing: a higher number on the cliff. It also makes every connection cheaper to exhaust the memory the server has for actual work, since each backend reserves memory whether it is active or idle.

The second instinct is to shrink the app-side pool. Now the app queues requests waiting for a connection, and you have traded "too many clients" errors for latency that quietly grows until timeouts fire upstream. The pool was never the root cause; the number of independent connection consumers was.

The unglamorous real fix

Real fixes look like this:

- One pooler (pgbouncer or similar) between everything and Postgres, in transaction pooling mode, with a hard ceiling well below max_connections.
- Every client - every deploy job, every cron, every notebook - accounted for in the total.
- Alerting on connection count as a percentage of the ceiling, not on errors after the cliff.
- Retry logic with jitter and circuit breaking, so a rejection does not become a storm.

None of this is hard. All of it is invisible until the night it isn't there.

Postscript

We spent years being the person holding the pager in these stories. It is a large part of why we built lastbase: a managed Postgres-compatible database where the connection topology, the ceilings, and the restarts are handled before they become your 2 AM. If that sounds like sleep you are missing, take a look.