When the replica fell behind: a replication lag outage
The architecture diagram was beautiful. A primary, a streaming replica in another zone, and a load balancer sending all read traffic to the replica. It survived the design review, the game day, and six months of production. Then a migration script ran a multi-hour batch update, and everything broke.
The failure, step by step
Streaming replication in Postgres ships changes over the WAL as they happen. Under normal write load the replica applies changes milliseconds behind. Under a bulk write - a backfill, a mass UPDATE, a giant COPY - the WAL is generated far faster than a single replay process can apply it. Lag grows from milliseconds to minutes to hours.
Nobody notices while writes go to the primary. But every read now serves stale data. The first symptom is usually not a database error at all: a customer calls support asking why their record is gone, because they updated it, the write went to the primary, and the read came from a replica that is 40 minutes behind.
The second symptom is worse. A monitoring job notices the replica is unusable and fails it over - to a node that is missing hours of data. Applications reading the new primary see yesterday's world. Some of them write decisions based on it. You now have a consistency incident layered on top of a lag incident.
Why lag is invisible until it isn't
Replication lag is a metric that spends its life at effectively zero. Zero, zero, zero, then a bulk job or a network hiccup, and it is a cliff - same shape as connection exhaustion, same shape as most Postgres operational pain. The danger of metrics that live at zero is that humans stop looking at them, and the alert threshold gets set generously because "it sometimes spikes and it's fine."
The other trap is that the obvious mitigation - pausing writes until the replica catches up - is exactly what turns lag into a full outage. The queue backs up, the app's write timeouts fire, and the retries generate more WAL when the pause lifts, which is another lag spike. Systems under load do not fail independently; they fail in cascades, and each mitigation becomes a new failure mode.
What sane operation looks like
- Lag is monitored as a hard SLO, with reads routed to the primary the moment it exceeds a few seconds.
- Bulk jobs are chunked and rate-limited so WAL generation never outruns replay.
- Failover automation checks replay position before promotion, and refuses to promote a stale node.
- The read-replica pattern is treated as an optimization, not a correctness mechanism. Anything that must be fresh
reads from the primary.
Each of those is a paragraph. Living them is a full-time job, which is the actual lesson.
Postscript
Running Postgres well means running a distributed system, whether you meant to or not. lastbase is our attempt to hand you the good parts - a Postgres-compatible endpoint that is simply there, and correct, without you owning the replication topology. Details here.