On this page
Description
Every system design has a box labelled "database", and two questions decide most of what happens inside it: which kind of database fits the data, and how many copies of it you keep. This lesson is about both.
Picture an app on one PostgreSQL server. Reads have grown to 90% of the load and the CPU is pegged; worse, a disk failure last month meant restoring from a nightly backup - and losing a day of orders. Two problems, one tool: replication, keeping copies of the data on several machines. But copies bring problems of their own: which copy is right, and what does a user see when the copies disagree? Step by step:
- Choosing a data store from the way you read and write.
- Leader-follower replication.
- Synchronous versus asynchronous copies.
- Replication lag, and the odd things users see because of it.
- Failover, and the danger of two leaders.
- Which setup for which need.
Choosing a data store first
Before replication, the right store for the job. Start from the access patterns:
| Workload | Good fit | Why |
|---|---|---|
| Orders, payments, inventory - joins, constraints, multi-row transactions | Relational (PostgreSQL, MySQL) | ACID transactions, flexible queries |
| Sessions, carts, profiles read by ID at huge scale | Key-value (DynamoDB, Redis) | Fast lookups by key, easy to partition |
| Metrics, events, messages - heavy appends read by key and time range | Wide-column / time-series (Cassandra, a TSDB) | Writes spread across nodes; range scans within a partition |
| Self-contained records with varying fields | Document (MongoDB) | The record is one document |
| Full-text search | Search index (Elasticsearch) | Built alongside the source of truth, not instead of it |
| Photos, video, backups | Object store (S3) | Cheap, durable blobs; the database holds the metadata |
"NoSQL scales, SQL doesn't" isn't a reason. A relational database handles tens of thousands of transactions per second on one node and can be sharded; a key-value store gives up joins and (often) multi-item transactions to partition easily. Choose by what you read, how you write and which guarantees you need.
Leader-follower replication
One node is the leader: every write goes there. It records each change in a log and streams it to followers (replicas), which apply the changes in order.
- Reads can go to followers, so read capacity grows with each replica.
- Writes still all go to the leader - replicas don't scale writes; sharding does.
- If the leader dies, a follower is promoted to be the new leader.
Synchronous or asynchronous
| Synchronous | Asynchronous | |
|---|---|---|
| Write acknowledged after | A follower has the change too | Only the leader has it |
| Leader fails | No acknowledged write is lost | The last writes may be lost |
| Write latency | Higher - waits for a follower round trip | Lower |
| A follower is slow or down | Writes stall | Writes carry on; the follower lags |
A common middle ground is semi-synchronous: one follower synchronous (so every acknowledged write exists on two machines), the rest asynchronous.
Replication lag
Asynchronous followers run behind - usually by milliseconds, but by seconds or minutes when they're overloaded or catching up. That creates anomalies users notice:
- Reading your own writes. A user changes their display name, the page reloads from a follower that hasn't got the change yet, and the old name is back. Fix: read a user's own recent writes (say, for a minute after a change) from the leader, or remember the log position of their last write and read from a follower that has reached it.
- Monotonic reads. A user refreshes twice and hits two followers: a new comment appears, then disappears. Fix: pin each user's reads to one follower (for example by hashing the user ID).
Lag is also why "read from a replica" is wrong for anything a decision depends on - the stock count at checkout, a balance before a payment.
Failover
When the leader fails, something must notice (health checks with a timeout), choose the most up-to-date follower, promote it, and point writes at it. The dangers:
- Lost writes. With asynchronous replication, writes the old leader acknowledged but hadn't shipped are gone once a follower is promoted.
- Split brain. The old leader was only unreachable, not dead. It comes back still believing it's the leader and accepts writes - now two leaders diverge. Failover must fence the old leader (revoke its access, or reject writes carrying an old leadership epoch).
- Timeouts. Too short and a GC pause triggers an unnecessary failover; too long and writes are down for longer.
Read replicas for read scaling
For a read-heavy app (90% reads), adding read replicas is usually the first scaling step: cheap, no application sharding, and it doubles as failover capacity. Route queries that can tolerate slight staleness to replicas, and the rest (and all writes) to the leader.
Which setup?
| Need | Setup |
|---|---|
| More read capacity | Asynchronous read replicas |
| Never lose an acknowledged write | At least one synchronous (or quorum) follower |
| Survive a zone failure | A follower in another zone, automated failover with fencing |
| Survive a region failure | Cross-region replica (asynchronous: accept some loss) and a tested promotion runbook |
| More write capacity | Sharding (the next skill), not replicas |
| Reads that a decision depends on | The leader |
Going further. The deep dive (next tab) covers how replication is implemented (physical versus logical), leaderless quorums and their math, multi-leader conflicts and why last-write-wins loses data, consensus (Raft) and how it prevents split brain, RPO and RTO, and the storage engines underneath - B-trees versus LSM trees.
Check yourself
Try to answer before opening each one - recalling beats re-reading.
1. Why don't read replicas help a write-bound database?
Show answer
Every write still goes to the one leader - and every replica must apply every write too. Replicas multiply read capacity; to scale writes you partition (shard) the data.
2. A user posts a comment and immediately sees the page without it. What's likely happening, and one fix?
Show answer
The page read came from an asynchronous follower that hasn't received the write yet (replication lag). Read the user's own recent writes from the leader, or wait for a follower that has caught up to the write's log position.
3. After a failover the old leader rejoins and keeps accepting writes. What is this called, and how is it prevented?
Show answer
Split brain - two nodes both acting as leader, with diverging data. Failover fences the old leader: its writes are rejected (for example by a leadership epoch the storage checks) or its access is revoked before the new leader starts.
Key takeaways
- Choose the store by access pattern and guarantees, not by slogan.
- Leader-follower: writes to the leader, reads from followers, promote a follower on failure.
- Synchronous replication: no loss, slower writes, stalls on a stuck follower. Asynchronous: fast, can lose the last writes.
- Lag breaks read-your-writes and monotonic reads - route accordingly.
- Failover must fence the old leader to avoid split brain.
- Replicas scale reads; sharding scales writes.