These are the notes for System Design episode 11. Last episode, indexes made our database thousands of times faster. But it's still one machine, on one disk. Here that disk dies in the middle of lunch rush, and you invent the fix: keep a live copy. A primary takes the writes, a replica replays the primary's change log, and replicas can serve reads. Then the trouble starts: the copy is always a little behind (replication lag), so we invent read-your-writes. We measure asynchronous vs synchronous replication (fast but can lose writes, or safe but slower), switch to the copy when the primary dies (failover), and meet the scariest bug in this topic: split brain, two databases that both think they're the boss. The fix is a majority vote, a term number and fencing. Every demo is plain Python (some with the built-in sqlite3) and runs as-is; outputs are real (Python 3.13, SQLite 3.53). Timings vary a little from run to run.
The six problems
| Problem | What it teaches |
|---|---|
| The only database dies | Single point of failure (unable to open database file) |
| Keep a live copy | Primary, replica, change log (PostgreSQL's WAL), read replicas |
| The copy is behind | Replication lag, stale reads |
| Asha sees her old address | Read-your-writes: route a user's own reads to the primary |
| When is a write saved? | Asynchronous vs synchronous replication, measured |
| Switch to the copy, safely | Failover, split brain, majority, term number, fencing |
Problem 1: the only database dies (single point of failure)
It's 1 pm at ChaiNow. Lunch rush. Orders arrive every second, and every one of them is stored in one database, on one machine. Then the disk dies. No bug, no mistake: disks are machines, and machines break. With enough of them, one breaks somewhere every day.
Now every new order fails, and every old order is gone too. When one part breaking takes down the whole system, that part is called a single point of failure.
Demo 1: the database disappears
A real SQLite database file, orders.db. We save one order, then "kill the disk" by deleting the file, and the app tries to open the database again. file:orders.db?mode=rw with uri=True opens the file in read-write mode: it must already exist (plain sqlite3.connect("orders.db") would quietly create a new, empty file instead, which hides the problem).
import sqlite3, os
db = sqlite3.connect("orders.db")
db.execute("CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT)")
db.execute("INSERT INTO orders (item) VALUES ('masala chai')")
db.commit()
print("orders saved:", db.execute("SELECT COUNT(*) FROM orders").fetchone()[0])
db.close()
os.remove("orders.db") # 1 pm: the disk dies
print("1 pm: the database disk died")
db = sqlite3.connect("file:orders.db?mode=rw", uri=True)
print(db.execute("SELECT * FROM orders").fetchall())
Output:
orders saved: 1
1 pm: the database disk died
Traceback (most recent call last):
File "main.py", line 11, in <module>
db = sqlite3.connect("file:orders.db?mode=rw", uri=True)
sqlite3.OperationalError: unable to open database file
One order saved, then sqlite3.OperationalError: unable to open database file. The only copy is gone: every order, every customer, every bill.
Why not just take a nightly backup? A backup is a copy saved at one moment, like a photo. Last night's backup still loses every order from this morning, and restoring it takes time while the shop is closed. Backups are still important (see the misconceptions below), but to stay open we need a copy that is live: updated all day, ready to take over.
Problem 2: keep a live copy (primary, replica, change log)
An old Indian shop solved this with paper:
| In a shop | In a database |
|---|---|
| The main register: the owner writes every sale here | Primary: the one database that accepts writes |
| The day book: every sale, in order, line by line | Change log: an ordered list of everything that changed |
| The accountant's copy: he copies the day book into his own register | Replica: a copy that replays the log, in the same order |
The key idea: replay the same changes, in the same order, and you get the same data. Copying changes like this is called replication. To replay means to apply each logged change again, one by one, on the copy.
Demo 2: a primary, a replica and a log
Two in-memory SQLite databases (":memory:" means the database lives in RAM and disappears when closed) and a Python list as the change log.
import sqlite3
primary = sqlite3.connect(":memory:")
replica = sqlite3.connect(":memory:")
log = [] # the change log
def write(sql, args=()):
primary.execute(sql, args) # 1. change the primary
log.append((sql, args)) # 2. note it in the log
def replicate():
while log: # replay, in the same order
replica.execute(*log.pop(0))
write("CREATE TABLE orders (id INTEGER PRIMARY KEY, item TEXT)")
write("INSERT INTO orders (item) VALUES (?)", ("masala chai",))
write("INSERT INTO orders (item) VALUES (?)", ("samosa",))
replicate()
print("primary:", primary.execute("SELECT * FROM orders").fetchall())
print("replica:", replica.execute("SELECT * FROM orders").fetchall())
primary.close() # 1 pm: the primary dies
print("after crash, replica:", replica.execute("SELECT * FROM orders").fetchall())
Output:
primary: [(1, 'masala chai'), (2, 'samosa')]
replica: [(1, 'masala chai'), (2, 'samosa')]
after crash, replica: [(1, 'masala chai'), (2, 'samosa')]
Primary and replica hold the same two orders, and after the primary dies, the replica still has both. Nothing lost.
This is how real databases do it. PostgreSQL's change log is called the write-ahead log (WAL). PostgreSQL calls its replicas standby servers, "ready to take over operations if the primary server fails". Its docs describe streaming replication like this: "The standby connects to the primary, which streams WAL records to the standby as they're generated."
writes WAL records (the change log)
App ─────────────▶ Primary ──────────────────────────────▶ Standby / replica
(writes) (replays the log)
Read replicas: the free bonus
Copies give us a bonus. Writes still go to one place, the primary, which ships its changes to replica 1 and replica 2. But reads can go to any replica. People open the ChaiNow menu far more often than they order, so spreading reads over replicas takes most of the load off the primary. A replica used this way is called a read replica. PostgreSQL's docs: "A standby server can also be used for read-only queries."
┌──────────▶ Replica 1 (reads)
App ──▶ Primary (all writes)
└──────────▶ Replica 2 (reads)
Replicas are read-only. If the app wrote to a replica, that change would never reach the primary, and the copies would drift apart. So replicas refuse writes (see the first error in the mistakes table). Rule of thumb: writes go to the primary, reads can go to any replica.
Problem 3: the copy is behind (replication lag)
Asha moves from Pune to Bengaluru. She opens ChaiNow and saves her new address: a write, so it goes to the primary. The app reloads her profile: a read, so it goes to a replica. And the screen shows Old flat, Pune. Asha thinks her change wasn't saved, so she types it again, or worse, her biryani is delivered to Pune.
Demo 3: reading back from a replica that's behind
A background thread (a second line of work running at the same time as the main program) ships the log to the replica every half second. That half second stands in for network time plus the time to apply the change. daemon=True makes it a daemon thread: Python doesn't wait for it when the main program ends. check_same_thread=False lets both threads use the same SQLite connection (without it you get the ProgrammingError in the mistakes table).
import sqlite3, threading, time
primary = sqlite3.connect(":memory:", check_same_thread=False)
replica = sqlite3.connect(":memory:", check_same_thread=False)
for db in (primary, replica):
db.execute("CREATE TABLE users (id INTEGER, address TEXT)")
db.execute("INSERT INTO users VALUES (1, 'Old flat, Pune')")
log = []
def ship_log(): # runs in the background
while True:
time.sleep(0.5) # network + apply time = lag
while log:
replica.execute(*log.pop(0))
threading.Thread(target=ship_log, daemon=True).start()
change = ("UPDATE users SET address = ? WHERE id = 1", ("New flat, Bengaluru",))
primary.execute(*change); log.append(change) # saved on the primary
print("right away:", replica.execute("SELECT address FROM users").fetchone()[0])
time.sleep(1)
print("1 s later :", replica.execute("SELECT address FROM users").fetchone()[0])
Output:
right away: Old flat, Pune
1 s later : New flat, Bengaluru
Right away: the old address. One second later: the new one. The replica wasn't wrong, just late. That gap is called replication lag, and a read that returns old data because of it is a stale read. PostgreSQL's docs say "there is a small delay between committing a transaction in the primary and the changes becoming visible in the standby", "typically under one second". (To commit means to make a change final: "saved".) When the network or the replica is busy, the lag grows. No error appears anywhere, which is what makes this bug sneaky.
Problem 4: read your own writes
The fix comes from the bank:
| At the bank | In the app |
|---|---|
| You deposit cash; the clerk updates the main ledger | Your write goes to the primary |
| To check your balance now, ask that same clerk, not a branch that syncs tonight | Read from the primary for a few seconds after you write |
| Everyone else can wait for the evening sync | Others read replicas: a little lag is fine for them |
For a few seconds after a user writes, that user's reads go to the primary. This guarantee is called read-your-writes: you always see your own changes. Ravi doesn't care if he sees Asha's address a second late, so most reads stay cheap.
Demo 4: route Asha to the primary, Ravi to a replica
Plain dictionaries stand in for the two databases, so we can focus on the routing.
import time
last_write = {} # user -> time of their last write
def save(user, primary):
primary[user] = "New flat, Bengaluru"
last_write[user] = time.time()
def pick_db(user, primary, replica):
if time.time() - last_write.get(user, 0) < 5: # wrote in the last 5 s?
return "primary", primary # read your own writes
return "replica", replica # everyone else: the replica
primary = {"asha": "Old flat, Pune", "ravi": "MG Road"}
replica = dict(primary) # not caught up yet
save("asha", primary)
for user in ["asha", "ravi"]:
name, db = pick_db(user, primary, replica)
print(f"{user:5} reads from the {name}: {db[user]}")
Output:
asha reads from the primary: New flat, Bengaluru
ravi reads from the replica: MG Road
Asha sees her new flat; Ravi reads from the (still behind) replica and gets MG Road, which is correct for him. The 5 seconds is a choice: pick a window longer than your normal lag. Only the person who just wrote pays the cost of reading from the primary. In a real app, last_write would live somewhere every app server can see (for example, a timestamp in the user's session), not in one process's memory.
Problem 5: when is a write "saved"? (sync vs async)
When the app tells Asha "order placed", is the order really safe? You can send a parcel two ways:
| Sending a parcel | Replicating a write |
|---|---|
| Drop it in the post box and walk away | Asynchronous: the primary says OK at once and ships the change to the replica a little later. Fast, but not yet safe |
| Wait at the counter for a signed receipt | Synchronous: the primary says OK only after a replica confirms it has the change. Safe, but you wait |
| A birthday card vs a property deed | A trade-off: speed, or never losing a confirmed write |
Demo 5: measure both
Each trip to the replica and back costs 50 ms (time.sleep(NET) stands in for the network). We place 20 orders. In sync mode every order waits for the replica's OK. In async mode orders go into a queue that is shipped in batches of 8. Then the primary crashes right after order 20.
import time
NET = 0.05 # 50 ms to the replica and back
def run(mode):
primary, replica, queue = [], [], []
start = time.perf_counter()
for order in range(1, 21): # 20 orders
primary.append(order)
if mode == "sync":
time.sleep(NET) # wait for the replica's OK
replica.append(order)
else:
queue.append(order) # ship later, in batches of 8
if len(queue) == 8:
time.sleep(NET); replica.extend(queue); queue.clear()
ms = (time.perf_counter() - start) * 1000 # then the primary crashes
print(f"{mode:5}: {ms:4.0f} ms for 20 orders, replica has "
f"{len(replica)}, lost {len(primary) - len(replica)}")
run("async"); run("sync")
Output:
async: 100 ms for 20 orders, replica has 16, lost 4
sync : 1006 ms for 20 orders, replica has 20, lost 0
Async: about 100 ms for 20 orders, but the replica has only 16. Four confirmed orders are lost: the customers were told "order placed", and those orders never left the primary. Sync: about 1,000 ms, ten times slower, but the replica has all 20.
This matches PostgreSQL's docs. "Streaming replication is asynchronous by default", and if the primary crashes, "some transactions that were committed may not have been replicated to the standby server, causing data loss". Synchronous replication confirms that changes reached "one or more synchronous standby servers", and it "necessarily increases the response time for the requesting transaction".
Choosing sync or async
| Choice | Good for | Why |
|---|---|---|
| Async | Most data: likes, views, chat read receipts | Losing the last few is acceptable; speed matters |
| Sync | Money and bookings: payments, seat bookings, orders | A confirmed write must never be lost |
| Middle way (semi-synchronous) | Wait for 1 replica, let the others catch up async | One safe copy without waiting for all |
| Far-away replica | Another city | Every synchronous write waits for the signal to travel there and back |
The question to ask for each kind of data: what can you afford to lose?
Problem 6: switch to the copy (failover), and the split-brain trap
Back to 1 pm, but this time we have a replica. A monitor sends the primary a heartbeat, a tiny "are you alive?" message, every second. Ten seconds of silence, and we call it dead. Then:
- Promote the replica: it becomes the new primary and starts accepting writes.
- Point the app's writes at the new primary.
Switching to the copy like this is called failover. The shop is open again in seconds, not hours. But there's a trap in step one: silence doesn't always mean death. Maybe a network cable failed. The monitor can't reach the primary, but the primary is fine, and some customers can still reach it. Now there are two primaries: the old one never stopped, and the new one was just promoted. Both accept writes. This is called split brain. And there's one biryani plate left.
Demo 6: two bosses sell one biryani twice
# The network splits. The replica is promoted to primary,
# but the old primary is still running. ONE biryani plate left.
old_primary = {"biryani": 1}
new_primary = {"biryani": 1}
def order(db, name):
if db["biryani"] > 0:
db["biryani"] -= 1
return f"{name}: order confirmed"
return f"{name}: sold out"
print(order(old_primary, "Asha")) # Asha's app reaches the old one
print(order(new_primary, "Ravi")) # Ravi's app reaches the new one
print("plates promised: 2, plates in the kitchen: 1")
Output:
Asha: order confirmed
Ravi: order confirmed
plates promised: 2, plates in the kitchen: 1
Each primary was right by its own data; together they were wrong. When the network heals, the two copies disagree, and there's no clean way to merge them. PostgreSQL's docs warn about exactly this: "you must have a mechanism for informing the old primary that it is no longer the primary", which is "necessary to avoid situations where both systems think they are the primary, which will lead to confusion and ultimately data loss."
Only one boss: majority, term number, fencing
How does a cricket team avoid two captains?
| On a cricket team | In a cluster |
|---|---|
| The team votes; a captain needs a majority | Majority vote (quorum): a new primary needs more than half the nodes to agree. A node cut off from most of the others can't win, so it can't be elected |
| Captaincy number 2: every new captain gets a higher number | Term number (epoch): goes up by one at every failover |
| The umpire checks the number and ignores the old captain's calls | Fencing: whatever accepts writes rejects writes stamped with an old term |
(A cluster is a group of machines working as one system; each machine is a node.) PostgreSQL's docs name the blunt version of fencing: "This is sometimes known as STONITH (Shoot The Other Node In The Head)". The idea: forcibly stop the old primary, so it truly stops acting as one.
Demo 7: fencing with a term number
The kitchen is the one thing both primaries can reach. It remembers the highest term it has seen and refuses any order with an older term.
class Kitchen: # the one thing both can reach
def __init__(self):
self.term, self.plates = 1, 1
def take_order(self, term, name):
if term < self.term:
raise PermissionError(f"term {term} is fenced off, "
f"current term is {self.term}")
self.term = term
self.plates -= 1
return f"{name}: confirmed (term {term})"
kitchen = Kitchen()
print(kitchen.take_order(2, "Ravi")) # new primary was elected: term 2
print(kitchen.take_order(1, "Asha")) # old one: term 1
Output:
Ravi: confirmed (term 2)
Traceback (most recent call last):
File "main.py", line 15, in <module>
print(kitchen.take_order(1, "Asha")) # old one: term 1
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "main.py", line 7, in take_order
raise PermissionError(f"term {term} is fenced off, "
f"current term is {self.term}")
PermissionError: term 1 is fenced off, current term is 2
Ravi's order, from the new primary (term 2), is confirmed. Asha's, from the old primary (still on term 1), is refused. Her app gets the error, retries, finds the new primary, which correctly says sold out (practice 5 shows this). One plate, one sale.
The whole chain
- One database on one disk dies → single point of failure (
unable to open database file). - Write every change to a change log and replay it in order on a copy → primary + replica (PostgreSQL streams its WAL to a standby).
- Spread reads over copies → read replicas (read-only; writes go to the primary).
- The copy is late → replication lag, stale reads → read-your-writes.
- When is a write saved? Async: fast, may lose the last writes. Sync: safe, slower. Semi-sync in between.
- The primary dies → failover (heartbeat, promote) → the trap: split brain → majority vote + term number + fencing (STONITH).
Common mistakes (and the real errors)
| Mistake | Code | Error / symptom | Fix |
|---|---|---|---|
| Writing to a replica | replica = sqlite3.connect("file:shop.db?mode=ro", uri=True) then INSERT |
sqlite3.OperationalError: attempt to write a readonly database |
Send every INSERT, UPDATE, DELETE to the primary |
| Only one copy | sqlite3.connect("file:orders.db?mode=rw", uri=True), the file was on the dead disk |
sqlite3.OperationalError: unable to open database file |
Keep at least one replica on another machine |
| Reading your own write from a replica | save(address) → primary, show(address) ← replica |
right away: Old flat, Pune (no error) |
Read a user's own recent writes from the primary |
| Still using the dead primary | primary.close() then primary.execute(...) |
sqlite3.ProgrammingError: Cannot operate on a closed database. |
After failover, send every write to the new primary |
| Sharing a connection across threads | sqlite3.connect(":memory:") used inside a Thread |
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. ... |
check_same_thread=False (and only one writer at a time), or one connection per thread |
Forgetting uri=True |
sqlite3.connect("shop.db?mode=ro") |
No error: a new file literally named shop.db?mode=ro appears |
Pass uri=True and start the name with file: |
| Trusting silence | Promote the replica while the old primary still runs | Two "order confirmed" for one plate (no error) | Majority vote, term numbers, fencing |
The errors in full:
import sqlite3
primary = sqlite3.connect("shop.db")
primary.execute("CREATE TABLE IF NOT EXISTS orders (item TEXT)")
primary.commit()
replica = sqlite3.connect("file:shop.db?mode=ro", uri=True)
print(replica.execute("SELECT COUNT(*) FROM orders").fetchone()[0])
replica.execute("INSERT INTO orders VALUES ('chai')")
Output:
0
Traceback (most recent call last):
File "main.py", line 7, in <module>
replica.execute("INSERT INTO orders VALUES ('chai')")
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: attempt to write a readonly database
Reading through the read-only connection works (0 orders); writing fails. Here both connections open the same file, so mode=ro plays the part of a replica that refuses writes.
The other errors in the table were produced the same way (run as main.py); the thread error also prints the two thread ids, which differ on every run. Opening a file that doesn't exist with mode=ro fails the same way as Demo 1: sqlite3.OperationalError: unable to open database file.
Misconceptions
- "A replica is a backup." No. A replica copies every change, including mistakes. Run
DELETE FROM ordersby accident and the replica deletes everything too (practice 6). Keep real backups as well, so you can go back in time. - "Replicas always show the latest data." They show it after the lag. Usually under a second, but never zero.
- "Synchronous replication is always better." It never loses a confirmed write, but every write waits, and a far-away or slow replica slows every write.
- "No heartbeat means the primary is dead." It may only be cut off. Promoting a replica without fencing the old primary causes split brain.
- "More replicas make writes faster." Replicas spread reads. Every write still goes to one primary, and every replica has to replay every write. Splitting the writes is next episode: sharding.
Mini project: a tiny replicated key-value store
Three nodes (db1 is the primary, db2 and db3 are replicas), a change log on each node, replication lag, read-your-writes routing, and failover with a term number. A write counts as saved only when a majority (2 of 3) of nodes have it in their log, so a cut-off node can't save anything on its own, and failover never loses a confirmed write. Each node shows a change in its data only LAG seconds after it arrives (the primary shows it at once). A fake clock (self.now, moved by tick) makes the output identical on every run.
# Mini project: a tiny replicated key-value store for ChaiNow.
# 3 nodes (1 primary + 2 replicas), a change log, replication lag,
# read-your-writes, and failover with a term number and a majority.
# A fake clock (self.now) makes the output the same on every run.
LAG = 1.0 # a replica shows a change 1 s after it arrives
RYW_WINDOW = 5.0 # after you write, read from the primary for 5 s
class Node:
def __init__(self, name):
self.name, self.term = name, 1
self.log, self.data, self.applied = [], {}, 0 # applied = entries replayed
class Cluster:
def __init__(self):
self.nodes = {n: Node(n) for n in ["db1", "db2", "db3"]}
self.primary, self.now = "db1", 0.0
self.last_write = {} # user -> time of their last write
self.cut_off = set() # nodes the network can't reach
def reachable(self, a, b):
return a == b or (a not in self.cut_off and b not in self.cut_off)
def write(self, via, user, key, value):
p = self.nodes[via]
entry = (p.term, self.now, key, value)
got = 0
for n in self.nodes.values(): # ship the change to every node
if not self.reachable(via, n.name):
continue
if n.term > p.term: # fencing: an old term is refused
return f"REFUSED: term {p.term} is fenced off, current term is {n.term}"
n.log.append(entry)
got += 1
if got < 2: # majority of 3 = 2
return f"NOT SAVED: only {got} of 3 nodes have it (no majority)"
self.last_write[user] = self.now
return f"saved by {via} (term {p.term}), {got} of 3 nodes have it"
def tick(self, seconds): # time passes: replay the log
self.now += seconds
for n in self.nodes.values():
while n.applied < len(n.log) and (n.name == self.primary
or n.log[n.applied][1] + LAG <= self.now):
_, _, key, value = n.log[n.applied]
n.data[key] = value
n.applied += 1
def read(self, user, key):
recent = self.now - self.last_write.get(user, -99) < RYW_WINDOW
name = self.primary if recent else "db2" # replicas take normal reads
return f"{user:4} reads {key} from {name}: {self.nodes[name].data.get(key)}"
def failover(self): # the monitor saw no heartbeat
voters = [n for n in self.nodes.values() if n.name not in self.cut_off]
if len(voters) < 2:
return "no majority, no new primary"
best = max(voters, key=lambda n: len(n.log)) # most complete log wins
new_term = max(n.term for n in voters) + 1
for n in voters:
n.term = new_term # everyone who voted remembers it
old, self.primary = self.primary, best.name
self.tick(0)
return f"failover: {old} -> {best.name}, term {new_term}"
c = Cluster()
print(c.write("db1", "asha", "asha_address", "Old flat, Pune")); c.tick(2)
print(c.write("db1", "asha", "asha_address", "New flat, Bengaluru")); c.tick(0)
print(c.read("asha", "asha_address")) # her own write: primary
print(c.read("ravi", "asha_address")) # others: replica, still behind
c.tick(1)
print(c.read("ravi", "asha_address")) # 1 s later: caught up
print("-- 1 pm: the network cuts db1 off --")
c.cut_off = {"db1"}
print(c.failover())
print(c.write("db1", "asha", "plates_left", 0)) # old primary, alone
print(c.write("db2", "ravi", "plates_left", 0)) # new primary, with db3
print("-- the network heals --")
c.cut_off = set()
print(c.write("db1", "asha", "plates_left", 0)) # old term 1 is fenced
Output:
saved by db1 (term 1), 3 of 3 nodes have it
saved by db1 (term 1), 3 of 3 nodes have it
asha reads asha_address from db1: New flat, Bengaluru
ravi reads asha_address from db2: Old flat, Pune
ravi reads asha_address from db2: New flat, Bengaluru
-- 1 pm: the network cuts db1 off --
failover: db1 -> db2, term 2
NOT SAVED: only 1 of 3 nodes have it (no majority)
saved by db2 (term 2), 2 of 3 nodes have it
-- the network heals --
REFUSED: term 1 is fenced off, current term is 2
Every idea from the episode in one place: Asha reads her own write from the primary while Ravi sees the lag; failover promotes db2 with term 2; the cut-off db1 can't save the last biryani plate because it has no majority; and when the network heals, db1's term-1 write is fenced off. To extend it: make db1 step down and catch up when refused (throwing away the unsaved entry in its log), or add nodes and work out the new majority.
Practice
- (Homework) In the lag demo (Demo 3), change
0.5to2. What does the1 s laterline print now? Change the program to check the replica every 0.25 s and print when it catches up. - (Homework) In the sync demo (Demo 5), set
NET = 0.2(a replica in another city). How slow is sync now? Does async lose more or fewer orders? - (Homework) In the fencing demo (Demo 7), what happens if Asha's term were 3? Is that a bug?
- (Homework) Would you replicate UPI payments sync or async? Why?
- Change Demo 7 so the kitchen also checks stock, and Asha's app retries after the error through the new primary. What does she see?
- Show that a replica is not a backup: in Demo 2, run
DELETE FROM orders(noWHERE) throughwrite, thenreplicate(). - In Demo 5's async mode, shipping happens in batches of 8. Without running it, how many orders would be lost with batches of 5? Of 20?
- Why does a 3-node cluster need 2 votes to elect a primary? Why can't both halves of a split network win?
Solutions and hints
1. Change line 10 of Demo 3 to time.sleep(2). The replica is still behind one second later:
right away: Old flat, Pune
1 s later : Old flat, Pune
Make the main program wait time.sleep(2.5) instead of time.sleep(1) and the new address appears: the replica catches up after about 2 s, when the background thread first wakes up. Lesson: your read-your-writes window (5 s in Demo 4) must be longer than your real lag.
2. Sync pays the trip on every order: 20 × 200 ms ≈ 4 s. Async pays it only twice (two full batches), about 400 ms. Async still loses 4: the loss depends on what is waiting in the queue, not on the network speed.
async: 400 ms for 20 orders, replica has 16, lost 4
sync : 4003 ms for 20 orders, replica has 20, lost 0
3. Term 3 is higher than 2, so the kitchen accepts it. Change the last line of Demo 7 to kitchen.take_order(3, "Asha") and print kitchen.term and kitchen.plates:
Ravi: confirmed (term 2)
Asha: confirmed (term 3)
term: 3 plates: -1
The term check did its job: a higher term means a newer, properly elected primary, so it wins. But plates went to -1, because this tiny kitchen never checks stock. Fencing stops an old boss; it doesn't replace normal checks like "is there a plate left?". (And an old primary can't give itself term 3: a term only goes up through a majority vote.)
4. A sensible answer: synchronous (or semi-synchronous: wait for at least one replica). Once a customer sees "payment successful", that payment must survive a crash, so we accept slower writes. Async would be fine for things like a list of recent searches. (This is reasoning about the trade-off, not a claim about how any real payment system is built.)
5.
class Kitchen:
def __init__(self):
self.term, self.plates = 1, 1
def take_order(self, term, name):
if term < self.term:
raise PermissionError(f"term {term} is fenced off, "
f"current term is {self.term}")
self.term = term
if self.plates == 0: # the stock check fencing can't do
return f"{name}: sold out"
self.plates -= 1
return f"{name}: confirmed (term {term})"
kitchen = Kitchen()
print(kitchen.take_order(2, "Ravi"))
try:
print(kitchen.take_order(1, "Asha")) # old primary: fenced
except PermissionError as e:
print("Asha's app got:", e, "-> retrying on the new primary")
print(kitchen.take_order(2, "Asha")) # the retry, through term 2
Output:
Ravi: confirmed (term 2)
Asha's app got: term 1 is fenced off, current term is 2 -> retrying on the new primary
Asha: sold out
One plate, one sale, and Asha gets an honest "sold out" instead of a false confirmation.
6. In Demo 2, replace everything after the first replicate() with:
write("DELETE FROM orders") # oops: no WHERE clause
replicate()
print("primary:", primary.execute("SELECT * FROM orders").fetchall())
print("replica:", replica.execute("SELECT * FROM orders").fetchall())
Output:
primary: []
replica: []
The replica faithfully copied the mistake. Replicas protect you from a machine dying; backups protect you from mistakes (and need to be tested by actually restoring them).
7. Batches of 5: 20 is a multiple of 5, so all 4 batches are shipped and nothing is lost (but only by luck: crash after order 19 and 4 are lost). Batches of 20: the batch ships exactly at order 20, so 0 lost; crash after order 19 and all 19 are lost. With async, what you can lose is whatever hasn't been shipped yet.
8. 2 is more than half of 3. A split network divides the nodes into two groups, and only one group can have more than half, so at most one side can elect a primary. The side with one node can't reach a majority, so it can't elect anyone (and in the mini project, it can't even save a write).
Quick quiz
# async replication, batches of 8
# orders 1-16 shipped to the replica
# orders 17-20 waiting in the queue
primary.crash()
replica.promote()
# which orders are lost?
- A. None
- B. Orders 1-16
- C. Orders 17-20
- D. All 20
Answer
C, orders 17-20. Only shipped changes reach the replica. Orders 17 to 20 were confirmed to customers, but never left the primary. That's the price of async (Demo 5 prints exactly this: replica has 16, lost 4).
Interview questions
- What is a single point of failure? Name one in a typical one-server web app.
- How does a replica stay in sync with the primary? (By replaying the primary's change log, in order; in PostgreSQL, streamed WAL records.)
- What is a read replica? Why must replicas be read-only?
- What is replication lag? Give an example of a bug it causes, and how you would fix it.
- Explain read-your-writes. How long should the "read from the primary" window be?
- Compare synchronous and asynchronous replication. What does each one cost? What is semi-synchronous replication?
- With async replication, what exactly can be lost when the primary crashes?
- What is failover? How does a system decide the primary is dead, and why is that hard?
- What is split brain? How do majority votes, term numbers and fencing prevent it? What is STONITH?
- Why do clusters usually have an odd number of nodes (3, 5)?
- Is a replica a backup? Why not?
- Replicas help with reads. What do you do when writes or data size outgrow one primary? (Sharding: next episode.)
Glossary
| Term | Meaning |
|---|---|
| Single point of failure | One part whose failure takes down the whole system |
| Backup | A copy of the data saved at one moment, used to go back in time |
| Replication | Keeping copies of a database up to date on other machines |
| Primary | The one database that accepts writes (also called the leader) |
| Replica | A copy that replays the primary's changes; serves reads |
| Standby | PostgreSQL's name for a replica, "ready to take over operations if the primary server fails" |
| Change log | An ordered list of every change made to the primary |
| WAL (write-ahead log) | PostgreSQL's change log; the primary streams its records to standbys |
| Streaming replication | The primary sends log records to the standby continuously, as they're generated |
| Replay | Apply each logged change again, in the same order, on a copy |
| Read replica | A replica used to answer read queries, taking load off the primary |
| Read-only | Can be read but refuses writes (attempt to write a readonly database) |
| Write / read | A change (INSERT, UPDATE, DELETE) / a query that only looks (SELECT) |
| Commit | Make a change final: the moment the database says "saved" |
| Transaction | A group of changes committed together |
| Replication lag | The delay between a commit on the primary and the change appearing on a replica |
| Stale read | A read that returns old data because the replica is behind |
| Read-your-writes | A guarantee that you always see your own changes; usually by reading from the primary right after you write |
| Asynchronous replication | The primary says OK at once and ships the change later: fast, may lose the last writes |
| Synchronous replication | The primary says OK only after a replica confirms: no loss, every write waits |
| Semi-synchronous replication | Wait for one replica to confirm; the others catch up asynchronously |
| Batch | Several changes shipped together in one trip |
| Round trip | A message there and the reply back |
| Failover | Switching to a replica when the primary fails |
| Promote | Turn a replica into the new primary, so it starts accepting writes |
| Heartbeat | A tiny regular "are you alive?" message; missing heartbeats suggest a failure |
| Monitor | A program that watches heartbeats and starts a failover |
| Network partition | A network failure that splits machines into groups that can't reach each other |
| Split brain | Two nodes both acting as primary at once, accepting conflicting writes |
| Cluster / node | A group of machines working as one system / one machine in it |
| Majority / quorum | More than half of the nodes (2 of 3, 3 of 5); needed to elect a primary or save a write |
| Term / epoch | A number that goes up at every failover; the higher term is the current primary |
| Fencing | Rejecting writes from an old primary (an old term), so only one boss can write |
| STONITH | "Shoot The Other Node In The Head": forcibly stopping the old primary so it can't keep acting as one |
| Sharding | Splitting the data itself across machines (next episode) |
sqlite3.connect(":memory:") |
A database that lives in RAM and vanishes when closed |
sqlite3 URI (file:..., uri=True) |
Open a database by URI so you can pass options like mode |
mode=ro / mode=rw |
URI options: open read-only / read-write; the file must already exist |
check_same_thread=False |
Lets a sqlite3 connection be used from threads other than the one that created it |
| Thread | A second line of work running at the same time as the main program |
| Daemon thread | A background thread that Python doesn't wait for when the program ends (daemon=True) |
time.sleep() / time.perf_counter() |
Pause for some seconds / a precise clock for measuring elapsed time |
PermissionError |
A built-in Python exception, used here to mean "not allowed: your term is old" |
OperationalError / ProgrammingError |
sqlite3 errors: the database operation failed / the API was misused |
Summary
- One database on one disk is a single point of failure: when the disk dies, every order fails and every old order is gone. A nightly backup still loses the day's orders.
- A replica stays in sync by replaying the primary's change log in the same order. PostgreSQL streams its WAL to a standby.
- Writes go to the primary; reads can go to any replica (read replicas are read-only).
- Replicas are always a little behind: replication lag (PostgreSQL: "typically under one second"), which causes stale reads. Fix: read-your-writes, a user's own recent reads go to the primary.
- Async replication is fast but can lose the last confirmed writes (about 100 ms, lost 4); sync never loses them but every write waits (about 1,000 ms, lost 0). PostgreSQL is asynchronous by default.
- Failover promotes a replica when heartbeats stop. But a cut-off primary may still be alive: split brain. Prevent it with a majority vote, a term number, and fencing (STONITH).
Next episode: replicas give us copies, but every copy holds all the data, and every write still goes to one primary. What happens when there's too much data for one machine? Sharding: you'll split the database itself.