SyntaxPilot
Courses/System Design/Lesson 11
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

System Design · Lesson 11 of 40

Replication

queued

These are the notes for System Design episode 11. Last episode, indexes made our database thousands of times faster. But it's still one machine, on one disk. Here that disk dies in the middle of lunch rush, and you invent the fix: keep a live copy. A primary takes the writes, a replica replays the primary's change log, and replicas can serve reads. Then the trouble starts: the copy is always a little behind (replication lag), so we invent read-your-writes. We measure asynchronous vs synchronous replication (fast but can lose writes, or safe but slower), switch to the copy when the primary dies (failover), and meet the scariest bug in this topic: split brain, two databases that both think they're the boss. The fix is a majority vote, a term number and fencing. Every demo is plain Python (some with the built-in sqlite3) and runs as-is; outputs are real (Python 3.13, SQLite 3.53). Timings vary a little from run to run.

The six problems

Problem What it teaches
The only database dies Single point of failure (unable to open database file)
Keep a live copy Primary, replica, change log (PostgreSQL's WAL), read replicas
The copy is behind Replication lag, stale reads
Asha sees her old address Read-your-writes: route a user's own reads to the primary
When is a write saved? Asynchronous vs synchronous replication, measured
Switch to the copy, safely Failover, split brain, majority, term number, fencing

Problem 1: the only database dies (single point of failure)

It's 1 pm at ChaiNow. Lunch rush. Orders arrive every second, and every one of them is stored in one database, on one machine. Then the disk dies. No bug, no mistake: disks are machines, and machines break. With enough of them, one breaks somewhere every day.

Now every new order fails, and every old order is gone too. When one part breaking takes down the whole system, that part is called a single point of failure.

Demo 1: the database disappears

A real SQLite database file, orders.db. We save one order, then "kill the disk" by deleting the file, and the app tries to open the database again. file:orders.db?mode=rw with uri=True opens the file in read-write mode: it must already exist (plain sqlite3.connect("orders.db") would quietly create a new, empty file instead, which hides the problem).

import sqlite3, os
db = sqlite3.connect("orders.db")
db.execute("CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT)")
db.execute("INSERT INTO orders (item) VALUES ('masala chai')")
db.commit()
print("orders saved:", db.execute("SELECT COUNT(*) FROM orders").fetchone()[0])
db.close()

os.remove("orders.db")                       # 1 pm: the disk dies
print("1 pm: the database disk died")
db = sqlite3.connect("file:orders.db?mode=rw", uri=True)
print(db.execute("SELECT * FROM orders").fetchall())

Output:

orders saved: 1
1 pm: the database disk died
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    db = sqlite3.connect("file:orders.db?mode=rw", uri=True)
sqlite3.OperationalError: unable to open database file

One order saved, then sqlite3.OperationalError: unable to open database file. The only copy is gone: every order, every customer, every bill.

Why not just take a nightly backup? A backup is a copy saved at one moment, like a photo. Last night's backup still loses every order from this morning, and restoring it takes time while the shop is closed. Backups are still important (see the misconceptions below), but to stay open we need a copy that is live: updated all day, ready to take over.

Problem 2: keep a live copy (primary, replica, change log)

An old Indian shop solved this with paper:

In a shop In a database
The main register: the owner writes every sale here Primary: the one database that accepts writes
The day book: every sale, in order, line by line Change log: an ordered list of everything that changed
The accountant's copy: he copies the day book into his own register Replica: a copy that replays the log, in the same order

The key idea: replay the same changes, in the same order, and you get the same data. Copying changes like this is called replication. To replay means to apply each logged change again, one by one, on the copy.

Demo 2: a primary, a replica and a log

Two in-memory SQLite databases (":memory:" means the database lives in RAM and disappears when closed) and a Python list as the change log.

import sqlite3
primary = sqlite3.connect(":memory:")
replica = sqlite3.connect(":memory:")
log = []                                   # the change log
def write(sql, args=()):
    primary.execute(sql, args)             # 1. change the primary
    log.append((sql, args))                # 2. note it in the log
def replicate():
    while log:                             # replay, in the same order
        replica.execute(*log.pop(0))
write("CREATE TABLE orders (id INTEGER PRIMARY KEY, item TEXT)")
write("INSERT INTO orders (item) VALUES (?)", ("masala chai",))
write("INSERT INTO orders (item) VALUES (?)", ("samosa",))
replicate()
print("primary:", primary.execute("SELECT * FROM orders").fetchall())
print("replica:", replica.execute("SELECT * FROM orders").fetchall())
primary.close()                            # 1 pm: the primary dies
print("after crash, replica:", replica.execute("SELECT * FROM orders").fetchall())

Output:

primary: [(1, 'masala chai'), (2, 'samosa')]
replica: [(1, 'masala chai'), (2, 'samosa')]
after crash, replica: [(1, 'masala chai'), (2, 'samosa')]

Primary and replica hold the same two orders, and after the primary dies, the replica still has both. Nothing lost.

This is how real databases do it. PostgreSQL's change log is called the write-ahead log (WAL). PostgreSQL calls its replicas standby servers, "ready to take over operations if the primary server fails". Its docs describe streaming replication like this: "The standby connects to the primary, which streams WAL records to the standby as they're generated."

          writes                      WAL records (the change log)
  App ─────────────▶  Primary  ──────────────────────────────▶  Standby / replica
                     (writes)                                   (replays the log)

Read replicas: the free bonus

Copies give us a bonus. Writes still go to one place, the primary, which ships its changes to replica 1 and replica 2. But reads can go to any replica. People open the ChaiNow menu far more often than they order, so spreading reads over replicas takes most of the load off the primary. A replica used this way is called a read replica. PostgreSQL's docs: "A standby server can also be used for read-only queries."

              ┌──────────▶  Replica 1  (reads)
  App ──▶ Primary (all writes)
              └──────────▶  Replica 2  (reads)

Replicas are read-only. If the app wrote to a replica, that change would never reach the primary, and the copies would drift apart. So replicas refuse writes (see the first error in the mistakes table). Rule of thumb: writes go to the primary, reads can go to any replica.

Problem 3: the copy is behind (replication lag)

Asha moves from Pune to Bengaluru. She opens ChaiNow and saves her new address: a write, so it goes to the primary. The app reloads her profile: a read, so it goes to a replica. And the screen shows Old flat, Pune. Asha thinks her change wasn't saved, so she types it again, or worse, her biryani is delivered to Pune.

Demo 3: reading back from a replica that's behind

A background thread (a second line of work running at the same time as the main program) ships the log to the replica every half second. That half second stands in for network time plus the time to apply the change. daemon=True makes it a daemon thread: Python doesn't wait for it when the main program ends. check_same_thread=False lets both threads use the same SQLite connection (without it you get the ProgrammingError in the mistakes table).

import sqlite3, threading, time
primary = sqlite3.connect(":memory:", check_same_thread=False)
replica = sqlite3.connect(":memory:", check_same_thread=False)
for db in (primary, replica):
    db.execute("CREATE TABLE users (id INTEGER, address TEXT)")
    db.execute("INSERT INTO users VALUES (1, 'Old flat, Pune')")
log = []
def ship_log():                            # runs in the background
    while True:
        time.sleep(0.5)                    # network + apply time = lag
        while log:
            replica.execute(*log.pop(0))
threading.Thread(target=ship_log, daemon=True).start()
change = ("UPDATE users SET address = ? WHERE id = 1", ("New flat, Bengaluru",))
primary.execute(*change); log.append(change)   # saved on the primary
print("right away:", replica.execute("SELECT address FROM users").fetchone()[0])
time.sleep(1)
print("1 s later :", replica.execute("SELECT address FROM users").fetchone()[0])

Output:

right away: Old flat, Pune
1 s later : New flat, Bengaluru

Right away: the old address. One second later: the new one. The replica wasn't wrong, just late. That gap is called replication lag, and a read that returns old data because of it is a stale read. PostgreSQL's docs say "there is a small delay between committing a transaction in the primary and the changes becoming visible in the standby", "typically under one second". (To commit means to make a change final: "saved".) When the network or the replica is busy, the lag grows. No error appears anywhere, which is what makes this bug sneaky.

Problem 4: read your own writes

The fix comes from the bank:

At the bank In the app
You deposit cash; the clerk updates the main ledger Your write goes to the primary
To check your balance now, ask that same clerk, not a branch that syncs tonight Read from the primary for a few seconds after you write
Everyone else can wait for the evening sync Others read replicas: a little lag is fine for them

For a few seconds after a user writes, that user's reads go to the primary. This guarantee is called read-your-writes: you always see your own changes. Ravi doesn't care if he sees Asha's address a second late, so most reads stay cheap.

Demo 4: route Asha to the primary, Ravi to a replica

Plain dictionaries stand in for the two databases, so we can focus on the routing.

import time
last_write = {}                            # user -> time of their last write

def save(user, primary):
    primary[user] = "New flat, Bengaluru"
    last_write[user] = time.time()

def pick_db(user, primary, replica):
    if time.time() - last_write.get(user, 0) < 5:   # wrote in the last 5 s?
        return "primary", primary          # read your own writes
    return "replica", replica              # everyone else: the replica

primary = {"asha": "Old flat, Pune", "ravi": "MG Road"}
replica = dict(primary)                    # not caught up yet
save("asha", primary)
for user in ["asha", "ravi"]:
    name, db = pick_db(user, primary, replica)
    print(f"{user:5} reads from the {name}: {db[user]}")

Output:

asha  reads from the primary: New flat, Bengaluru
ravi  reads from the replica: MG Road

Asha sees her new flat; Ravi reads from the (still behind) replica and gets MG Road, which is correct for him. The 5 seconds is a choice: pick a window longer than your normal lag. Only the person who just wrote pays the cost of reading from the primary. In a real app, last_write would live somewhere every app server can see (for example, a timestamp in the user's session), not in one process's memory.

Problem 5: when is a write "saved"? (sync vs async)

When the app tells Asha "order placed", is the order really safe? You can send a parcel two ways:

Sending a parcel Replicating a write
Drop it in the post box and walk away Asynchronous: the primary says OK at once and ships the change to the replica a little later. Fast, but not yet safe
Wait at the counter for a signed receipt Synchronous: the primary says OK only after a replica confirms it has the change. Safe, but you wait
A birthday card vs a property deed A trade-off: speed, or never losing a confirmed write

Demo 5: measure both

Each trip to the replica and back costs 50 ms (time.sleep(NET) stands in for the network). We place 20 orders. In sync mode every order waits for the replica's OK. In async mode orders go into a queue that is shipped in batches of 8. Then the primary crashes right after order 20.

import time
NET = 0.05                                 # 50 ms to the replica and back
def run(mode):
    primary, replica, queue = [], [], []
    start = time.perf_counter()
    for order in range(1, 21):             # 20 orders
        primary.append(order)
        if mode == "sync":
            time.sleep(NET)                # wait for the replica's OK
            replica.append(order)
        else:
            queue.append(order)            # ship later, in batches of 8
            if len(queue) == 8:
                time.sleep(NET); replica.extend(queue); queue.clear()
    ms = (time.perf_counter() - start) * 1000   # then the primary crashes
    print(f"{mode:5}: {ms:4.0f} ms for 20 orders, replica has "
          f"{len(replica)}, lost {len(primary) - len(replica)}")
run("async"); run("sync")

Output:

async:  100 ms for 20 orders, replica has 16, lost 4
sync : 1006 ms for 20 orders, replica has 20, lost 0

Async: about 100 ms for 20 orders, but the replica has only 16. Four confirmed orders are lost: the customers were told "order placed", and those orders never left the primary. Sync: about 1,000 ms, ten times slower, but the replica has all 20.

This matches PostgreSQL's docs. "Streaming replication is asynchronous by default", and if the primary crashes, "some transactions that were committed may not have been replicated to the standby server, causing data loss". Synchronous replication confirms that changes reached "one or more synchronous standby servers", and it "necessarily increases the response time for the requesting transaction".

Choosing sync or async

Choice Good for Why
Async Most data: likes, views, chat read receipts Losing the last few is acceptable; speed matters
Sync Money and bookings: payments, seat bookings, orders A confirmed write must never be lost
Middle way (semi-synchronous) Wait for 1 replica, let the others catch up async One safe copy without waiting for all
Far-away replica Another city Every synchronous write waits for the signal to travel there and back

The question to ask for each kind of data: what can you afford to lose?

Problem 6: switch to the copy (failover), and the split-brain trap

Back to 1 pm, but this time we have a replica. A monitor sends the primary a heartbeat, a tiny "are you alive?" message, every second. Ten seconds of silence, and we call it dead. Then:

  1. Promote the replica: it becomes the new primary and starts accepting writes.
  2. Point the app's writes at the new primary.

Switching to the copy like this is called failover. The shop is open again in seconds, not hours. But there's a trap in step one: silence doesn't always mean death. Maybe a network cable failed. The monitor can't reach the primary, but the primary is fine, and some customers can still reach it. Now there are two primaries: the old one never stopped, and the new one was just promoted. Both accept writes. This is called split brain. And there's one biryani plate left.

Demo 6: two bosses sell one biryani twice

# The network splits. The replica is promoted to primary,
# but the old primary is still running. ONE biryani plate left.
old_primary = {"biryani": 1}
new_primary = {"biryani": 1}

def order(db, name):
    if db["biryani"] > 0:
        db["biryani"] -= 1
        return f"{name}: order confirmed"
    return f"{name}: sold out"

print(order(old_primary, "Asha"))          # Asha's app reaches the old one
print(order(new_primary, "Ravi"))          # Ravi's app reaches the new one
print("plates promised: 2, plates in the kitchen: 1")

Output:

Asha: order confirmed
Ravi: order confirmed
plates promised: 2, plates in the kitchen: 1

Each primary was right by its own data; together they were wrong. When the network heals, the two copies disagree, and there's no clean way to merge them. PostgreSQL's docs warn about exactly this: "you must have a mechanism for informing the old primary that it is no longer the primary", which is "necessary to avoid situations where both systems think they are the primary, which will lead to confusion and ultimately data loss."

Only one boss: majority, term number, fencing

How does a cricket team avoid two captains?

On a cricket team In a cluster
The team votes; a captain needs a majority Majority vote (quorum): a new primary needs more than half the nodes to agree. A node cut off from most of the others can't win, so it can't be elected
Captaincy number 2: every new captain gets a higher number Term number (epoch): goes up by one at every failover
The umpire checks the number and ignores the old captain's calls Fencing: whatever accepts writes rejects writes stamped with an old term

(A cluster is a group of machines working as one system; each machine is a node.) PostgreSQL's docs name the blunt version of fencing: "This is sometimes known as STONITH (Shoot The Other Node In The Head)". The idea: forcibly stop the old primary, so it truly stops acting as one.

Demo 7: fencing with a term number

The kitchen is the one thing both primaries can reach. It remembers the highest term it has seen and refuses any order with an older term.

class Kitchen:                             # the one thing both can reach
    def __init__(self):
        self.term, self.plates = 1, 1

    def take_order(self, term, name):
        if term < self.term:
            raise PermissionError(f"term {term} is fenced off, "
                                  f"current term is {self.term}")
        self.term = term
        self.plates -= 1
        return f"{name}: confirmed (term {term})"

kitchen = Kitchen()
print(kitchen.take_order(2, "Ravi"))       # new primary was elected: term 2
print(kitchen.take_order(1, "Asha"))       # old one: term 1

Output:

Ravi: confirmed (term 2)
Traceback (most recent call last):
  File "main.py", line 15, in <module>
    print(kitchen.take_order(1, "Asha"))       # old one: term 1
          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "main.py", line 7, in take_order
    raise PermissionError(f"term {term} is fenced off, "
                          f"current term is {self.term}")
PermissionError: term 1 is fenced off, current term is 2

Ravi's order, from the new primary (term 2), is confirmed. Asha's, from the old primary (still on term 1), is refused. Her app gets the error, retries, finds the new primary, which correctly says sold out (practice 5 shows this). One plate, one sale.

The whole chain

  1. One database on one disk dies → single point of failure (unable to open database file).
  2. Write every change to a change log and replay it in order on a copy → primary + replica (PostgreSQL streams its WAL to a standby).
  3. Spread reads over copies → read replicas (read-only; writes go to the primary).
  4. The copy is late → replication lag, stale reads → read-your-writes.
  5. When is a write saved? Async: fast, may lose the last writes. Sync: safe, slower. Semi-sync in between.
  6. The primary dies → failover (heartbeat, promote) → the trap: split brain → majority vote + term number + fencing (STONITH).

Common mistakes (and the real errors)

Mistake Code Error / symptom Fix
Writing to a replica replica = sqlite3.connect("file:shop.db?mode=ro", uri=True) then INSERT sqlite3.OperationalError: attempt to write a readonly database Send every INSERT, UPDATE, DELETE to the primary
Only one copy sqlite3.connect("file:orders.db?mode=rw", uri=True), the file was on the dead disk sqlite3.OperationalError: unable to open database file Keep at least one replica on another machine
Reading your own write from a replica save(address) → primary, show(address) ← replica right away: Old flat, Pune (no error) Read a user's own recent writes from the primary
Still using the dead primary primary.close() then primary.execute(...) sqlite3.ProgrammingError: Cannot operate on a closed database. After failover, send every write to the new primary
Sharing a connection across threads sqlite3.connect(":memory:") used inside a Thread sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. ... check_same_thread=False (and only one writer at a time), or one connection per thread
Forgetting uri=True sqlite3.connect("shop.db?mode=ro") No error: a new file literally named shop.db?mode=ro appears Pass uri=True and start the name with file:
Trusting silence Promote the replica while the old primary still runs Two "order confirmed" for one plate (no error) Majority vote, term numbers, fencing

The errors in full:

import sqlite3
primary = sqlite3.connect("shop.db")
primary.execute("CREATE TABLE IF NOT EXISTS orders (item TEXT)")
primary.commit()
replica = sqlite3.connect("file:shop.db?mode=ro", uri=True)
print(replica.execute("SELECT COUNT(*) FROM orders").fetchone()[0])
replica.execute("INSERT INTO orders VALUES ('chai')")

Output:

0
Traceback (most recent call last):
  File "main.py", line 7, in <module>
    replica.execute("INSERT INTO orders VALUES ('chai')")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: attempt to write a readonly database

Reading through the read-only connection works (0 orders); writing fails. Here both connections open the same file, so mode=ro plays the part of a replica that refuses writes.

The other errors in the table were produced the same way (run as main.py); the thread error also prints the two thread ids, which differ on every run. Opening a file that doesn't exist with mode=ro fails the same way as Demo 1: sqlite3.OperationalError: unable to open database file.

Misconceptions

  • "A replica is a backup." No. A replica copies every change, including mistakes. Run DELETE FROM orders by accident and the replica deletes everything too (practice 6). Keep real backups as well, so you can go back in time.
  • "Replicas always show the latest data." They show it after the lag. Usually under a second, but never zero.
  • "Synchronous replication is always better." It never loses a confirmed write, but every write waits, and a far-away or slow replica slows every write.
  • "No heartbeat means the primary is dead." It may only be cut off. Promoting a replica without fencing the old primary causes split brain.
  • "More replicas make writes faster." Replicas spread reads. Every write still goes to one primary, and every replica has to replay every write. Splitting the writes is next episode: sharding.

Mini project: a tiny replicated key-value store

Three nodes (db1 is the primary, db2 and db3 are replicas), a change log on each node, replication lag, read-your-writes routing, and failover with a term number. A write counts as saved only when a majority (2 of 3) of nodes have it in their log, so a cut-off node can't save anything on its own, and failover never loses a confirmed write. Each node shows a change in its data only LAG seconds after it arrives (the primary shows it at once). A fake clock (self.now, moved by tick) makes the output identical on every run.

# Mini project: a tiny replicated key-value store for ChaiNow.
# 3 nodes (1 primary + 2 replicas), a change log, replication lag,
# read-your-writes, and failover with a term number and a majority.
# A fake clock (self.now) makes the output the same on every run.

LAG = 1.0           # a replica shows a change 1 s after it arrives
RYW_WINDOW = 5.0    # after you write, read from the primary for 5 s

class Node:
    def __init__(self, name):
        self.name, self.term = name, 1
        self.log, self.data, self.applied = [], {}, 0   # applied = entries replayed

class Cluster:
    def __init__(self):
        self.nodes = {n: Node(n) for n in ["db1", "db2", "db3"]}
        self.primary, self.now = "db1", 0.0
        self.last_write = {}                 # user -> time of their last write
        self.cut_off = set()                 # nodes the network can't reach

    def reachable(self, a, b):
        return a == b or (a not in self.cut_off and b not in self.cut_off)

    def write(self, via, user, key, value):
        p = self.nodes[via]
        entry = (p.term, self.now, key, value)
        got = 0
        for n in self.nodes.values():        # ship the change to every node
            if not self.reachable(via, n.name):
                continue
            if n.term > p.term:              # fencing: an old term is refused
                return f"REFUSED: term {p.term} is fenced off, current term is {n.term}"
            n.log.append(entry)
            got += 1
        if got < 2:                          # majority of 3 = 2
            return f"NOT SAVED: only {got} of 3 nodes have it (no majority)"
        self.last_write[user] = self.now
        return f"saved by {via} (term {p.term}), {got} of 3 nodes have it"

    def tick(self, seconds):                 # time passes: replay the log
        self.now += seconds
        for n in self.nodes.values():
            while n.applied < len(n.log) and (n.name == self.primary
                    or n.log[n.applied][1] + LAG <= self.now):
                _, _, key, value = n.log[n.applied]
                n.data[key] = value
                n.applied += 1

    def read(self, user, key):
        recent = self.now - self.last_write.get(user, -99) < RYW_WINDOW
        name = self.primary if recent else "db2"   # replicas take normal reads
        return f"{user:4} reads {key} from {name}: {self.nodes[name].data.get(key)}"

    def failover(self):                      # the monitor saw no heartbeat
        voters = [n for n in self.nodes.values() if n.name not in self.cut_off]
        if len(voters) < 2:
            return "no majority, no new primary"
        best = max(voters, key=lambda n: len(n.log))   # most complete log wins
        new_term = max(n.term for n in voters) + 1
        for n in voters:
            n.term = new_term                # everyone who voted remembers it
        old, self.primary = self.primary, best.name
        self.tick(0)
        return f"failover: {old} -> {best.name}, term {new_term}"

c = Cluster()
print(c.write("db1", "asha", "asha_address", "Old flat, Pune")); c.tick(2)
print(c.write("db1", "asha", "asha_address", "New flat, Bengaluru")); c.tick(0)
print(c.read("asha", "asha_address"))      # her own write: primary
print(c.read("ravi", "asha_address"))      # others: replica, still behind
c.tick(1)
print(c.read("ravi", "asha_address"))      # 1 s later: caught up

print("-- 1 pm: the network cuts db1 off --")
c.cut_off = {"db1"}
print(c.failover())
print(c.write("db1", "asha", "plates_left", 0))     # old primary, alone
print(c.write("db2", "ravi", "plates_left", 0))     # new primary, with db3
print("-- the network heals --")
c.cut_off = set()
print(c.write("db1", "asha", "plates_left", 0))     # old term 1 is fenced

Output:

saved by db1 (term 1), 3 of 3 nodes have it
saved by db1 (term 1), 3 of 3 nodes have it
asha reads asha_address from db1: New flat, Bengaluru
ravi reads asha_address from db2: Old flat, Pune
ravi reads asha_address from db2: New flat, Bengaluru
-- 1 pm: the network cuts db1 off --
failover: db1 -> db2, term 2
NOT SAVED: only 1 of 3 nodes have it (no majority)
saved by db2 (term 2), 2 of 3 nodes have it
-- the network heals --
REFUSED: term 1 is fenced off, current term is 2

Every idea from the episode in one place: Asha reads her own write from the primary while Ravi sees the lag; failover promotes db2 with term 2; the cut-off db1 can't save the last biryani plate because it has no majority; and when the network heals, db1's term-1 write is fenced off. To extend it: make db1 step down and catch up when refused (throwing away the unsaved entry in its log), or add nodes and work out the new majority.

Practice

  1. (Homework) In the lag demo (Demo 3), change 0.5 to 2. What does the 1 s later line print now? Change the program to check the replica every 0.25 s and print when it catches up.
  2. (Homework) In the sync demo (Demo 5), set NET = 0.2 (a replica in another city). How slow is sync now? Does async lose more or fewer orders?
  3. (Homework) In the fencing demo (Demo 7), what happens if Asha's term were 3? Is that a bug?
  4. (Homework) Would you replicate UPI payments sync or async? Why?
  5. Change Demo 7 so the kitchen also checks stock, and Asha's app retries after the error through the new primary. What does she see?
  6. Show that a replica is not a backup: in Demo 2, run DELETE FROM orders (no WHERE) through write, then replicate().
  7. In Demo 5's async mode, shipping happens in batches of 8. Without running it, how many orders would be lost with batches of 5? Of 20?
  8. Why does a 3-node cluster need 2 votes to elect a primary? Why can't both halves of a split network win?
Solutions and hints

1. Change line 10 of Demo 3 to time.sleep(2). The replica is still behind one second later:

right away: Old flat, Pune
1 s later : Old flat, Pune

Make the main program wait time.sleep(2.5) instead of time.sleep(1) and the new address appears: the replica catches up after about 2 s, when the background thread first wakes up. Lesson: your read-your-writes window (5 s in Demo 4) must be longer than your real lag.

2. Sync pays the trip on every order: 20 × 200 ms ≈ 4 s. Async pays it only twice (two full batches), about 400 ms. Async still loses 4: the loss depends on what is waiting in the queue, not on the network speed.

async:  400 ms for 20 orders, replica has 16, lost 4
sync : 4003 ms for 20 orders, replica has 20, lost 0

3. Term 3 is higher than 2, so the kitchen accepts it. Change the last line of Demo 7 to kitchen.take_order(3, "Asha") and print kitchen.term and kitchen.plates:

Ravi: confirmed (term 2)
Asha: confirmed (term 3)
term: 3 plates: -1

The term check did its job: a higher term means a newer, properly elected primary, so it wins. But plates went to -1, because this tiny kitchen never checks stock. Fencing stops an old boss; it doesn't replace normal checks like "is there a plate left?". (And an old primary can't give itself term 3: a term only goes up through a majority vote.)

4. A sensible answer: synchronous (or semi-synchronous: wait for at least one replica). Once a customer sees "payment successful", that payment must survive a crash, so we accept slower writes. Async would be fine for things like a list of recent searches. (This is reasoning about the trade-off, not a claim about how any real payment system is built.)

5.

class Kitchen:
    def __init__(self):
        self.term, self.plates = 1, 1

    def take_order(self, term, name):
        if term < self.term:
            raise PermissionError(f"term {term} is fenced off, "
                                  f"current term is {self.term}")
        self.term = term
        if self.plates == 0:               # the stock check fencing can't do
            return f"{name}: sold out"
        self.plates -= 1
        return f"{name}: confirmed (term {term})"

kitchen = Kitchen()
print(kitchen.take_order(2, "Ravi"))
try:
    print(kitchen.take_order(1, "Asha"))   # old primary: fenced
except PermissionError as e:
    print("Asha's app got:", e, "-> retrying on the new primary")
    print(kitchen.take_order(2, "Asha"))   # the retry, through term 2

Output:

Ravi: confirmed (term 2)
Asha's app got: term 1 is fenced off, current term is 2 -> retrying on the new primary
Asha: sold out

One plate, one sale, and Asha gets an honest "sold out" instead of a false confirmation.

6. In Demo 2, replace everything after the first replicate() with:

write("DELETE FROM orders")                # oops: no WHERE clause
replicate()
print("primary:", primary.execute("SELECT * FROM orders").fetchall())
print("replica:", replica.execute("SELECT * FROM orders").fetchall())

Output:

primary: []
replica: []

The replica faithfully copied the mistake. Replicas protect you from a machine dying; backups protect you from mistakes (and need to be tested by actually restoring them).

7. Batches of 5: 20 is a multiple of 5, so all 4 batches are shipped and nothing is lost (but only by luck: crash after order 19 and 4 are lost). Batches of 20: the batch ships exactly at order 20, so 0 lost; crash after order 19 and all 19 are lost. With async, what you can lose is whatever hasn't been shipped yet.

8. 2 is more than half of 3. A split network divides the nodes into two groups, and only one group can have more than half, so at most one side can elect a primary. The side with one node can't reach a majority, so it can't elect anyone (and in the mini project, it can't even save a write).

Quick quiz

# async replication, batches of 8
# orders 1-16 shipped to the replica
# orders 17-20 waiting in the queue

primary.crash()
replica.promote()

# which orders are lost?
  • A. None
  • B. Orders 1-16
  • C. Orders 17-20
  • D. All 20
Answer

C, orders 17-20. Only shipped changes reach the replica. Orders 17 to 20 were confirmed to customers, but never left the primary. That's the price of async (Demo 5 prints exactly this: replica has 16, lost 4).

Interview questions

  • What is a single point of failure? Name one in a typical one-server web app.
  • How does a replica stay in sync with the primary? (By replaying the primary's change log, in order; in PostgreSQL, streamed WAL records.)
  • What is a read replica? Why must replicas be read-only?
  • What is replication lag? Give an example of a bug it causes, and how you would fix it.
  • Explain read-your-writes. How long should the "read from the primary" window be?
  • Compare synchronous and asynchronous replication. What does each one cost? What is semi-synchronous replication?
  • With async replication, what exactly can be lost when the primary crashes?
  • What is failover? How does a system decide the primary is dead, and why is that hard?
  • What is split brain? How do majority votes, term numbers and fencing prevent it? What is STONITH?
  • Why do clusters usually have an odd number of nodes (3, 5)?
  • Is a replica a backup? Why not?
  • Replicas help with reads. What do you do when writes or data size outgrow one primary? (Sharding: next episode.)

Glossary

Term Meaning
Single point of failure One part whose failure takes down the whole system
Backup A copy of the data saved at one moment, used to go back in time
Replication Keeping copies of a database up to date on other machines
Primary The one database that accepts writes (also called the leader)
Replica A copy that replays the primary's changes; serves reads
Standby PostgreSQL's name for a replica, "ready to take over operations if the primary server fails"
Change log An ordered list of every change made to the primary
WAL (write-ahead log) PostgreSQL's change log; the primary streams its records to standbys
Streaming replication The primary sends log records to the standby continuously, as they're generated
Replay Apply each logged change again, in the same order, on a copy
Read replica A replica used to answer read queries, taking load off the primary
Read-only Can be read but refuses writes (attempt to write a readonly database)
Write / read A change (INSERT, UPDATE, DELETE) / a query that only looks (SELECT)
Commit Make a change final: the moment the database says "saved"
Transaction A group of changes committed together
Replication lag The delay between a commit on the primary and the change appearing on a replica
Stale read A read that returns old data because the replica is behind
Read-your-writes A guarantee that you always see your own changes; usually by reading from the primary right after you write
Asynchronous replication The primary says OK at once and ships the change later: fast, may lose the last writes
Synchronous replication The primary says OK only after a replica confirms: no loss, every write waits
Semi-synchronous replication Wait for one replica to confirm; the others catch up asynchronously
Batch Several changes shipped together in one trip
Round trip A message there and the reply back
Failover Switching to a replica when the primary fails
Promote Turn a replica into the new primary, so it starts accepting writes
Heartbeat A tiny regular "are you alive?" message; missing heartbeats suggest a failure
Monitor A program that watches heartbeats and starts a failover
Network partition A network failure that splits machines into groups that can't reach each other
Split brain Two nodes both acting as primary at once, accepting conflicting writes
Cluster / node A group of machines working as one system / one machine in it
Majority / quorum More than half of the nodes (2 of 3, 3 of 5); needed to elect a primary or save a write
Term / epoch A number that goes up at every failover; the higher term is the current primary
Fencing Rejecting writes from an old primary (an old term), so only one boss can write
STONITH "Shoot The Other Node In The Head": forcibly stopping the old primary so it can't keep acting as one
Sharding Splitting the data itself across machines (next episode)
sqlite3.connect(":memory:") A database that lives in RAM and vanishes when closed
sqlite3 URI (file:..., uri=True) Open a database by URI so you can pass options like mode
mode=ro / mode=rw URI options: open read-only / read-write; the file must already exist
check_same_thread=False Lets a sqlite3 connection be used from threads other than the one that created it
Thread A second line of work running at the same time as the main program
Daemon thread A background thread that Python doesn't wait for when the program ends (daemon=True)
time.sleep() / time.perf_counter() Pause for some seconds / a precise clock for measuring elapsed time
PermissionError A built-in Python exception, used here to mean "not allowed: your term is old"
OperationalError / ProgrammingError sqlite3 errors: the database operation failed / the API was misused

Summary

  • One database on one disk is a single point of failure: when the disk dies, every order fails and every old order is gone. A nightly backup still loses the day's orders.
  • A replica stays in sync by replaying the primary's change log in the same order. PostgreSQL streams its WAL to a standby.
  • Writes go to the primary; reads can go to any replica (read replicas are read-only).
  • Replicas are always a little behind: replication lag (PostgreSQL: "typically under one second"), which causes stale reads. Fix: read-your-writes, a user's own recent reads go to the primary.
  • Async replication is fast but can lose the last confirmed writes (about 100 ms, lost 4); sync never loses them but every write waits (about 1,000 ms, lost 0). PostgreSQL is asynchronous by default.
  • Failover promotes a replica when heartbeats stop. But a cut-off primary may still be alive: split brain. Prevent it with a majority vote, a term number, and fencing (STONITH).

Next episode: replicas give us copies, but every copy holds all the data, and every write still goes to one primary. What happens when there's too much data for one machine? Sharding: you'll split the database itself.