These are the notes for System Design episode 8. Last episode you invented the cache, but we only ever read from it. Here people start writing: new prices, new likes, new menus. Every write asks one question: who updates the cache, and when? You invent four answers, cache-aside, invalidation on write, write-through and write-back, break each one, and finish with the cache stampede that happens when a popular entry expires. Every Python demo below is self-contained and runs as-is; the outputs shown are real (Python 3.13). Demos that use time.sleep or threads vary by a few milliseconds per run.
The six problems
| Problem | What it teaches |
|---|---|
| The cache you already built | Cache-aside (lazy loading) |
| The price changed | Invalidate on write: database first, then delete the copy |
| Never stale, please | Write-through, two trips per write, cache churn |
| 1,000 likes a second | Write-back (write-behind): batch the writes |
| The server crashed | Write-back can lose writes |
| Everyone misses at once | Cache stampede (thundering herd), lock, TTL jitter |
Problem 1: the cache you already built (cache-aside)
A waiter keeps a notepad with today's specials and prices. When a customer asks about a dish that isn't on the notepad, he walks to the kitchen, asks, and jots it down. The kitchen never writes on his notepad: the waiter does everything.
That is the cache from episode 7, and it has a name: cache-aside. The cache sits aside; the application talks to both the cache and the database. On a read:
- Look in the cache. Found it? That's a hit: return it.
- Not there? That's a miss: read the database...
- ...and put the answer in the cache for next time.
AWS's ElastiCache documentation calls the same thing lazy loading: it "loads data into the cache only when necessary". Its strengths: only data someone actually asked for is cached, and if the cache dies the app still works (just slower). Its cost: "each cache miss results in three trips" (check the cache, read the database, write the cache).
read: app ──► cache ──(miss)──► database
▲ │
└──── fill the cache ◄─────┘
Demo 1: cache-aside, and a write that skips the cache
database = {"masala chai": 20}
cache = {}
def read_price(item): # cache-aside: the app does it
if item in cache:
return cache[item], "hit"
price = database[item] # miss: read the database
cache[item] = price # then fill the cache
return price, "miss"
def update_price(item, price): # a write goes to the database only
database[item] = price
print(read_price("masala chai"))
update_price("masala chai", 25) # the owner raises the price
print(read_price("masala chai"), "<- database says 25")
Output:
(20, 'miss')
(20, 'hit') <- database says 25
The second read is a hit, but it is stale: the database says 25, the cache says 20. The write changed the original, and nobody told the copy. A TTL (episode 7) limits how long this lasts, but a customer could still pay the wrong price until it expires.
Problem 2: the price changed (invalidate on write)
What should a write do to the cache? Two obvious ideas:
| Idea | What happens |
|---|---|
Update the copy too (cache[item] = 25) |
Works with one writer. But if two writes happen at the same moment and finish in a different order (write A to the DB, write B to the DB, write B to the cache, write A to the cache), the cache keeps the older value forever. |
Delete the copy (cache.pop(item, None)) |
The next read misses and fetches the fresh value from the database. Simple and safe. This is invalidation. |
And the order matters: write the database first, then delete the cached copy. The original changes first, and the copy disappears right after.
Demo 2: the full cache-aside pattern
database = {"masala chai": 20}
cache = {}
def read_price(item): # cache-aside read
if item in cache:
return cache[item], "hit"
cache[item] = database[item]
return cache[item], "miss"
def update_price(item, price): # cache-aside write
database[item] = price # 1. write the database
cache.pop(item, None) # 2. delete the cached copy
print(read_price("masala chai"))
update_price("masala chai", 25)
print(read_price("masala chai"))
print(read_price("masala chai"))
Output:
(20, 'miss')
(25, 'miss')
(25, 'hit')
After the write, the next read is a miss and gets the new price, 25; then it's a hit again. This is the most common strategy you'll meet: read through the cache, write to the database, invalidate. Its cost: the first read after every write is a miss.
Problem 3: never stale, please (write-through)
At a railway station, when the station master changes a train's time, he changes it in his register and on the notice board, at the same moment. The board is never wrong.
Write-through does the same: every write goes to the database and the cache together. In AWS's words, write-through "adds data or updates data in the cache whenever data is written to the database", so "data in the cache is never stale", and reads of freshly written data never miss.
The price: "every write involves two trips", so writes get slower. And everything you write is cached, even if nobody ever reads it.
Demo 3: write-through caches everything you write
import random
database, cache = {}, {}
reads = set()
def write_through(item_id, price):
database[item_id] = price # write the database
cache[item_id] = price # and the cache, every time
def read_price(item_id):
reads.add(item_id)
return cache[item_id] # never stale, never a miss here
for i in range(1000): # 1,000 menu items are written
write_through(i, random.randint(10, 99))
for _ in range(500): # but people only read 3 of them
read_price(random.choice([0, 1, 2]))
print("items in cache:", len(cache))
print("items ever read:", len(reads))
Output:
items in cache: 1000
items ever read: 3
1,000 items take memory, and only 3 were ever read. AWS calls this cache churn: "most data is never read, which is a waste of resources." That is why write-through is usually paired with a TTL, so unread items expire on their own. AWS's own example is cache.set(customer_id, customer_record, 300): the record expires 300 seconds (5 minutes) after it was written.
Problem 4: 1,000 likes a second (write-back)
A reel goes viral. Every like is a tiny write: add one to a counter. With write-through, every single like is a database write.
Demo 4: every like hits the database
import time
def db_write(count): # the database: about 2 ms a write
time.sleep(0.002)
start = time.perf_counter()
likes = 0
for user in range(1000): # 1,000 people like a reel
likes += 1
db_write(likes) # write-through: every like hits the DB
ms = (time.perf_counter() - start) * 1000
print(f"1000 likes -> 1000 database writes, {ms:.0f} ms")
Output:
1000 likes -> 1000 database writes, 2196 ms
About 2 seconds of database work to count to a thousand (timing varies slightly). Do we really need every like saved the instant it happens?
A kirana shop owner doesn't open his big khata for every packet of biscuits. He scribbles each sale on a rough slip at the counter, and at night he adds it up and writes the total into the khata, once. That is write-back, also called write-behind: writes go to the cache only, and the cache is flushed to the database later, in a batch.
Demo 5: write-back batches the writes
import time
def db_write(count): # the database: about 2 ms a write
time.sleep(0.002)
start = time.perf_counter()
cache = {"likes": 0}
db_writes = 0
for user in range(1000):
cache["likes"] += 1 # write-back: update the cache only
if user % 500 == 499: # flush to the database now and then
db_write(cache["likes"])
db_writes += 1
ms = (time.perf_counter() - start) * 1000
print(f"1000 likes -> {db_writes} database writes, {ms:.0f} ms")
Output:
1000 likes -> 2 database writes, 5 ms
2 database writes instead of 1,000. Real systems usually flush on a timer (every second or so) instead of every 500 writes. Counters, view counts, likes and analytics events are often buffered like this.
Problem 5: the server crashed (write-back loses data)
If the slip blows away before night, the day's sales are gone.
Demo 6: a crash before the flush
cache = {"likes": 0}
database = {"likes": 0}
def flush(): # copy the cache into the database
database["likes"] = cache["likes"]
for user in range(1, 1001):
cache["likes"] += 1 # write-back
if user % 500 == 0:
flush()
if user == 937:
print("server crashed! the cache is gone")
break
print("likes people gave:", 937)
print("likes the database saved:", database["likes"])
Output:
server crashed! the cache is gone
likes people gave: 937
likes the database saved: 500
437 likes are lost forever, because they lived only in memory. TechTarget's definition of write-back puts it plainly: "there's a risk of data loss if the associated application closes or crashes before the cache is finally committed."
| OK for write-back | Never write-back |
|---|---|
| Likes, views, play counts | UPI payments, bank balances |
| "Last seen" timestamps | Food orders |
| Analytics events | Train seat bookings |
| Game scores (saved often) | Anything you've told the user is "done" |
Systems that need both speed and safety keep the pending writes in a durable log (a file or a message queue that survives a crash) and apply them later. You'll meet that idea in the message queues episode.
Problem 6: everyone misses at once (cache stampede)
The cached menu has a TTL, and it runs out at 1 pm, right in the lunch rush. A hundred users arrive in the same instant. All of them miss, and each one thinks: fine, I'll go to the database and refill it. The database gets a hundred copies of the same heavy query at once.
This is a cache stampede, also called the thundering herd problem: when a popular entry expires, "every request that wanted it misses simultaneously, and all of them query the database at once" (Redisson glossary).
Demo 7: 100 users, 100 identical queries
import threading, time
cache, db_queries = {}, 0
def db_get_menu(): # a slow, heavy query: 200 ms
global db_queries
db_queries += 1
time.sleep(0.2)
return ["chai", "samosa", "poha"]
def get_menu():
if "menu" not in cache: # the menu just expired...
cache["menu"] = db_get_menu()
return cache["menu"]
users = [threading.Thread(target=get_menu) for _ in range(100)]
for u in users: u.start() # 100 users arrive at once
for u in users: u.join()
print("database queries:", db_queries)
Output:
database queries: 100
Every thread checked the cache before anyone had refilled it. They all saw empty, and they all went. A thread here plays one user: a separate line of work that runs at the same time as the others.
Demo 8: a lock, and check again
At a dhaba, when the dal pot runs empty, one waiter takes the ladle and the rest wait a moment. In code, the ladle is a lock: only one thread may hold it at a time. One request refills the cache, the others wait. This is called single-flight (or request coalescing).
import threading, time
cache, queries = {}, []
lock = threading.Lock()
def db_get_menu(): # a slow, heavy query: 200 ms
queries.append(1) # count every database query
time.sleep(0.2)
return ["chai", "samosa", "poha"]
def get_menu():
if "menu" not in cache:
with lock: # only one user may refill it
if "menu" not in cache: # check again inside
cache["menu"] = db_get_menu()
return cache["menu"]
users = [threading.Thread(target=get_menu) for _ in range(100)]
for u in users: u.start()
for u in users: u.join()
print("database queries:", len(queries))
Output:
database queries: 1
The second check inside the lock is the key line. Threads that were waiting for the lock find the cache already full and skip the database. Without it (see the mistakes below), every waiting thread still refills the cache, one after another.
Other fixes from the same toolbox:
| Fix | Idea | Good for |
|---|---|---|
| Lock / single-flight | One request refills; others wait | One very hot key |
| TTL jitter | Add a few random seconds to each TTL so keys written together don't expire together | Many keys cached at the same time |
| Early refresh | Refresh a hot entry a little before it expires | Keys that must never be empty |
Choosing a strategy
| Strategy | Read path | Write path | Stale? | Best for |
|---|---|---|---|---|
| Cache-aside (lazy loading) | App reads cache; on a miss reads DB and fills cache | Write DB, then delete the copy | Briefly, without invalidation | The default: menus, profiles, product pages |
| Write-through | Read cache | Write DB and cache together | Never | Data read right after it's written; pair with a TTL |
| Write-back (write-behind) | Read cache | Write cache only; flush to DB later | No, but writes can be lost | Huge write volume where small losses are fine: likes, views |
| Write-around | Read cache; fill on a miss | Write DB only | Until the next miss/TTL | Data that is rarely read back: logs, uploads |
Real systems mix these: different strategies for different data, in the same app.
The whole chain
- Our cache filled itself on a miss → cache-aside (lazy loading).
- A write made it stale → write the database first, then delete the copy (invalidate).
- We wanted it never stale → write-through: two trips per write, and cache churn.
- A viral reel flooded us with writes → write-back: batch them, flush later.
- A crash lost the unflushed likes → write-back is only for data where small losses are fine.
- A hot key expired and everyone missed together → cache stampede → a lock with a second check, or TTL jitter.
Common mistakes (and the real errors)
| Mistake | Code | What you see | Fix |
|---|---|---|---|
| A write that never touches the cache | database[item] = p only |
(20, 'hit') <- database says 25 (no error: a silent stale hit) |
Write the DB, then cache.pop(item, None) |
pop without a default |
cache.pop("masala chai") on an item that isn't cached |
KeyError: 'masala chai' |
cache.pop(item, None) |
| A lock with no second check | with lock: cache["menu"] = db_get_menu() |
database queries: 100 (one by one) |
Check the cache again inside the lock |
| Write-back for money | buffering payments in memory | lost writes after a crash | Write money to the database first (with a transaction, next episode) |
| The same TTL on everything cached at once | TTL = 60 for 10,000 keys loaded at startup |
all of them expire together | Add jitter: TTL + random.randint(0, 15) |
The KeyError in full:
cache = {}
cache.pop("masala chai")
Traceback (most recent call last):
File "main.py", line 2, in <module>
cache.pop("masala chai")
~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'masala chai'
The lock without a second check:
import threading, time
cache, db_queries = {}, 0
lock = threading.Lock()
def db_get_menu():
global db_queries
db_queries += 1
time.sleep(0.2)
return ["chai", "samosa", "poha"]
def get_menu():
if "menu" not in cache:
with lock: # lock, but NO second check
cache["menu"] = db_get_menu()
return cache["menu"]
users = [threading.Thread(target=get_menu) for _ in range(100)]
for u in users: u.start()
for u in users: u.join()
print("database queries:", db_queries)
Output:
database queries: 100
The lock only makes the 100 threads take turns; each one still queries the database.
Misconceptions
- "Update the cache on every write, then it's always right." Two concurrent writes can finish in a different order and leave the older value in the cache. Deleting the copy is safer.
- "Write-through means the cache is never stale, so I need no TTL." It's never stale, but it fills up with data nobody reads (cache churn). Add a TTL.
- "Write-back is just faster write-through." It's faster because it delays the database write, and anything delayed can be lost in a crash.
- "A lock fixes the stampede by itself." Only with the second check inside the lock.
- "Cache-aside is old-fashioned." It is the most common strategy in real systems.
Mini project: the ChaiNow price service
One class that combines the episode: cache-aside reads with a lock and a second check, writes that invalidate, and write-back like counters with a flush.
# ChaiNow price service: cache-aside reads, invalidate on write,
# a lock against stampedes, and write-back like counters.
import threading, time
class PriceService:
def __init__(self):
self.db = {"chai": 20, "samosa": 15} # the "database"
self.db_reads = self.db_writes = 0
self.cache = {} # item -> price
self.likes = {} # write-back buffer
self.lock = threading.Lock()
def _db_read(self, item):
self.db_reads += 1
time.sleep(0.05) # a slow query
return self.db[item]
def get_price(self, item): # cache-aside + lock
if item in self.cache:
return self.cache[item]
with self.lock:
if item not in self.cache: # check again
self.cache[item] = self._db_read(item)
return self.cache[item]
def set_price(self, item, price): # write DB, then invalidate
self.db[item] = price
self.db_writes += 1
self.cache.pop(item, None)
def like(self, item): # write-back: memory only
self.likes[item] = self.likes.get(item, 0) + 1
def flush(self): # batch the likes to the DB
for item, n in self.likes.items():
self.db[item + ":likes"] = self.db.get(item + ":likes", 0) + n
self.db_writes += 1
self.likes.clear()
s = PriceService()
threads = [threading.Thread(target=s.get_price, args=("chai",)) for _ in range(50)]
for t in threads: t.start()
for t in threads: t.join()
print("50 users at once -> db reads:", s.db_reads)
s.set_price("chai", 25)
print("after the price change:", s.get_price("chai"), "| db reads:", s.db_reads)
for _ in range(1000): s.like("chai")
s.flush()
print("1000 likes -> db writes:", s.db_writes, "| saved likes:", s.db["chai:likes"])
Output:
50 users at once -> db reads: 1
after the price change: 25 | db reads: 2
1000 likes -> db writes: 2 | saved likes: 1000
Ideas to extend it: add a TTL to each cached price; flush the likes on a timer with threading.Timer; count hits and misses; make set_price write-through instead and compare.
Practice
- In Demo 8, remove the second check inside the lock. How many database queries now?
- In Demo 6, flush every 100 likes instead of every 500. How many likes are lost at like 937?
- Add a TTL to write-through (like AWS's 300-second example). What happens to a read at
t=400? - Ten thousand keys are cached at startup with
TTL = 60. What happens att=60? How does jitter help? - Which strategy would you choose for: (a) a user's profile page, (b) the live view count of a cricket stream, (c) a UPI payment, (d) server log lines?
- Why should a write update the database before deleting the cached copy, not after?
Solutions and hints
1. 100 queries: the lock only makes the threads take turns.
import threading, time
cache, db_queries = {}, 0
lock = threading.Lock()
def db_get_menu():
global db_queries
db_queries += 1
time.sleep(0.2)
return ["chai", "samosa", "poha"]
def get_menu():
if "menu" not in cache:
with lock: # lock, but NO second check
cache["menu"] = db_get_menu()
return cache["menu"]
users = [threading.Thread(target=get_menu) for _ in range(100)]
for u in users: u.start()
for u in users: u.join()
print("database queries:", db_queries)
Output:
database queries: 100
2. 37 likes lost (the database saved 900).
cache = {"likes": 0}
database = {"likes": 0}
for user in range(1, 1001):
cache["likes"] += 1
if user % 100 == 0: # flush every 100 likes
database["likes"] = cache["likes"]
if user == 937:
break # crash
print("likes the database saved:", database["likes"])
print("likes lost:", 937 - database["likes"])
Output:
likes the database saved: 900
likes lost: 37
Flushing more often loses less in a crash, but costs more database writes. That's the write-back trade-off.
3. At t=10 it's a hit; at t=400 the copy has expired, so it's a miss and is refreshed from the database. Items nobody reads simply expire.
database, cache = {}, {} # cache: item -> (price, expires_at)
TTL = 300 # seconds, like AWS's example
def write_through(item, price, now):
database[item] = price
cache[item] = (price, now + TTL)
def read_price(item, now):
if item in cache and now < cache[item][1]:
return cache[item][0], "hit"
cache[item] = (database[item], now + TTL) # expired or missing
return database[item], "miss"
write_through("chai", 20, now=0)
print("t=10 ", read_price("chai", now=10))
print("t=400", read_price("chai", now=400))
Output:
t=10 (20, 'hit')
t=400 (20, 'miss')
4. All 10,000 expire at the same second, and the next requests all miss together: a stampede across many keys. Jitter spreads the expiries out:
import random
random.seed(7)
TTL = 60
plain = [TTL for key in range(5)] # all saved at t=0
jitter = [TTL + random.randint(0, 15) for key in range(5)]
print("expire at, no jitter: ", plain)
print("expire at, with jitter:", jitter)
Output:
expire at, no jitter: [60, 60, 60, 60, 60]
expire at, with jitter: [70, 64, 72, 61, 62]
5. (a) Cache-aside with invalidation. (b) Write-back: a flood of tiny writes, small losses are fine. (c) No write-back: write the database first, in a transaction; cache only what's safe to be briefly stale. (d) Write-around: write to storage, rarely read back.
6. If you delete the copy first, a read can arrive in between, miss, read the old value from the database, and put it back in the cache. Writing the database first means any read after the delete sees the new value.
Quick quiz
# An app shows the live view count
# of a cricket stream. Millions of
# +1 writes per minute. Losing a few
# views in a crash is OK.
# Which strategy?
- A. Write-through
- B. Write-back
- C. No cache at all
- D. Cache-aside, no TTL
Answer
B, write-back. A huge flood of tiny writes, and small losses are fine: batch them in the cache and flush every second.
Interview questions
- What is cache-aside (lazy loading)? Walk through a read and a write.
- After a write, would you update the cached value or delete it? Why?
- Compare write-through and write-back: speed, staleness and the risk of data loss.
- What is cache churn, and how does a TTL help with write-through?
- What is a cache stampede (thundering herd)? Give three ways to prevent it.
- Why do you need to check the cache again after taking the lock?
- Which strategy would you use for likes, for profile pages and for payments? Why?
Glossary
| Term | Meaning |
|---|---|
| Cache | A saved copy of data, kept close and fast (episode 7) |
| Cache hit / miss | Found in the cache / not found, so the real source is asked |
| Stale data | A cached copy that no longer matches the database |
| TTL (time to live) | Seconds until a cached copy expires |
| Invalidation | Deleting a cached copy on purpose because the real data changed |
| Cache strategy | The rule for when the cache is filled and who updates it on writes |
| Cache-aside | The app reads the cache, fills it on a miss, and writes go to the database (plus invalidation) |
| Lazy loading | AWS's name for cache-aside: data is loaded into the cache only when needed |
| Trip | One request to the cache or the database and its reply |
| Write-through | Every write goes to the database and the cache together |
| Cache churn | Filling the cache with data that is never read |
| Write-back / write-behind | Writes go to the cache only; the cache is flushed to the database later |
| Flush | Copying buffered writes from the cache into the database |
| Batch | Many small writes sent together as one |
| Write-around | Writes go only to the database; the cache fills on later reads |
| Durable | Survives a crash or a power cut (stored on disk) |
| Durable log | A file or queue of pending writes that survives a crash |
| Cache stampede / thundering herd | A popular entry expires and many requests miss and hit the database at the same time |
| Thread | A separate line of work in a program that runs at the same time as others |
Lock (threading.Lock) |
Lets only one thread at a time run a piece of code |
| Single-flight / request coalescing | Only one request recomputes a missing value; the others wait and share it |
| Double check | Checking the cache again after taking the lock |
| TTL jitter | Adding a small random amount to each TTL so keys don't expire together |
| Early refresh | Refreshing a hot entry a little before it expires |
dict.pop(key, default) |
Removes a key and returns its value, or default if it's missing |
KeyError |
Python's error for a dictionary key that isn't there |
set() |
A Python collection of unique values |
random.randint(a, b) |
A random whole number from a to b |
Summary
- Cache-aside (lazy loading): the app reads the cache and fills it on a miss. On a write: update the database, then delete the cached copy.
- Write-through: write the database and cache together. Never stale, but every write is two trips and the cache fills with unread data. Pair it with a TTL.
- Write-back (write-behind): write the cache only and flush later in batches. 1,000 likes became 2 database writes, but a crash lost 437 of them. Never use it for money or orders.
- Write-around: write the database only; good for data that's rarely read back.
- Cache stampede: a hot key expires and everyone misses together (100 queries). A lock with a second check makes it 1; TTL jitter spreads expiries out.
Next episode: we've protected the database for two episodes. But what is the database? SQL vs NoSQL: you'll invent tables, keys and transactions, and then find out when they get in the way.