These are the notes for System Design episode 5. Last episode, we saw that past your throughput limit a queue forms and latency explodes. Here we fix that for real. Your app gets popular, one machine isn't enough, and you invent both kinds of scaling yourself: capacity, vertical scaling (a bigger machine) and its limits, the single point of failure, horizontal scaling (more machines), and why servers must be stateless with a shared store. Every Python demo below is self-contained and runs as-is; the outputs shown are real (Python 3.13). The 100 requests per second per server and 25 per CPU core are teaching numbers, not measurements of a real machine.
The six problems
| Problem | What it teaches |
|---|---|
| Traffic doubles overnight | Capacity: past it, requests fail |
| "Just buy a bigger server" | Vertical scaling (scale up) |
| The big server dies at 2 AM | Single point of failure |
| Many small servers | Horizontal scaling (scale out) |
| "Why am I logged out?" | State stuck in one server's memory |
| One shared store | Stateless servers |
Problem 1: your chai app goes viral (capacity)
You build a little app called ChaiNow: people order chai from their phones, and it all runs on one server. Then someone posts a reel about it and it goes viral. Traffic doubles, then doubles again, and orders start failing: users see "something went wrong". Nothing in your code changed. So what broke?
Every server has a capacity: the most requests per second it can handle. Say ours handles 100.
Demo 1: traffic past capacity
# one server can handle 100 requests per second
CAPACITY = 100
for traffic in [50, 100, 200, 400]:
served = min(traffic, CAPACITY)
failed = traffic - served
print(f"{traffic:>3} req/s -> served {served:>3}, failed {failed:>3}")
Output:
50 req/s -> served 50, failed 0
100 req/s -> served 100, failed 0
200 req/s -> served 100, failed 100
400 req/s -> served 100, failed 300
min(traffic, CAPACITY): the server serves up to its capacity, never more. The rest fail.- At 50 and 100 req/s, everyone is served. At 200, half the requests fail. At 400, three out of every four users see an error.
- Your code isn't wrong. You asked one machine to do more than it can. You need more capacity, and there are only two ways to get it.
Traffic > capacity = errors.
Problem 2: "just buy a bigger server" (vertical scaling)
| At the chai stall | In your system |
|---|---|
| A bigger stove: 4 burners → 16 burners | More CPU cores: more work at the same time |
| A bigger fridge: more milk kept close | More RAM: more data kept in memory |
| Upgrade the same stall, no new stalls | Vertical scaling ("scale up"): replace your machine with a bigger one |
Same stall, just bigger.
Demo 2: scale up
# scale UP: buy a bigger machine (more cores)
PER_CORE = 25 # requests per second per CPU core
TRAFFIC = 400
for cores in [4, 8, 16]:
capacity = cores * PER_CORE
failed = max(0, TRAFFIC - capacity)
print(f"{cores:>2} cores -> capacity {capacity:>3} req/s, failed {failed:>3}")
Output:
4 cores -> capacity 100 req/s, failed 300
8 cores -> capacity 200 req/s, failed 200
16 cores -> capacity 400 req/s, failed 0
max(0, ...)stops "failed" going negative when capacity is bigger than traffic.- 16 cores gives a capacity of 400 and zero failures. It worked!
- This is a great first move: no code changes at all, you just rent a bigger machine.
A real example: in 2016, Stack Overflow (Nick Craver, Stack Overflow: The Architecture - 2016 Edition) reported 209,420,973 HTTP requests to its load balancer in one day (66,294,789 page loads), served by 11 web servers (9 primary) and 4 SQL Servers. It added that it was down to needing only 1 web server, with the warning: "I'm saying it works. I'm not saying it's a good idea."
The three walls of scaling up
| Wall | Why it hurts |
|---|---|
| There's a biggest machine | AWS EC2 High Memory U7i instances offer "up to 32TiB of memory and 896 vCPUs". Huge, but a ceiling. Traffic doesn't have one. |
| Upgrading means downtime | The AWS EC2 user guide: "You must stop your instance before you can change its instance type", and "Ensure that you plan for downtime while your instance is stopped." Every upgrade is a small outage. |
| It's still ONE machine | If it dies, everything dies. |
The third wall is the scariest. Let's see it.
Problem 3: the big server dies at 2 AM (single point of failure)
However big it is, it's still one machine. Let's see what that means when a disk dies at 2 AM.
Demo 3: one big machine crashes
# one big machine, then it crashes
servers = {"big-server": True} # True = up
servers["big-server"] = False # a disk dies at 2 AM
up = [name for name, ok in servers.items() if ok]
print("servers up:", up)
print("handling request on", up[0])
Output:
servers up: []
Traceback (most recent call last):
File "main.py", line 8, in <module>
print("handling request on", up[0])
~~^^^
IndexError: list index out of range
- The list comprehension
[name for name, ok in servers.items() if ok]keeps only the servers that are up. Here, none. up[0]asks for the first item of an empty list:IndexError. There is no server left to take the request.- For a real app, every single user is down, not just some. A part whose failure takes down the whole system is called a single point of failure.
Vertical scaling on one page
- → Replace your machine with a bigger one.
- + Simple: no code changes.
- − A ceiling: there's a biggest machine.
- − Downtime while you upgrade.
- − Still a single point of failure.
Problem 4: many small servers (horizontal scaling)
| On a busy street | In your system |
|---|---|
| Open 5 chai stalls, same recipe in each | 5 servers running the same code |
| Customers spread out: any stall will do | Traffic is split: each server takes a share |
| Busy? Open another. No ceiling, no closing | Horizontal scaling ("scale out"): add machines, not a bigger one |
Demo 4: scale out, then crash one
# scale OUT: many small machines
PER_SERVER = 100
TRAFFIC = 400
servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}
servers["s3"] = False # one machine crashes
up = [name for name, ok in servers.items() if ok]
capacity = len(up) * PER_SERVER
print("servers up:", up)
print(f"capacity {capacity} req/s for {TRAFFIC} req/s of traffic")
# chance that ALL of them are down at once (each up 99%)
print(f"all 5 down: {0.01 ** 5:.10f}")
Output:
servers up: ['s1', 's2', 's4', 's5']
capacity 400 req/s for 400 req/s of traffic
all 5 down: 0.0000000001
- Same disaster, server
s3crashes, but four servers are still up and together they handle 400 req/s. One crash, and nobody noticed. - The last line: if each server is up 99% of the time (down 1% =
0.01) and they fail independently, the chance that all five are down at the same moment is0.01 ** 5= 0.0000000001, which is 1 in 10 billion. Remember the nines from last episode? Many ordinary machines can be far more available than one great one. - Honest warning: that maths assumes the failures are independent. If all five servers share one power supply, they can die together. That's why big systems spread machines across buildings and cities.
Problem 5: "why am I logged out?" (state in memory)
You order 2 samosas at stall 1. The person there remembers you: 2 samosas, pay later. Then you go to pick it up, but this time you walk into stall 2. Stall 2 says: which order? It has never seen you. What stall 1 remembered stayed in stall 1's head. Servers do exactly the same thing.
A session is what a server remembers about you between requests, like your login and your cart.
Demo 5: sessions in each server's own memory
# each server keeps logged-in users in its OWN memory
class Server:
def __init__(self, name):
self.name = name
self.sessions = {}
def login(self, user):
self.sessions[user] = "cart: 2 samosas"
print(f"{self.name}: {user} logged in")
def cart(self, user):
print(f"{self.name}: {user} -> {self.sessions[user]}")
s1, s2 = Server("s1"), Server("s2")
s1.login("riya") # request 1 lands on s1
s1.cart("riya") # request 2 lands on s1: fine
s2.cart("riya") # request 3 lands on s2...
Output:
s1: riya logged in
s1: riya -> cart: 2 samosas
Traceback (most recent call last):
File "main.py", line 17, in <module>
s2.cart("riya") # request 3 lands on s2...
~~~~~~~^^^^^^^^
File "main.py", line 12, in cart
print(f"{self.name}: {user} -> {self.sessions[user]}")
~~~~~~~~~~~~~^^^^^^
KeyError: 'riya'
- Each
Serverobject has its ownsessionsdictionary. Riya's session was written intos1's dictionary only. - When request 3 lands on
s2,self.sessions["riya"]doesn't exist:KeyError: 'riya'. - In a real app, Riya sees please log in again, and her cart is empty. Add more servers, and it happens more often. These servers are stateful: each one keeps its own memory of users.
Problem 6: one shared store (stateless servers)
| On the street | In your system |
|---|---|
| One shared order book: every stall writes in it | A shared store: Redis or a database |
| Any stall can serve you: it just checks the book | Stateless servers: keep nothing about you between requests |
| Add stalls freely: no one's memory is lost | Easy scale out: add or remove any server |
Redis describes itself as "an open-source, in-memory data structure store" used "as a database, cache, message broker".
Demo 6: stateless servers + a shared store
# fix: servers keep NO session in memory; one shared store holds it
shared_store = {} # in real life: Redis or a database
class Server:
def __init__(self, name):
self.name = name
def login(self, user):
shared_store[user] = "cart: 2 samosas"
print(f"{self.name}: {user} logged in")
def cart(self, user):
print(f"{self.name}: {user} -> {shared_store[user]}")
s1, s2, s3 = Server("s1"), Server("s2"), Server("s3")
s1.login("riya")
s2.cart("riya")
s3.cart("riya")
Output:
s1: riya logged in
s2: riya -> cart: 2 samosas
s3: riya -> cart: 2 samosas
- One change: there's no
sessionsdictionary inside the server any more.loginwrites to oneshared_storethat every server reads. s2knows her cart,s3knows her cart. NoKeyError, no logout. Any server can serve any request.- The big idea: make your servers stateless and push the state into a shared store. Then scaling out is just: start one more copy.
Horizontal scaling on one page
- → Add more machines running the same code.
- + No ceiling: just add one more.
- + One crash ≠ everyone down.
- − Servers must be stateless.
- − More moving parts to run.
- ★ Real systems do both: scale up first while it's cheap and simple, then scale out when you need more room or more safety.
Scale up vs scale out
| Vertical · scale up (one bigger machine) | Horizontal · scale out (many machines) | |
|---|---|---|
| Capacity | Limited by the biggest machine | Add another machine |
| Upgrade | Stop, resize, restart | One at a time, no outage |
| Failure | One crash = all down | One crash = a small dip |
| Code | No changes needed | Servers must be stateless |
| State | In memory is fine | In a shared store |
| In short | Simple, but it has a ceiling | More work, but it keeps growing |
The whole chain
- Traffic > capacity → requests fail
- Bigger machine → vertical scaling (scale up)
- It crashed → single point of failure
- More machines → horizontal scaling (scale out)
- Logged out → state stuck in one server
- Shared store → stateless servers
- ? Who sends each request to which server? (In our code we picked
s1,s2,s3by hand.)
Common mistakes (and the real errors)
| Mistake | Code | Error | Fix |
|---|---|---|---|
| Every server is down: nothing left to pick | up = [] then up[0] |
IndexError: list index out of range |
Run more than one server, and check the list isn't empty first |
| The session lives in another server's memory | s2.sessions["riya"] # she logged in on s1 |
KeyError: 'riya' |
Keep sessions in a shared store; servers stay stateless |
| Planning capacity without a real number | capacity = len(up) * PER_SERVER # PER_SERVER not set |
NameError: name 'PER_SERVER' is not defined |
Measure one server's limit with a load test first |
The third one in full:
up = ["s1", "s2"]
capacity = len(up) * PER_SERVER # PER_SERVER not set
Traceback (most recent call last):
File "main.py", line 2, in <module>
capacity = len(up) * PER_SERVER # PER_SERVER not set
^^^^^^^^^^
NameError: name 'PER_SERVER' is not defined
Misconceptions
- "Errors under load mean my code has a bug." Often the code is fine; traffic simply went past capacity.
- "A bigger machine is a safer machine." Bigger ≠ safer. However big it is, one machine is still a single point of failure.
- "Vertical scaling is always wrong." It's a great first move: simple, no code changes. Teams scale up first, then out.
- "Five servers means five times the problems." Five independent servers that are each up 99% are all down together far less often than one server is down. (The catch: the failures must really be independent.)
- "Adding servers is enough." Not if each server keeps sessions in its own memory: users get logged out. Servers must be stateless.
Mini project: the ChaiNow cluster
One program that combines everything: capacity, a crash, stateless servers with a shared store, and what happens when nothing is left. Requests are still picked by hand (request number % number of servers up); next episode a load balancer does this for us.
# ChaiNow cluster: capacity, crashes and stateless servers
PER_SERVER = 100 # teaching number: req/s per server
shared_store = {} # sessions live here, not in a server
servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}
def up():
return [name for name, ok in servers.items() if ok]
def report(traffic):
capacity = len(up()) * PER_SERVER
failed = max(0, traffic - capacity)
print(f" {len(up())} up, capacity {capacity} req/s, traffic {traffic}, failed {failed}")
def handle(request_no, user, action):
live = up()
if not live:
return "503: no server is up"
server = live[request_no % len(live)] # picked by hand for now
if action == "login":
shared_store[user] = "cart: 2 samosas"
return f"{server}: {user} logged in"
return f"{server}: {user} -> {shared_store.get(user, 'please log in')}"
print("Normal day:")
report(400)
print("A disk dies on s3:")
servers["s3"] = False
report(400)
print("Riya's requests:")
for n, action in enumerate(["login", "cart", "cart", "cart"]):
print(" ", handle(n, "riya", action))
print("Total blackout:")
for name in servers:
servers[name] = False
print(" ", handle(4, "riya", "cart"))
Output:
Normal day:
5 up, capacity 500 req/s, traffic 400, failed 0
A disk dies on s3:
4 up, capacity 400 req/s, traffic 400, failed 0
Riya's requests:
s1: riya logged in
s2: riya -> cart: 2 samosas
s4: riya -> cart: 2 samosas
s5: riya -> cart: 2 samosas
Total blackout:
503: no server is up
- Riya's four requests land on four different servers, and every one knows her cart, because the session lives in
shared_store. shared_store.get(user, 'please log in')returns a default instead of raisingKeyError.- Instead of crashing with
IndexErrorwhen no server is up,handlechecks first and returns a clear error. 503 is the HTTP status code for "Service Unavailable".
Practice
- Set
TRAFFIC = 1000withPER_SERVER = 100. How many servers do you need? How many if you also want to survive one crash? - In the scale-out demo (Demo 4), crash two servers. How many requests per second fail?
- Add a
logout()method to the stateless demo (Demo 6). After Riya logs out on one server, what does another server show? - Each server is up 99%, failing independently. Print the chance that all of them are down for 1, 2, 3 and 5 servers. How many nines does each give?
- You run ChaiNow on one AWS EC2 instance and want to change it to a bigger instance type. What must you plan for, and why?
- Would you scale ChaiNow up or out first? Why?
Solutions and hints
1. Divide and round up (a part of a server is still a whole server).
import math
PER_SERVER = 100
TRAFFIC = 1000
needed = math.ceil(TRAFFIC / PER_SERVER)
print("servers needed:", needed)
print("with one spare:", needed + 1)
servers needed: 10
with one spare: 11
With exactly 10, one crash drops you to 900 req/s and 100 req/s fail. One spare server lets you lose any one machine and still serve everyone.
2.
# scale OUT: many small machines
PER_SERVER = 100
TRAFFIC = 400
servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}
servers["s3"] = False # one machine crashes
servers["s5"] = False # ...and then another
up = [name for name, ok in servers.items() if ok]
capacity = len(up) * PER_SERVER
print("servers up:", up)
print(f"capacity {capacity} req/s for {TRAFFIC} req/s of traffic")
print("failed:", max(0, TRAFFIC - capacity), "req/s")
servers up: ['s1', 's2', 's4']
capacity 300 req/s for 400 req/s of traffic
failed: 100 req/s
A quarter of the traffic fails, but three quarters is still served. With one big server, a crash means everything fails.
3.
shared_store = {} # in real life: Redis or a database
class Server:
def __init__(self, name):
self.name = name
def login(self, user):
shared_store[user] = "cart: 2 samosas"
print(f"{self.name}: {user} logged in")
def logout(self, user):
shared_store.pop(user, None)
print(f"{self.name}: {user} logged out")
def cart(self, user):
session = shared_store.get(user)
if session is None:
print(f"{self.name}: {user} -> please log in")
else:
print(f"{self.name}: {user} -> {session}")
s1, s2, s3 = Server("s1"), Server("s2"), Server("s3")
s1.login("riya")
s2.cart("riya")
s3.logout("riya")
s1.cart("riya")
s1: riya logged in
s2: riya -> cart: 2 samosas
s3: riya logged out
s1: riya -> please log in
She logged out on s3, and s1 knows it at once, because the session lived in the shared store. pop(user, None) deletes the key without an error if it's already gone.
4.
for n in [1, 2, 3, 5]:
all_down = 0.01 ** n # each server up 99%, failing independently
print(f"{n} server(s): all down {all_down:.10f} -> up {(1 - all_down) * 100:.8f}%")
1 server(s): all down 0.0100000000 -> up 99.00000000%
2 server(s): all down 0.0001000000 -> up 99.99000000%
3 server(s): all down 0.0000010000 -> up 99.99990000%
5 server(s): all down 0.0000000001 -> up 99.99999999%
Each extra server adds two nines (two nines, four, six, ten). Remember this is "at least one server is up"; whether one server has enough capacity for all the traffic is a separate question (see question 1).
5. Downtime. The AWS EC2 user guide says you must stop the instance before changing its type, and to plan for downtime while it is stopped. With only one machine, that means an outage. With several stateless servers, you can upgrade them one at a time.
6. Either answer can be right if you explain it. A common one: scale up first, because it's simple and needs no code changes; move to scaling out when you hit the ceiling, can't accept upgrade downtime, or can't accept a single point of failure. Make the servers stateless before you scale out.
Quick quiz
Plan A: one server, 64 cores. Plan B: four servers, 16 cores each. Same total power. At 2 AM, one machine crashes. Which plan still serves users?
- A. Plan A
- B. Plan B
- C. Both
- D. Neither
Answer
B, Plan B. Plan B loses a quarter of its capacity but keeps serving. Plan A was one machine, a single point of failure, so everyone is down.
Interview questions
- What is the difference between vertical and horizontal scaling? Give a real-life analogy.
- What are the limits of vertical scaling? (A biggest machine, downtime to upgrade, a single point of failure.)
- What is a single point of failure? How does horizontal scaling remove it, and what assumption does that rely on? (Independent failures.)
- What does "stateless server" mean? Why must servers be stateless before you scale out?
- Where do sessions go if not in the server's memory? (A shared store such as Redis or a database.)
- If each server is up 99% and failures are independent, how likely is it that all three are down at once? (0.01³ = one in a million.)
- When would you choose to scale up instead of out?
- How do you work out how many servers you need? (Measure one server's capacity with a load test, divide traffic by it, round up, add spares.)
Glossary
| Term | Meaning |
|---|---|
| Server | A computer (or program) that answers requests from users' apps |
| Request | One message from an app asking the server to do something, like "show my cart" |
| req/s (requests per second) | How many requests arrive or are handled each second |
| Capacity | The most requests per second a server (or a group of servers) can handle |
| Traffic | How many requests per second users are actually sending |
| Scaling | Growing a system so it can handle more traffic |
| Vertical scaling (scale up) | Replacing your machine with a bigger one: more CPU cores, more RAM |
| Horizontal scaling (scale out) | Adding more machines that run the same code and share the work |
| CPU core | One "worker" inside a processor; more cores can do more work at the same time |
| vCPU | A virtual CPU: a share of a real processor that a cloud provider rents to you |
| RAM (memory) | The fast, temporary storage a computer works from; emptied when the program stops |
| TiB (tebibyte) | A unit of size: 2⁴⁰ bytes, about 1.1 trillion bytes |
| Instance / instance type | A rented cloud machine (on AWS EC2) / its size: how many vCPUs and how much memory |
| AWS EC2 | Amazon's service for renting virtual machines |
| Downtime / outage | Time when a system is not working for users |
| Single point of failure | One part whose failure takes down the whole system |
| Availability / nines | The share of time a system works; 99.99% is "four nines" (episode 4) |
| Independent failures | Failures that don't cause each other, so they rarely happen at the same moment |
| Session | What a server remembers about you between requests: your login, your cart |
| State | Data a program remembers between requests |
| Stateful server | A server that keeps users' state in its own memory |
| Stateless server | A server that keeps nothing about you between requests; any copy can serve you |
| Shared store | One place all servers read and write state, such as Redis or a database |
| Redis | An open-source, in-memory data store, used as a database, cache or message broker |
| Database | A program that stores data safely and lets you search and update it |
| Load test | Sending lots of fake traffic to measure how much one server can really handle |
| Load balancer | The part that decides which server gets each request (next episode) |
| 503 | The HTTP status code for "Service Unavailable": no server can handle the request right now |
IndexError |
Python's error for asking for a list position that doesn't exist |
KeyError |
Python's error for reading a dictionary key that isn't there |
NameError |
Python's error for using a variable name that was never set |
| List comprehension | Python's short way to build a list: [x for x in items if test] |
Summary
- Every server has a capacity. Past it, requests fail, even with perfect code.
- Vertical scaling (scale up) = a bigger machine. Simple, no code changes, but it has a ceiling, needs downtime to upgrade, and is still a single point of failure.
- Horizontal scaling (scale out) = more machines running the same code. No ceiling, and one crash is only a small dip. Independent machines multiply your nines.
- Sessions kept in one server's memory break when you scale out (
KeyError, user logged out). - Fix: stateless servers + a shared store (Redis or a database). Then scaling out is just starting one more copy.
- In practice: scale up first, then scale out.
Next episode: today we picked servers by hand. Next, we invent the thing that does it for us, millions of times a second: the load balancer. Round robin, least connections, and health checks.