These are the notes for System Design episode 4. Last episode, DNS lookups took 90 ms, or 0 ms with a cache. Here we ask what "fast" actually means, and invent the numbers every engineer uses: latency, throughput and bandwidth, why distance sets a floor on latency, how parallel workers raise throughput, how queueing makes latency explode, why averages lie (p50, p99, tail latency), and availability nines. Every Python demo below is self-contained and runs as-is; the outputs shown are real. Timing demos vary by a few milliseconds per run, so your numbers will be close but not identical.
The six problems
| Problem | What it teaches |
|---|---|
| "The app is slow, buy faster internet!" | Bandwidth is not latency |
| Distance you can't code away | The speed of light sets a floor |
| One chaiwala, a long line | Throughput and parallelism |
| Rush hour | Queueing |
| "The average is fine", but users are angry | p50 and p99 |
| How often is it even up? | Availability nines |
Latency, throughput, bandwidth: the Mumbai local train
| On the local train (Churchgate → Borivali) | On the network | Unit |
|---|---|---|
| Journey time of one trip | Latency: time for ONE request | milliseconds (ms) |
| People who reach Borivali per hour | Throughput: work done per second | requests per second |
| Coaches per train (12-car vs 15-car) | Bandwidth: max data per second the pipe can carry | megabits per second (Mbps) |
Add three more coaches and more people get home every hour. But does any single person reach Borivali one minute sooner? No.
More coaches ≠ a faster trip. Bigger is not the same as faster.
Demo 1: does a faster internet plan help?
Total time to load something = latency (the trip) + size / bandwidth (the transfer).
# total time = latency (distance) + size / bandwidth
def load_time_ms(size_kb, latency_ms, mbps):
transfer_ms = size_kb * 8 / (mbps * 1000) * 1000
return latency_ms + transfer_ms
for mbps in [10, 100, 1000]:
small = load_time_ms(2, 200, mbps) # a 2 KB API reply
big = load_time_ms(50_000, 200, mbps) # a 50 MB video file
print(f"{mbps:>4} Mbps | 2 KB: {small:7.1f} ms | 50 MB: {big:8.1f} ms")
Output:
10 Mbps | 2 KB: 201.6 ms | 50 MB: 40200.0 ms
100 Mbps | 2 KB: 200.2 ms | 50 MB: 4200.0 ms
1000 Mbps | 2 KB: 200.0 ms | 50 MB: 600.0 ms
size_kb * 8converts kilobytes to kilobits;mbps * 1000is kilobits per second;* 1000turns seconds into ms.- Big file (50 MB video): about 40 s → 4.2 s → 0.6 s. A faster plan is a huge win.
- Small reply (2 KB, the kind your apps make all day): 201.6 → 200.2 → 200 ms. A hundred times the bandwidth, barely faster.
For small requests almost all the time is the trip itself, the latency, and bandwidth does nothing for the trip.
Why the trip takes time: distance
- Data travels as light inside glass fibre at roughly 203,000 km per second, about two thirds of light's speed in a vacuum.
- TeleGeography, which maps the world's internet cables, gives a rule of thumb: every 1,000 km of fibre adds roughly 10 ms of delay.
- A server 10,000 km from your users costs about 100 ms on every trip, before your code even starts. You can't optimise that away: it's physics.
The fixes are about the trip:
- Move the server closer. TeleGeography notes that a data centre in Singapore can reach users in India within about 70 ms.
- Make fewer trips, which is exactly what the DNS cache did last episode.
Problem: one chaiwala, a long line
A chai stall has one chaiwala. Each cup takes 50 seconds: the latency of one order. College lets out and 20 students arrive at once. He still makes one cup at a time, so the last student waits more than 16 minutes for a 50-second chai. The chai didn't get slower; the stall can't finish enough cups per minute. That is a throughput problem.
Demo 2: one worker
import time
def handle(order):
time.sleep(0.05) # each order takes 50 ms
return order
start = time.perf_counter()
for order in range(20):
handle(order)
total = time.perf_counter() - start
print(f"latency per order: 50 ms")
print(f"20 orders took : {total*1000:.0f} ms")
print(f"throughput : {20/total:.0f} orders/second")
Output:
latency per order: 50 ms
20 orders took : 1005 ms
throughput : 20 orders/second
time.perf_counter() is a high-resolution clock for measuring elapsed time. 20 orders took about one second, so throughput is 20 orders per second, no matter how fast customers arrive. If a thousand people order in one second, most of them wait.
The fix: hire more chaiwalas (parallel workers)
| At the chai stall | In a server |
|---|---|
| 10 chaiwalas make 10 cups at the same time | Parallel workers: threads, processes, servers |
| Each cup still takes 50 s | Latency: same |
| 10x the cups per minute | Throughput: up |
Demo 3: ten workers
import time
from concurrent.futures import ThreadPoolExecutor
def handle(order):
time.sleep(0.05) # each order still takes 50 ms
return order
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=10) as pool:
list(pool.map(handle, range(20)))
total = time.perf_counter() - start
print(f"latency per order: 50 ms")
print(f"20 orders took : {total*1000:.0f} ms")
print(f"throughput : {20/total:.0f} orders/second")
Output:
latency per order: 50 ms
20 orders took : 103 ms
throughput : 194 orders/second
ThreadPoolExecutor(max_workers=10)creates 10 worker threads;pool.maphands the 20 orders out to them.- 20 orders in about 100 ms instead of 1,000. Throughput jumped from 20 to nearly 200 orders per second.
- Each order still took 50 ms. We didn't make anything faster; we did more at once.
Problem: rush hour (queueing)
A stall can't hire people for ever. With 4 chaiwalas and 4 students, everyone is served right away. With 40 students, 36 stand in a line: a queue. From the student's side, the latency they feel is waiting + cooking.
Demo 4: queueing makes latency explode
import time
from concurrent.futures import ThreadPoolExecutor
def handle(sent_at):
time.sleep(0.05) # 50 ms of real work
return (time.perf_counter() - sent_at) * 1000
with ThreadPoolExecutor(max_workers=4) as pool: # only 4 cooks
for rush in [4, 40]:
now = time.perf_counter()
waits = list(pool.map(handle, [now] * rush))
first, last = waits[0], waits[-1]
print(f"{rush:>2} orders: first {first:3.0f} ms, last {last:3.0f} ms")
Output:
4 orders: first 51 ms, last 53 ms
40 orders: first 50 ms, last 509 ms
- Each order records how long it took from when it was sent until it was done: waiting plus work.
- 4 orders on 4 workers: first and last both about 50 ms.
- 40 orders on 4 workers: the first still takes 50 ms, but the last takes about 500 ms: 10 rounds of 4. Ten times worse, and the work itself didn't change.
The key link between the two numbers: below your throughput limit, latency is steady. Push past it, requests queue up and latency explodes. That is why sites crawl during a big sale or a Tatkal booking rush.
Latency vs throughput on one page
| Term | Meaning | How to improve it |
|---|---|---|
| Latency | Time for ONE request (ms) | Get closer, make fewer trips |
| Throughput | Requests finished per second | More workers in parallel |
| Bandwidth | Max data per second (Mbps) | A bigger pipe (helps big transfers) |
Past max throughput → a queue forms → latency explodes.
For a sense of scale: UPI handled 24.07 billion payments in September 2026, according to NPCI. That is about 802 million a day, or more than 9,000 every second on average (802 million / 86,400 s ≈ 9,300).
Problem: "the average is fine"
The dashboard shows an average response time of 40 ms. Looks great. But users keep complaining that the app hangs. The average is hiding very different experiences. Instead, sort every request from fastest to slowest and look at the slow end.
Demo 5: averages lie, percentiles don't
import statistics
# 100 requests: 98 fast ones, 2 slow ones (a busy disk, a GC pause...)
times_ms = [20] * 98 + [1000, 1000]
print("average:", statistics.mean(times_ms), "ms")
cuts = statistics.quantiles(times_ms, n=100)
print("p50 :", cuts[49], "ms")
print("p99 :", cuts[98], "ms")
# one page asks 100 servers; each is slow 1 time in 100
print(f"slow pages: {(1 - 0.99 ** 100) * 100:.0f}%")
Output:
average: 39.6 ms
p50 : 20.0 ms
p99 : 1000.0 ms
slow pages: 63%
statistics.quantiles(times_ms, n=100)cuts the sorted data into 100 pieces and returns the 99 cut points;cuts[49]is the 50th percentile andcuts[98]the 99th.- Average 39.6 ms: a number no single user actually felt (98 felt 20 ms, 2 felt 1,000 ms).
- p50 (median) = 20 ms: half of requests are faster than this.
- p99 = 1000 ms: the slowest 1 in 100 requests take a full second. That is the user who says "it hangs".
Fan-out: the tail becomes everyone's problem
One page often asks many servers at once and waits for all of them. If it waits for 100 servers, and each is slow just 1 time in 100, then the chance that at least one is slow is 1 − 0.99¹⁰⁰ ≈ 63%. The same number appears in The Tail at Scale by Google's Jeff Dean and Luiz Barroso. At scale, the rare slow request becomes everyone's problem. That is why engineers talk about p99 and tail latency, not the average.
Percentiles
| Percentile | Meaning |
|---|---|
| p50 (median) | Half of requests are faster: the typical user |
| p95 | 95 in 100 requests are faster |
| p99 | 99 in 100 are faster: the tail (the slowest 1%) |
A real speed goal doesn't say "average 40 ms". It says something like "p99 under 300 ms".
Problem: how often is it up? (availability)
A late train is high latency: slow, but you still get home. A cancelled train is worse: you don't get home at all. The share of time a system actually works is its availability. Running 999 days out of 1,000 is 99.9%, which engineers call three nines.
From the Google SRE book's availability table:
| Availability | Name | Allowed downtime per year |
|---|---|---|
| 99% | two nines | 3.65 days |
| 99.9% | three nines | 8.76 hours |
| 99.99% | four nines | 52.6 minutes |
| 99.999% | five nines | 5.26 minutes |
Each extra nine is ten times less downtime, and usually much more cost: more servers, more places, more care.
Glossary
| Term | Meaning |
|---|---|
| Latency | Time for one request, usually in ms |
| Throughput | How many requests (or units of work) finish per second |
| Bandwidth | The maximum data per second a link can carry, in Mbps |
| Propagation delay | Time for the signal to travel the distance (≈10 ms per 1,000 km of fibre) |
| Parallelism | Doing several pieces of work at the same time (threads, processes, servers) |
| Queue | Requests waiting for a free worker |
| Percentile (pXX) | The value below which XX% of measurements fall |
| p50 / median | The typical request |
| p99 / tail latency | The slow end: the slowest 1% of requests |
| Fan-out | One request that calls many servers and waits for all of them |
| Availability | The share of time a system works |
| Nines | Shorthand for availability: 99.9% = three nines |
Common mistakes (and the real errors)
| Mistake | Code | Error | Fix |
|---|---|---|---|
| Reporting before measuring | statistics.mean(times_ms) with times_ms = [] |
statistics.StatisticsError: mean requires at least one data point |
Check the list isn't empty before you report |
| Dividing by a zero time | print(20 / total) with total = 0 |
ZeroDivisionError: division by zero |
Throughput = work / time: time a real run with time.perf_counter() |
Using a pool after the with block |
pool.submit(handle, 1) after the block |
RuntimeError: cannot schedule new futures after shutdown |
Submit work inside the with block |
The third one in full:
import time
from concurrent.futures import ThreadPoolExecutor
def handle(order):
time.sleep(0.05)
return order
with ThreadPoolExecutor(max_workers=10) as pool:
list(pool.map(handle, range(20)))
pool.submit(handle, 1)
Traceback (most recent call last):
File "main.py", line 10, in <module>
pool.submit(handle, 1)
~~~~~~~~~~~^^^^^^^^^^^
File ".../concurrent/futures/thread.py", line 171, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Misconceptions
- "Faster internet makes every app faster." It helps big transfers; small requests are dominated by latency.
- "More workers make each request faster." They raise throughput; each request still takes the same time.
- "Average response time tells you how users feel." It hides the tail. Use p50 and p99.
- "Slow and down are the same." Slow is a latency problem; down is an availability problem.
Practice
- In Demo 4, send 40 orders with
max_workersset to 4, 8 and 40. Predict the last order's wait each time, then run it. - Add a p95 line to Demo 5. What does it print, and what does that tell you?
- Using ≈10 ms per 1,000 km, estimate the latency floor for 1,400 km, 5,000 km and 10,000 km.
- How many minutes of downtime per year does 99.95% allow? Write a short Python snippet to check the table above.
- A page waits for 10 servers, each slow 1 time in 100. What percentage of pages are slow?
Solutions and hints
1. Prediction: 40 orders / workers = rounds, each about 50 ms. So about 500, 250 and 50 ms.
import time
from concurrent.futures import ThreadPoolExecutor
def handle(sent_at):
time.sleep(0.05)
return (time.perf_counter() - sent_at) * 1000
for workers in [4, 8, 40]:
with ThreadPoolExecutor(max_workers=workers) as pool:
now = time.perf_counter()
waits = list(pool.map(handle, [now] * 40))
print(f"{workers:>2} workers, 40 orders: last {waits[-1]:3.0f} ms")
4 workers, 40 orders: last 503 ms
8 workers, 40 orders: last 253 ms
40 workers, 40 orders: last 66 ms
With 40 workers it is a little above 50 ms because starting 40 threads takes time too.
2. Add print("p95 :", cuts[94], "ms"). It prints p95 : 20.0 ms. Only 2% of requests are slow, so even p95 hides them: you need p99 to see this tail.
3. About 14 ms, 50 ms and 100 ms for each trip. Real routes are longer than straight lines, so actual numbers are higher.
4.
year_min = 365 * 24 * 60
for a in [99.0, 99.9, 99.95, 99.99, 99.999]:
down = (1 - a / 100) * year_min
print(f"{a}% -> {down:8.1f} minutes down per year")
99.0% -> 5256.0 minutes down per year
99.9% -> 525.6 minutes down per year
99.95% -> 262.8 minutes down per year
99.99% -> 52.6 minutes down per year
99.999% -> 5.3 minutes down per year
99.95% allows about 263 minutes (about 4.4 hours). 5,256 minutes is 3.65 days and 525.6 minutes is 8.76 hours, matching the table.
5. (1 - 0.99 ** 10) * 100 ≈ 10%. With 100 servers it was 63%: the more servers a page waits for, the more the tail matters.
Quick quiz
A server handles each request in 50 ms, one at a time. You add 9 more workers. What happens to a single request's latency?
- A. It drops to 5 ms
- B. It stays about 50 ms
- C. It grows to 500 ms
- D. It becomes 0 ms
Answer
B, it stays about 50 ms. Parallel workers raise throughput (about 10x more requests per second), but each request still takes the same time, just like the chai stall.
Interview questions
- What is the difference between latency, throughput and bandwidth? Give a real-life analogy.
- Why doesn't more bandwidth speed up small API calls?
- What sets the lower limit on latency between two cities? How do you reduce it? (Move closer, e.g. regions or edge caches; make fewer round trips.)
- How do you increase throughput? Does that change latency?
- What happens to latency when load goes past a system's maximum throughput? (Queues form and latency explodes.)
- Why do teams set goals on p99 instead of the average? What is tail latency, and why does fan-out make it worse?
- How much downtime per year does 99.9% availability allow? 99.99%?
Summary
- Latency = time for one request; throughput = requests finished per second; bandwidth = max data per second.
- Bandwidth helps big files, not small requests. Distance sets a floor: about 10 ms per 1,000 km of fibre.
- Parallel workers raise throughput, not latency.
- Past the throughput limit, queues make latency explode.
- Report p50 and p99, not averages; fan-out turns the rare slow request into a common one.
- Availability is measured in nines; each extra nine is 10x less downtime.
Next episode: today we added workers inside one program. What happens when one machine isn't enough? Scaling, vertical vs horizontal: one bigger cook, or more cooks?