SyntaxPilot
Courses/System Design/Lesson 4

System Design · Lesson 4 of 40 · 12 min

Latency vs throughput

scheduled

These are the notes for System Design episode 4. Last episode, DNS lookups took 90 ms, or 0 ms with a cache. Here we ask what "fast" actually means, and invent the numbers every engineer uses: latency, throughput and bandwidth, why distance sets a floor on latency, how parallel workers raise throughput, how queueing makes latency explode, why averages lie (p50, p99, tail latency), and availability nines. Every Python demo below is self-contained and runs as-is; the outputs shown are real. Timing demos vary by a few milliseconds per run, so your numbers will be close but not identical.

The six problems

Problem What it teaches
"The app is slow, buy faster internet!" Bandwidth is not latency
Distance you can't code away The speed of light sets a floor
One chaiwala, a long line Throughput and parallelism
Rush hour Queueing
"The average is fine", but users are angry p50 and p99
How often is it even up? Availability nines

Latency, throughput, bandwidth: the Mumbai local train

On the local train (Churchgate → Borivali) On the network Unit
Journey time of one trip Latency: time for ONE request milliseconds (ms)
People who reach Borivali per hour Throughput: work done per second requests per second
Coaches per train (12-car vs 15-car) Bandwidth: max data per second the pipe can carry megabits per second (Mbps)

Add three more coaches and more people get home every hour. But does any single person reach Borivali one minute sooner? No.

More coaches ≠ a faster trip. Bigger is not the same as faster.

Demo 1: does a faster internet plan help?

Total time to load something = latency (the trip) + size / bandwidth (the transfer).

# total time = latency (distance) + size / bandwidth
def load_time_ms(size_kb, latency_ms, mbps):
    transfer_ms = size_kb * 8 / (mbps * 1000) * 1000
    return latency_ms + transfer_ms

for mbps in [10, 100, 1000]:
    small = load_time_ms(2, 200, mbps)        # a 2 KB API reply
    big = load_time_ms(50_000, 200, mbps)     # a 50 MB video file
    print(f"{mbps:>4} Mbps | 2 KB: {small:7.1f} ms | 50 MB: {big:8.1f} ms")

Output:

  10 Mbps | 2 KB:   201.6 ms | 50 MB:  40200.0 ms
 100 Mbps | 2 KB:   200.2 ms | 50 MB:   4200.0 ms
1000 Mbps | 2 KB:   200.0 ms | 50 MB:    600.0 ms
  • size_kb * 8 converts kilobytes to kilobits; mbps * 1000 is kilobits per second; * 1000 turns seconds into ms.
  • Big file (50 MB video): about 40 s → 4.2 s → 0.6 s. A faster plan is a huge win.
  • Small reply (2 KB, the kind your apps make all day): 201.6 → 200.2 → 200 ms. A hundred times the bandwidth, barely faster.

For small requests almost all the time is the trip itself, the latency, and bandwidth does nothing for the trip.

Why the trip takes time: distance

  • Data travels as light inside glass fibre at roughly 203,000 km per second, about two thirds of light's speed in a vacuum.
  • TeleGeography, which maps the world's internet cables, gives a rule of thumb: every 1,000 km of fibre adds roughly 10 ms of delay.
  • A server 10,000 km from your users costs about 100 ms on every trip, before your code even starts. You can't optimise that away: it's physics.

The fixes are about the trip:

  1. Move the server closer. TeleGeography notes that a data centre in Singapore can reach users in India within about 70 ms.
  2. Make fewer trips, which is exactly what the DNS cache did last episode.

Problem: one chaiwala, a long line

A chai stall has one chaiwala. Each cup takes 50 seconds: the latency of one order. College lets out and 20 students arrive at once. He still makes one cup at a time, so the last student waits more than 16 minutes for a 50-second chai. The chai didn't get slower; the stall can't finish enough cups per minute. That is a throughput problem.

Demo 2: one worker

import time

def handle(order):
    time.sleep(0.05)              # each order takes 50 ms
    return order

start = time.perf_counter()
for order in range(20):
    handle(order)
total = time.perf_counter() - start

print(f"latency per order: 50 ms")
print(f"20 orders took   : {total*1000:.0f} ms")
print(f"throughput       : {20/total:.0f} orders/second")

Output:

latency per order: 50 ms
20 orders took   : 1005 ms
throughput       : 20 orders/second

time.perf_counter() is a high-resolution clock for measuring elapsed time. 20 orders took about one second, so throughput is 20 orders per second, no matter how fast customers arrive. If a thousand people order in one second, most of them wait.

The fix: hire more chaiwalas (parallel workers)

At the chai stall In a server
10 chaiwalas make 10 cups at the same time Parallel workers: threads, processes, servers
Each cup still takes 50 s Latency: same
10x the cups per minute Throughput: up

Demo 3: ten workers

import time
from concurrent.futures import ThreadPoolExecutor

def handle(order):
    time.sleep(0.05)              # each order still takes 50 ms
    return order

start = time.perf_counter()
with ThreadPoolExecutor(max_workers=10) as pool:
    list(pool.map(handle, range(20)))
total = time.perf_counter() - start

print(f"latency per order: 50 ms")
print(f"20 orders took   : {total*1000:.0f} ms")
print(f"throughput       : {20/total:.0f} orders/second")

Output:

latency per order: 50 ms
20 orders took   : 103 ms
throughput       : 194 orders/second
  • ThreadPoolExecutor(max_workers=10) creates 10 worker threads; pool.map hands the 20 orders out to them.
  • 20 orders in about 100 ms instead of 1,000. Throughput jumped from 20 to nearly 200 orders per second.
  • Each order still took 50 ms. We didn't make anything faster; we did more at once.

Problem: rush hour (queueing)

A stall can't hire people for ever. With 4 chaiwalas and 4 students, everyone is served right away. With 40 students, 36 stand in a line: a queue. From the student's side, the latency they feel is waiting + cooking.

Demo 4: queueing makes latency explode

import time
from concurrent.futures import ThreadPoolExecutor

def handle(sent_at):
    time.sleep(0.05)                       # 50 ms of real work
    return (time.perf_counter() - sent_at) * 1000

with ThreadPoolExecutor(max_workers=4) as pool:   # only 4 cooks
    for rush in [4, 40]:
        now = time.perf_counter()
        waits = list(pool.map(handle, [now] * rush))
        first, last = waits[0], waits[-1]
        print(f"{rush:>2} orders: first {first:3.0f} ms, last {last:3.0f} ms")

Output:

 4 orders: first  51 ms, last  53 ms
40 orders: first  50 ms, last 509 ms
  • Each order records how long it took from when it was sent until it was done: waiting plus work.
  • 4 orders on 4 workers: first and last both about 50 ms.
  • 40 orders on 4 workers: the first still takes 50 ms, but the last takes about 500 ms: 10 rounds of 4. Ten times worse, and the work itself didn't change.

The key link between the two numbers: below your throughput limit, latency is steady. Push past it, requests queue up and latency explodes. That is why sites crawl during a big sale or a Tatkal booking rush.

Latency vs throughput on one page

Term Meaning How to improve it
Latency Time for ONE request (ms) Get closer, make fewer trips
Throughput Requests finished per second More workers in parallel
Bandwidth Max data per second (Mbps) A bigger pipe (helps big transfers)

Past max throughput → a queue forms → latency explodes.

For a sense of scale: UPI handled 24.07 billion payments in September 2026, according to NPCI. That is about 802 million a day, or more than 9,000 every second on average (802 million / 86,400 s ≈ 9,300).

Problem: "the average is fine"

The dashboard shows an average response time of 40 ms. Looks great. But users keep complaining that the app hangs. The average is hiding very different experiences. Instead, sort every request from fastest to slowest and look at the slow end.

Demo 5: averages lie, percentiles don't

import statistics

# 100 requests: 98 fast ones, 2 slow ones (a busy disk, a GC pause...)
times_ms = [20] * 98 + [1000, 1000]

print("average:", statistics.mean(times_ms), "ms")
cuts = statistics.quantiles(times_ms, n=100)
print("p50    :", cuts[49], "ms")
print("p99    :", cuts[98], "ms")

# one page asks 100 servers; each is slow 1 time in 100
print(f"slow pages: {(1 - 0.99 ** 100) * 100:.0f}%")

Output:

average: 39.6 ms
p50    : 20.0 ms
p99    : 1000.0 ms
slow pages: 63%
  • statistics.quantiles(times_ms, n=100) cuts the sorted data into 100 pieces and returns the 99 cut points; cuts[49] is the 50th percentile and cuts[98] the 99th.
  • Average 39.6 ms: a number no single user actually felt (98 felt 20 ms, 2 felt 1,000 ms).
  • p50 (median) = 20 ms: half of requests are faster than this.
  • p99 = 1000 ms: the slowest 1 in 100 requests take a full second. That is the user who says "it hangs".

Fan-out: the tail becomes everyone's problem

One page often asks many servers at once and waits for all of them. If it waits for 100 servers, and each is slow just 1 time in 100, then the chance that at least one is slow is 1 − 0.99¹⁰⁰ ≈ 63%. The same number appears in The Tail at Scale by Google's Jeff Dean and Luiz Barroso. At scale, the rare slow request becomes everyone's problem. That is why engineers talk about p99 and tail latency, not the average.

Percentiles

Percentile Meaning
p50 (median) Half of requests are faster: the typical user
p95 95 in 100 requests are faster
p99 99 in 100 are faster: the tail (the slowest 1%)

A real speed goal doesn't say "average 40 ms". It says something like "p99 under 300 ms".

Problem: how often is it up? (availability)

A late train is high latency: slow, but you still get home. A cancelled train is worse: you don't get home at all. The share of time a system actually works is its availability. Running 999 days out of 1,000 is 99.9%, which engineers call three nines.

From the Google SRE book's availability table:

Availability Name Allowed downtime per year
99% two nines 3.65 days
99.9% three nines 8.76 hours
99.99% four nines 52.6 minutes
99.999% five nines 5.26 minutes

Each extra nine is ten times less downtime, and usually much more cost: more servers, more places, more care.

Glossary

Term Meaning
Latency Time for one request, usually in ms
Throughput How many requests (or units of work) finish per second
Bandwidth The maximum data per second a link can carry, in Mbps
Propagation delay Time for the signal to travel the distance (≈10 ms per 1,000 km of fibre)
Parallelism Doing several pieces of work at the same time (threads, processes, servers)
Queue Requests waiting for a free worker
Percentile (pXX) The value below which XX% of measurements fall
p50 / median The typical request
p99 / tail latency The slow end: the slowest 1% of requests
Fan-out One request that calls many servers and waits for all of them
Availability The share of time a system works
Nines Shorthand for availability: 99.9% = three nines

Common mistakes (and the real errors)

Mistake Code Error Fix
Reporting before measuring statistics.mean(times_ms) with times_ms = [] statistics.StatisticsError: mean requires at least one data point Check the list isn't empty before you report
Dividing by a zero time print(20 / total) with total = 0 ZeroDivisionError: division by zero Throughput = work / time: time a real run with time.perf_counter()
Using a pool after the with block pool.submit(handle, 1) after the block RuntimeError: cannot schedule new futures after shutdown Submit work inside the with block

The third one in full:

import time
from concurrent.futures import ThreadPoolExecutor

def handle(order):
    time.sleep(0.05)
    return order

with ThreadPoolExecutor(max_workers=10) as pool:
    list(pool.map(handle, range(20)))
pool.submit(handle, 1)
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    pool.submit(handle, 1)
    ~~~~~~~~~~~^^^^^^^^^^^
  File ".../concurrent/futures/thread.py", line 171, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

Misconceptions

  • "Faster internet makes every app faster." It helps big transfers; small requests are dominated by latency.
  • "More workers make each request faster." They raise throughput; each request still takes the same time.
  • "Average response time tells you how users feel." It hides the tail. Use p50 and p99.
  • "Slow and down are the same." Slow is a latency problem; down is an availability problem.

Practice

  1. In Demo 4, send 40 orders with max_workers set to 4, 8 and 40. Predict the last order's wait each time, then run it.
  2. Add a p95 line to Demo 5. What does it print, and what does that tell you?
  3. Using ≈10 ms per 1,000 km, estimate the latency floor for 1,400 km, 5,000 km and 10,000 km.
  4. How many minutes of downtime per year does 99.95% allow? Write a short Python snippet to check the table above.
  5. A page waits for 10 servers, each slow 1 time in 100. What percentage of pages are slow?
Solutions and hints

1. Prediction: 40 orders / workers = rounds, each about 50 ms. So about 500, 250 and 50 ms.

import time
from concurrent.futures import ThreadPoolExecutor

def handle(sent_at):
    time.sleep(0.05)
    return (time.perf_counter() - sent_at) * 1000

for workers in [4, 8, 40]:
    with ThreadPoolExecutor(max_workers=workers) as pool:
        now = time.perf_counter()
        waits = list(pool.map(handle, [now] * 40))
        print(f"{workers:>2} workers, 40 orders: last {waits[-1]:3.0f} ms")
 4 workers, 40 orders: last 503 ms
 8 workers, 40 orders: last 253 ms
40 workers, 40 orders: last  66 ms

With 40 workers it is a little above 50 ms because starting 40 threads takes time too.

2. Add print("p95 :", cuts[94], "ms"). It prints p95 : 20.0 ms. Only 2% of requests are slow, so even p95 hides them: you need p99 to see this tail.

3. About 14 ms, 50 ms and 100 ms for each trip. Real routes are longer than straight lines, so actual numbers are higher.

4.

year_min = 365 * 24 * 60
for a in [99.0, 99.9, 99.95, 99.99, 99.999]:
    down = (1 - a / 100) * year_min
    print(f"{a}% -> {down:8.1f} minutes down per year")
99.0% ->   5256.0 minutes down per year
99.9% ->    525.6 minutes down per year
99.95% ->    262.8 minutes down per year
99.99% ->     52.6 minutes down per year
99.999% ->      5.3 minutes down per year

99.95% allows about 263 minutes (about 4.4 hours). 5,256 minutes is 3.65 days and 525.6 minutes is 8.76 hours, matching the table.

5. (1 - 0.99 ** 10) * 100 ≈ 10%. With 100 servers it was 63%: the more servers a page waits for, the more the tail matters.

Quick quiz

A server handles each request in 50 ms, one at a time. You add 9 more workers. What happens to a single request's latency?

  • A. It drops to 5 ms
  • B. It stays about 50 ms
  • C. It grows to 500 ms
  • D. It becomes 0 ms
Answer

B, it stays about 50 ms. Parallel workers raise throughput (about 10x more requests per second), but each request still takes the same time, just like the chai stall.

Interview questions

  • What is the difference between latency, throughput and bandwidth? Give a real-life analogy.
  • Why doesn't more bandwidth speed up small API calls?
  • What sets the lower limit on latency between two cities? How do you reduce it? (Move closer, e.g. regions or edge caches; make fewer round trips.)
  • How do you increase throughput? Does that change latency?
  • What happens to latency when load goes past a system's maximum throughput? (Queues form and latency explodes.)
  • Why do teams set goals on p99 instead of the average? What is tail latency, and why does fan-out make it worse?
  • How much downtime per year does 99.9% availability allow? 99.99%?

Summary

  • Latency = time for one request; throughput = requests finished per second; bandwidth = max data per second.
  • Bandwidth helps big files, not small requests. Distance sets a floor: about 10 ms per 1,000 km of fibre.
  • Parallel workers raise throughput, not latency.
  • Past the throughput limit, queues make latency explode.
  • Report p50 and p99, not averages; fan-out turns the rare slow request into a common one.
  • Availability is measured in nines; each extra nine is 10x less downtime.

Next episode: today we added workers inside one program. What happens when one machine isn't enough? Scaling, vertical vs horizontal: one bigger cook, or more cooks?