These are the notes for System Design episode 24. ChaiNow's checkout calls a payment gateway over the network, the network blips, and the whole checkout crashes. You'll fix it with a retry, watch that retry charge Priya for her chai twice, invent the idempotency key (the way Stripe does it), see 1,000 phones turn retries into a retry storm, calm it with exponential backoff and jitter, pick a timeout from latency percentiles, and finally stop calling a dead service at all with a circuit breaker. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). The gateway, the failure rates and the timings are simulated with a fixed random seed, so you get exactly the same output every time you run them.
The problems
| Problem | What it teaches |
|---|---|
| The call fails | Network calls, transient failures, retries |
| Charged twice | Lost replies, idempotent operations, idempotency keys (Stripe's Idempotency-Key) |
| Everyone retries at once | Retry storms, exponential backoff, a cap, a maximum number of tries |
| Retries in waves | Jitter: random waits that spread the retries out |
| Waiting forever | Timeouts, chosen from latency percentiles (p99.9) |
| A dead service | Circuit breakers (closed, open, half-open), cooldown, fail fast, fallbacks |
Problem 1: the call fails
You phone the chai shop to order. Most days it works. Sometimes the call drops, because you walked into a lift. Nobody did anything wrong; the network failed for a moment. So you call again.
ChaiNow's checkout has the same problem. To take money it makes a network call: it sends a message to another computer and waits for its reply. That other computer is the payment gateway, the company that actually moves the money from the customer's card or bank to the shop (in this episode we use Stripe's documentation as the example). Between ChaiNow and the gateway there are cables, routers and Wi-Fi, and any of them can drop a message.
Here charge pretends to be the gateway. Three calls in ten, the network fails and Python raises a ConnectionError (Python's built-in error for "could not talk to the other side"). random.seed(2) fixes the random numbers, so every run fails in the same places.
import random
random.seed(2)
def charge(order_id, amount): # the payment gateway, far away
if random.random() < 0.3: # 3 in 10 calls: the network fails
raise ConnectionError("gateway did not answer")
return f"{order_id}: paid Rs {amount}"
for order_id in ["A1", "A2", "A3", "A4", "A5"]:
print(charge(order_id, 240))
Output:
A1: paid Rs 240
A2: paid Rs 240
Traceback (most recent call last):
File ".../flaky.py", line 10, in <module>
print(charge(order_id, 240))
~~~~~~^^^^^^^^^^^^^^^
File ".../flaky.py", line 6, in charge
raise ConnectionError("gateway did not answer")
ConnectionError: gateway did not answer
A1 and A2 paid. A3 crashed the whole program, and A4 and A5 never even got a chance. One short blip took down the checkout.
Failures like this, which fix themselves a moment later, are called transient failures ("transient" means short-lived). Examples: a dropped Wi-Fi packet, a server that is restarting, a gateway that is briefly overloaded and answers HTTP 503 ("Service Unavailable": "I can't take your request right now, try later"). Compare a permanent failure such as "card declined": asking again gives the same answer.
So, like the dropped phone call, let's simply try again. Trying a failed call again is a retry.
import random
random.seed(2)
def charge(order_id, amount): # the payment gateway, far away
if random.random() < 0.3: # 3 in 10 calls: the network fails
raise ConnectionError("gateway did not answer")
return f"{order_id}: paid Rs {amount}"
def charge_with_retry(order_id, amount, tries=3):
for attempt in range(1, tries + 1):
try:
return charge(order_id, amount)
except ConnectionError as e:
print(f" {order_id} attempt {attempt} failed: {e}")
raise ConnectionError(f"{order_id}: gave up after {tries} tries")
for order_id in ["A1", "A2", "A3", "A4", "A5"]:
print(charge_with_retry(order_id, 240))
Output:
A1: paid Rs 240
A2: paid Rs 240
A3 attempt 1 failed: gateway did not answer
A3 attempt 2 failed: gateway did not answer
A3: paid Rs 240
A4: paid Rs 240
A5: paid Rs 240
try/except ConnectionErrorcatches the failure instead of crashing.- The
forloop tries up totries(3) times;returnleaves as soon as one try works. - Only after 3 failures does it give up, with a clear error. A limit matters: a retry loop with no maximum can spin forever.
- A3 failed twice and the third try worked. All five orders paid, no crash. You just invented retries.
But notice what we assumed: that a failed call did nothing. Is that always true?
Problem 2: charged twice
A network call can fail in two places:
- On the way there. The request never reaches the gateway. Nothing happened; retrying is harmless.
- On the way back. The request arrives, the gateway moves the money (Priya's balance goes from 1000 to 760), and then the reply gets lost.
From the app's side, both look exactly the same: an error. So it retries, as we just taught it.
balance = {"priya": 1000}
def charge(customer, amount, lose_reply):
balance[customer] -= amount # the money moves...
if lose_reply: # ...but the reply is lost
raise ConnectionError("reply lost on the way back")
return "ok"
# the first reply gets lost, so the app retries
for attempt, lose in [(1, True), (2, False)]:
try:
print("attempt", attempt, charge("priya", 240, lose))
except ConnectionError as e:
print("attempt", attempt, "failed:", e)
print("Priya's balance:", balance["priya"], "(should be 760)")
Output:
attempt 1 failed: reply lost on the way back
attempt 2 ok
Priya's balance: 520 (should be 760)
Priya paid for one chai, twice. The app can't tell whether a failed call did nothing, or did everything and lost the reply.
Idempotent: safe to repeat
Retrying is only safe if doing something twice has the same effect as doing it once. That property is called idempotent (from Latin: "the same power").
| In real life | In an API | Safe to repeat? |
|---|---|---|
| Press the lift button 5 times: one lift comes | set_status(order, "delivered") |
Yes: idempotent |
| Pay the shopkeeper twice: you lose twice | charge(card, 240) (a POST /charge) |
No: not idempotent |
| Show your bill number: "bill 57 is already paid" | charge(card, 240) + an idempotency key |
Yes: the key makes it safe |
Stripe's API documentation notes that GET and DELETE requests "are idempotent by definition": reading something twice changes nothing, and deleting something twice still leaves it deleted. Calls with a side effect (something that changes the world: moving money, sending an SMS, adding to a cart) are the dangerous ones. The AWS summary used in this episode says it directly: "APIs with side effects aren't safe to retry unless they provide idempotency".
The idempotency key
Imagine every payment carries a bill number. If the shopkeeper sees bill 57 again, he says "already paid, here's your receipt" and takes no money. That bill number is an idempotency key: a unique id the client makes once per operation and sends with every retry of it. The server remembers each key's result; when the key comes again, it replays (returns again) the saved result instead of doing the work again.
import uuid
balance = {"priya": 1000}
done = {} # key -> saved result
def charge(key, customer, amount, lose_reply):
if key in done: # seen this key before:
return done[key] + " (replayed)" # same answer, no new charge
balance[customer] -= amount
done[key] = f"charged {amount}"
if lose_reply:
raise ConnectionError("reply lost on the way back")
return done[key]
key = str(uuid.uuid4()) # made ONCE per payment
for attempt, lose in [(1, True), (2, False), (3, False)]:
try:
print("attempt", attempt, charge(key, "priya", 240, lose))
except ConnectionError as e:
print("attempt", attempt, "failed:", e)
print("Priya's balance:", balance["priya"])
Output:
attempt 1 failed: reply lost on the way back
attempt 2 charged 240 (replayed)
attempt 3 charged 240 (replayed)
Priya's balance: 760
doneis the server's memory: key → saved result. In a real system it lives in a database, not a Python dictionary.uuid.uuid4()makes a random UUID (a 128-bit id, so random that two clients will practically never make the same one; episode 23). It is made once, before the loop, so every retry sends the same key.- Attempt 1 moved the money and lost the reply. Attempts 2 and 3 got the saved answer, marked
(replayed). Priya's balance is 760: charged once.
How Stripe does it
Stripe's documentation describes idempotency as "safely retrying requests without accidentally performing the same operation twice", and it works like our demo:
- The client sends the key in an HTTP header named
Idempotency-Key(a header is a labelled extra line sent along with a request). - "We suggest using V4 UUIDs" (exactly what
uuid.uuid4()makes), and keys can be up to 255 characters. - In API v1, Stripe saves the status code and body of the first request with that key. "Retries with the same key return the saved response, including 500 errors." (HTTP 500, "Internal Server Error", means "something broke on the server".) So a key's first answer is final, even when it was an error.
- Keys can be removed "after they're at least 24 hours old", so a retry should come within that window. (Our
donedictionary keeps every key forever; a real server must clean up.) - Stripe compares the incoming parameters with the first request's and "errors if they're not the same". Same key + different amount is a bug in the client, not a retry (Practice 1 builds this).
You just invented idempotency keys.
Problem 3: everyone retries at once
Tatkal booking opens at 10 a.m. The site slows down. So everyone hits refresh, again and again, and every refresh is one more request on a site that's already struggling. Now it's fully down: the retries finished it off. That is a retry storm: retries from many clients piling extra load onto a service that's already failing, keeping it down.
Our retry loop does the same to the gateway. Here 1,000 phones all fail at the same moment (t = 0) because the gateway went down; it comes back after 10 seconds. simulate counts every retry in 100 millisecond slots (time buckets), and wait(attempt) says how long a phone waits before its next try.
import random
random.seed(42)
CLIENTS, DOWN_UNTIL = 1000, 10 # all fail at t=0; gateway back at 10 s
def simulate(name, wait): # wait(attempt) -> seconds to wait
load = {} # 100 ms slot -> retries in that slot
for _ in range(CLIENTS):
t, attempt = 0.0, 0
while t < DOWN_UNTIL: # failed: wait, then retry
attempt += 1
t += wait(attempt)
slot = int(t * 10)
load[slot] = load.get(slot, 0) + 1
print(f"{name:<16}{sum(load.values()):>7,} retries,",
f"busiest 100 ms: {max(load.values()):,}")
simulate("every 0.5 s", lambda a: 0.5)
simulate("backoff", lambda a: min(2 ** a, 8))
simulate("backoff+jitter", lambda a: random.uniform(0, min(2 ** a, 8)))
Output:
every 0.5 s 20,000 retries, busiest 100 ms: 1,000
backoff 3,000 retries, busiest 100 ms: 1,000
backoff+jitter 4,411 retries, busiest 100 ms: 88
Three strategies:
- Every 0.5 s: each phone retries 20 times in 10 seconds: 1,000 × 20 = 20,000 retries on a service that is already down.
- Exponential backoff: wait longer after each failure, doubling each time: 2, then 4, then 8 seconds.
min(2 ** a, 8)puts a cap (an upper limit) of 8 s on the wait, so it doesn't grow to hours. Each phone now retries at t = 2, 6 and 14: only 3 retries each, 3,000 in total. But look at the busiest 100 ms: still all 1,000 phones, because they all failed together and all wait exactly the same amount. The retries arrive in waves. - Backoff + jitter: jitter means adding randomness. Each phone waits a random time between 0 and the backoff value:
random.uniform(0, min(2 ** a, 8)). The busiest 100 ms drops from 1,000 to 88. Same phones, spread out instead of arriving as a wave. (The total rises to 4,411 because some random waits are short, but no single moment is crushed.)
The AWS Builders' Library article "Timeouts, retries and backoff with jitter" (as summarised by Lumigo) recommends exactly this pair: "inject jitter to the backoff", with a wait of randint(0, base * 2 ** attempt). That is the same formula as our demo, with base = 1 second. You just invented backoff with jitter.
Retry rules
| Rule | Why |
|---|---|
| Retry only transient errors (a timeout, a 503), not "card declined" | A permanent error gives the same answer every time (see Common mistakes) |
| Retry only idempotent calls, or calls that carry an idempotency key | Otherwise a lost reply becomes a double charge |
| Back off, add jitter, and stop after a few tries | Otherwise retries become a storm |
| Retry at one layer | 3 layers × 3 tries = 27 calls for one tap (demo below) |
| A retry adds load: spend it carefully | The failing service is the one that can least afford extra work |
The layers rule, run for real. The app calls the order service, which calls the payment service, which calls the gateway, and each of them retries 3 times:
calls = 0
def gateway(): # down: every call fails
global calls
calls += 1
raise ConnectionError("down")
def with_retry(fn, tries=3): # each layer retries 3 times
def wrapped():
for attempt in range(tries):
try:
return fn()
except ConnectionError:
if attempt == tries - 1:
raise
return wrapped
payments = with_retry(gateway) # layer 3: payment service
orders = with_retry(payments) # layer 2: order service
app = with_retry(orders) # layer 1: the app
try:
app() # ONE tap on "Pay"
except ConnectionError:
print("1 tap ->", calls, "calls to the gateway")
Output:
1 tap -> 27 calls to the gateway
Each layer multiplies: 3 × 3 × 3 = 27. The AWS summary warns that "retrying at all layers ... multiplies the number of retries". Pick one layer (usually the one closest to the user, or the one closest to the failing service) to do the retrying.
Problem 4: waiting forever (timeouts)
Before the circuit breaker, every network call needs a timeout: the longest you'll wait for a reply before giving up and treating the call as failed. Without one, a gateway that never answers keeps your request (and the customer, and a slot on your server) waiting forever.
A real timeout, run for real. The "gateway" is stuck and answers only after 3 seconds; we wait at most 0.5 s:
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError
def gateway(): # stuck: answers after 3 s
time.sleep(3)
return "paid"
pool = ThreadPoolExecutor()
start = time.perf_counter()
future = pool.submit(gateway)
try:
print(future.result(timeout=0.5)) # wait at most 0.5 s
except TimeoutError:
took = time.perf_counter() - start
print(f"gave up after {took:.1f} s: show 'try again'")
Output:
gave up after 0.5 s: show 'try again'
(The program only exits after the stuck call finishes in the background; real HTTP libraries take a timeout= argument and close the connection instead.) A timed-out payment is exactly the "lost reply" case: the gateway may still have charged, so a retry after a timeout needs the idempotency key.
How long should the timeout be?
Too short, and you cut off healthy calls that were just a bit slow (a false timeout). Too long, and customers stare at a spinner when the gateway is dead.
Remember percentiles from episode 4: the p99 latency is the time that 99% of calls finish under; p99.9 is the time 99.9% finish under (only 1 call in 1,000 is slower). The AWS advice (via the Lumigo summary): pick an acceptable false-timeout rate, for example 0.1%, and use "the corresponding percentile latency for the downstream as the timeout". A 0.1% false-timeout rate means the p99.9 of the service you call (the "downstream").
Simulated latencies for 100,000 past gateway calls (most quick, a few slow):
import random
random.seed(7)
# 100,000 past calls to the gateway: most are quick, a few are slow
lat = sorted(random.lognormvariate(4.5, 0.5) for _ in range(100_000))
def pct(p): # the latency p% of calls are under
return lat[int(len(lat) * p / 100) - 1]
for p in [50, 90, 99, 99.9]:
print(f"p{p:<5} {pct(p):6.0f} ms")
for timeout in [200, pct(99.9), 2000]:
false = sum(x > timeout for x in lat) / len(lat) * 100
print(f"timeout {timeout:5.0f} ms -> {false:.2f}%",
"of healthy calls cut off")
Output:
p50 90 ms
p90 171 ms
p99 288 ms
p99.9 424 ms
timeout 200 ms -> 5.59% of healthy calls cut off
timeout 424 ms -> 0.10% of healthy calls cut off
timeout 2000 ms -> 0.00% of healthy calls cut off
lognormvariategives realistic latencies: a big bump of quick calls and a long tail of slow ones.- A 200 ms timeout looks generous next to the 90 ms median, but it cuts off 5.59% of healthy calls, and every one of those becomes a retry.
- The p99.9 (424 ms) cuts off exactly the 0.1% we decided to accept.
- 2,000 ms cuts off nothing, but when the gateway is dead every checkout waits 2 full seconds before failing.
So the full chain is: a timeout, then a few retries with backoff and jitter, and a circuit breaker around all of it.
Problem 5: a dead service (circuit breaker)
Now the gateway is down for minutes, not seconds. Every checkout waits for its timeout (say 2 s) before it fails, then retries, and fails again. Thousands of customers each stare at a spinner, and our servers hold every waiting request open (each one uses memory and a connection). Soon our service is slow too, and the services that call us start timing out. One failing service dragging down the ones that depend on it is a cascading failure.
Your house has the fix. When a heater shorts, the circuit breaker in the fuse box trips and cuts the power before the damage spreads. Later you flip it back carefully; if it trips again, you wait longer.
| At home | Circuit breaker in code |
|---|---|
| A faulty heater keeps sparking | A failing service: every call times out |
| The breaker trips: power cut, the house is safe | OPEN: don't even call, fail fast |
| Later, flip it back carefully | HALF-OPEN: let one test call through |
| It stays on | CLOSED: calls flow normally again |
The names come from electricity: a closed circuit lets current flow (calls go through); an open circuit is broken (calls are blocked).
class CircuitBreaker:
def __init__(self, limit=3, cooldown=5):
self.state, self.fails, self.opened_at = "CLOSED", 0, 0
self.limit, self.cooldown = limit, cooldown
def call(self, fn, now):
if self.state == "OPEN":
if now - self.opened_at < self.cooldown:
return "fail fast: offer cash on delivery"
self.state = "HALF-OPEN" # let ONE test call through
try:
result = fn()
except ConnectionError:
self.fails += 1
if self.state == "HALF-OPEN" or self.fails >= self.limit:
self.state, self.opened_at = "OPEN", now
return "failed after a 2 s timeout"
self.state, self.fails = "CLOSED", 0
return result
def gateway(now): # down until second 10
if now < 10:
raise ConnectionError("timeout")
return "paid"
cb = CircuitBreaker()
for now in [0, 1, 2, 3, 5, 7, 9, 12, 13]:
msg = cb.call(lambda: gateway(now), now)
print(f"t={now:<3} {msg:<34} {cb.state}")
Output:
t=0 failed after a 2 s timeout CLOSED
t=1 failed after a 2 s timeout CLOSED
t=2 failed after a 2 s timeout OPEN
t=3 fail fast: offer cash on delivery OPEN
t=5 fail fast: offer cash on delivery OPEN
t=7 failed after a 2 s timeout OPEN
t=9 fail fast: offer cash on delivery OPEN
t=12 paid CLOSED
t=13 paid CLOSED
Read it line by line:
- t = 0, 1, 2 (CLOSED): three slow failures. At the third (
limit= 3, the threshold), the breaker switches to OPEN and remembers when (opened_at= 2). - t = 3, 5 (OPEN): the gateway isn't even called. The answer is instant: fail fast, with a fallback (a backup answer): "offer cash on delivery". No 2 s wait, no extra load on the gateway.
- t = 7: the cooldown (5 s, how long the breaker stays open before testing) is over, so the state becomes HALF-OPEN and one test call goes through. It fails, so the breaker opens again (
opened_at= 7). - t = 9: still cooling down: fail fast.
- t = 12: cooldown over, the test call succeeds, the breaker closes and the failure count resets. Paid. t = 13 flows normally.
Failing fast frees your servers, gives the gateway room to recover (nobody is hammering it), and gives the customer an answer now instead of a spinner. Good fallbacks include cached data, putting the work in a queue to finish later, or a sensible default; for a payment, "cash on delivery" or "please try again in a minute".
The pattern was popularized by Martin Fowler, from Michael Nygard's book "Release It!". You just invented the circuit breaker.
The three states
| State | Calls go to the service? | Moves to | When |
|---|---|---|---|
| CLOSED | Yes, all of them; failures are counted | OPEN | Failures reach the threshold |
| OPEN | No: fail fast with a fallback | HALF-OPEN | The cooldown has passed |
| HALF-OPEN | Only one test call | CLOSED / OPEN | The test call succeeds / fails |
The whole chain
- A network blip crashed checkout (ConnectionError) → retry a few times.
- A lost reply + a retry charged Priya twice (520, should be 760) → idempotency key, made once per payment; the server replays the saved result (760).
- Retrying every 0.5 s: 20,000 retries on a dead gateway → exponential backoff with a cap: 3,000.
- Backoff alone: waves of 1,000 at once → jitter: the busiest 100 ms drops to 88.
- Waiting forever → a timeout from the downstream's latency percentiles (0.1% false timeouts → p99.9).
- Dead for minutes → circuit breaker (closed → open → half-open) + a fallback: fail fast.
Common mistakes (and the real outputs)
| Mistake | Code | Output | Fix |
|---|---|---|---|
| No retry at all | charge(order_id, 240) |
ConnectionError: gateway did not answer |
Retry transient errors, a few times |
| Retrying a payment without a key | retry charge("priya", 240, ...) |
Priya's balance: 520 (should be 760) |
Idempotency key, made once per payment |
| A new key on every attempt | uuid.uuid4() inside the retry loop |
Priya's balance: 520 |
Make the key once, before the loop |
| Retrying on a fixed short timer | time.sleep(0.5) between retries |
20,000 retries, busiest 100 ms: 1,000 |
Backoff + jitter + max tries |
| Backoff without jitter | min(2 ** a, 8) |
busiest 100 ms: 1,000 |
random.uniform(0, min(2 ** a, 8)): 88 |
| Retrying permanent errors | except Exception: around "card declined" |
3 calls; the answer never changes |
Retry only transient errors (timeouts, 503) |
| Retrying at every layer | 3 layers, 3 tries each | 1 tap -> 27 calls to the gateway |
Retry at one layer |
| Timeout too short | 200 ms when p50 is 90 ms | 5.59% of healthy calls cut off |
Use the p99.9 (or your chosen percentile) |
| Reusing a key for a different request | same key, amount 250 instead of 240 | key reused with amount 250, first request was 240 (Practice 1) |
New operation = new key |
The new-key mistake, run for real. The key is made inside the loop, so the server sees two different payments:
import uuid
balance = {"priya": 1000}
done = {}
def charge(key, customer, amount, lose_reply):
if key in done:
return done[key] + " (replayed)"
balance[customer] -= amount
done[key] = f"charged {amount}"
if lose_reply:
raise ConnectionError("reply lost on the way back")
return done[key]
for attempt, lose in [(1, True), (2, False)]:
key = str(uuid.uuid4()) # WRONG: a new key every attempt
try:
print("attempt", attempt, charge(key, "priya", 240, lose))
except ConnectionError as e:
print("attempt", attempt, "failed:", e)
print("Priya's balance:", balance["priya"])
Output:
attempt 1 failed: reply lost on the way back
attempt 2 charged 240
Priya's balance: 520
No (replayed): the key protects nothing if it changes. Make it when the customer taps "Pay", store it with the order, and reuse it for every retry.
Retrying a permanent error:
def charge(card, amount):
raise ValueError("card declined: insufficient funds")
tries = 0
for attempt in range(1, 4): # retries EVERY error: wrong
try:
tries += 1
charge("4111", 240)
except Exception as e:
print("attempt", attempt, "failed:", e)
print(tries, "calls; the answer never changes")
Output:
attempt 1 failed: card declined: insufficient funds
attempt 2 failed: card declined: insufficient funds
attempt 3 failed: card declined: insufficient funds
3 calls; the answer never changes
except Exception catches everything, including errors that will never fix themselves. Catch only the transient ones (except ConnectionError, timeouts, a 503), and show "card declined" to the customer straight away.
Mini project: a resilient payment client
One client that combines everything: a timeout (0.5 s), up to 3 tries with exponential backoff and jitter, one idempotency key per payment, and a circuit breaker with a fallback. The gateway is simulated on a fake clock (no real waiting): calls normally take 0.05-0.3 s, 15% of calls are very slow (3 s, so they time out after the money moved, the lost-reply case), and the gateway is completely down from t = 30 to t = 50. One customer checks out every 3 seconds.
# Mini project: a resilient payment client (simulated, seeded)
import random, uuid
random.seed(5)
class Gateway: # the far-away payment gateway
def __init__(self):
self.done, self.charged = {}, 0
def charge(self, key, amount, now):
"""Returns (reply, seconds it took). Down from t=30 to t=50."""
if 30 <= now < 50:
return None, 99.0 # no answer at all
took = random.uniform(0.05, 0.3)
if random.random() < 0.15:
took = 3.0 # a slow call, very late
if key in self.done: # idempotency: replay
return self.done[key] + " (replayed)", took
self.charged += amount # the money moves
self.done[key] = f"charged {amount}"
return self.done[key], took
class Breaker:
def __init__(self, limit=3, cooldown=8):
self.state, self.fails, self.opened_at = "CLOSED", 0, 0
self.limit, self.cooldown = limit, cooldown
def allow(self, now):
cooled = now - self.opened_at >= self.cooldown
if self.state == "OPEN" and cooled:
self.state = "HALF-OPEN" # let one test call through
return self.state != "OPEN"
def record(self, ok, now):
if ok:
self.state, self.fails = "CLOSED", 0
return
self.fails += 1
if self.state == "HALF-OPEN" or self.fails >= self.limit:
self.state, self.opened_at = "OPEN", now
class Client:
def __init__(self, gw, timeout=0.5, tries=3, base=0.2, cap=2.0):
self.gw, self.timeout, self.tries = gw, timeout, tries
self.base, self.cap = base, cap
self.breaker = Breaker()
def pay(self, amount, now):
key = str(uuid.UUID(int=random.getrandbits(128), version=4))
for attempt in range(1, self.tries + 1): # same key each try
if not self.breaker.allow(now):
return "fallback: cash on delivery", attempt - 1, now
reply, took = self.gw.charge(key, amount, now)
if took <= self.timeout: # answered in time
self.breaker.record(True, now)
return reply, attempt, now + took
now += self.timeout # gave up waiting
self.breaker.record(False, now)
if attempt < self.tries: # backoff + jitter
now += random.uniform(0, min(self.cap,
self.base * 2 ** attempt))
return "failed: try again later", self.tries, now
gw = Gateway()
client = Client(gw)
paid = 0
for i in range(25): # one checkout every 3 s
start = i * 3
result, calls, end = client.pay(240, start)
paid += result.startswith("charged")
print(f"t={start:<3} {result:<27} calls={calls} "
f"waited {end - start:4.2f} s {client.breaker.state}")
print(f"\npaid {paid} orders; gateway took Rs {gw.charged:,}"
f" = {gw.charged // 240} charges (no double charges)")
Output:
t=0 charged 240 calls=1 waited 0.25 s CLOSED
t=3 charged 240 calls=1 waited 0.06 s CLOSED
t=6 charged 240 (replayed) calls=2 waited 0.80 s CLOSED
t=9 charged 240 calls=1 waited 0.10 s CLOSED
t=12 charged 240 calls=1 waited 0.09 s CLOSED
t=15 charged 240 (replayed) calls=2 waited 0.95 s CLOSED
t=18 charged 240 calls=1 waited 0.12 s CLOSED
t=21 charged 240 calls=1 waited 0.10 s CLOSED
t=24 charged 240 calls=1 waited 0.27 s CLOSED
t=27 charged 240 (replayed) calls=3 waited 1.80 s CLOSED
t=30 failed: try again later calls=3 waited 2.01 s OPEN
t=33 fallback: cash on delivery calls=0 waited 0.00 s OPEN
t=36 fallback: cash on delivery calls=0 waited 0.00 s OPEN
t=39 fallback: cash on delivery calls=0 waited 0.00 s OPEN
t=42 fallback: cash on delivery calls=1 waited 0.65 s OPEN
t=45 fallback: cash on delivery calls=0 waited 0.00 s OPEN
t=48 fallback: cash on delivery calls=0 waited 0.00 s OPEN
t=51 charged 240 calls=1 waited 0.24 s CLOSED
t=54 charged 240 calls=1 waited 0.14 s CLOSED
t=57 charged 240 calls=1 waited 0.24 s CLOSED
t=60 charged 240 (replayed) calls=2 waited 0.84 s CLOSED
t=63 charged 240 calls=1 waited 0.19 s CLOSED
t=66 charged 240 (replayed) calls=2 waited 1.07 s CLOSED
t=69 charged 240 calls=1 waited 0.18 s CLOSED
t=72 charged 240 calls=1 waited 0.26 s CLOSED
paid 18 orders; gateway took Rs 4,320 = 18 charges (no double charges)
What to notice:
(replayed)lines (t = 6, 15, 27, 60, 66): the first call took the money but answered too late (timeout). The retry, with the same key, got the saved answer. Without the key, each of these would be a double charge.- t = 27: two slow calls in a row; the third try succeeded. The waits between tries grow (backoff) and are random (jitter).
- t = 30: the outage starts. Three timeouts (one per try) reach the breaker's threshold of 3, so it opens. That customer waited about 2 s and got "try again later".
- t = 33 to 39: fail fast: 0 calls, 0 s waiting, a fallback straight away. The dead gateway gets no traffic at all.
- t = 42: the 8 s cooldown is over; one test call (HALF-OPEN) times out, so the breaker opens again, and the client falls back instead of retrying into a dead service.
- t = 51: the next test call succeeds and the breaker closes. Normal service resumes.
- 18 orders paid, the gateway took exactly 18 × Rs 240 = Rs 4,320: no double charges.
uuid.UUID(int=random.getrandbits(128), version=4)builds a version 4 UUID from the seeded random numbers, so the run repeats exactly. In real code useuuid.uuid4().
Extend it: make the breaker's cooldown double after each failed test (Practice 3), reject a reused key with a different amount (Practice 1), and add a retry_after so the fallback can tell the customer when to try again.
Practice
- Homework: in the IDEMP demo, retry the same key with amount 250 instead of 240. What should the server do?
- Homework: in the STORM demo, try 2,000 phones. What changes?
- Homework: in the BREAKER demo, make the cooldown grow each time the test call fails.
- Add exponential backoff and jitter to the RETRY demo's
charge_with_retry. - Comment: which of your app's calls are idempotent? Classify: "get my orders", "cancel order 57", "add item to cart", "set address to Home".
- Four layers each retry 2 times (2 tries per layer). How many calls can one tap cause? And 3 layers with 5 tries each?
- Your gateway's p99 is 300 ms and its p99.9 is 450 ms. You can accept 1 false timeout in 100 calls. Which timeout? And for 1 in 1,000?
Solutions and hints
1. The server should reject it: the same key means "the same request", so a different amount is a client bug, not a retry. Stripe compares the incoming parameters and "errors if they're not the same". Save the amount with the result and compare:
import uuid
balance = {"priya": 1000}
done = {} # key -> (amount, saved result)
def charge(key, customer, amount, lose_reply):
if key in done:
old_amount, result = done[key]
if amount != old_amount: # same key, different request
raise ValueError(f"key reused with amount {amount}, "
f"first request was {old_amount}")
return result + " (replayed)"
balance[customer] -= amount
done[key] = (amount, f"charged {amount}")
if lose_reply:
raise ConnectionError("reply lost on the way back")
return done[key][1]
key = str(uuid.uuid4())
for attempt, amount, lose in [(1, 240, True), (2, 250, False),
(3, 240, False)]:
try:
print("attempt", attempt, charge(key, "priya", amount, lose))
except (ConnectionError, ValueError) as e:
print("attempt", attempt, "failed:", e)
print("Priya's balance:", balance["priya"])
attempt 1 failed: reply lost on the way back
attempt 2 failed: key reused with amount 250, first request was 240
attempt 3 charged 240 (replayed)
Priya's balance: 760
Replaying "charged 240" to a request for 250 would be a lie; charging 250 again would be a double charge. An error is the only honest answer. If the customer really changed the order, that's a new operation with a new key.
2. Change the first line after the seed to CLIENTS, DOWN_UNTIL = 2000, 10:
every 0.5 s 40,000 retries, busiest 100 ms: 2,000
backoff 6,000 retries, busiest 100 ms: 2,000
backoff+jitter 8,807 retries, busiest 100 ms: 153
Without jitter, the busiest moment is every phone at once (2,000), so it grows exactly with the number of clients. With jitter it is 153: still far smaller, because the retries are spread across the whole waiting window.
3. Double the cooldown each time a HALF-OPEN test fails (with a cap, here 16 s), and reset it when a call succeeds. The gateway is down until second 20; the loop makes one call per second and hides the fail-fast seconds:
class CircuitBreaker:
def __init__(self, limit=3, cooldown=2, max_cooldown=16):
self.state, self.fails, self.opened_at = "CLOSED", 0, 0
self.limit, self.base = limit, cooldown
self.cooldown, self.max_cooldown = cooldown, max_cooldown
def trip(self, now):
if self.state == "HALF-OPEN": # the test failed: wait longer
self.cooldown = min(self.cooldown * 2, self.max_cooldown)
self.state, self.opened_at = "OPEN", now
def call(self, fn, now):
if self.state == "OPEN":
if now - self.opened_at < self.cooldown:
return "fail fast"
self.state = "HALF-OPEN"
try:
result = fn()
except ConnectionError:
self.fails += 1
if self.state == "HALF-OPEN" or self.fails >= self.limit:
self.trip(now)
return "failed (timeout)"
self.state, self.fails = "CLOSED", 0
self.cooldown = self.base # healthy again: reset
return result
def gateway(now): # down until second 20
if now < 20:
raise ConnectionError("timeout")
return "paid"
cb = CircuitBreaker()
for now in range(0, 35): # one call per second
msg = cb.call(lambda: gateway(now), now)
if msg != "fail fast": # hide the fail-fast seconds
print(f"t={now:<3} {msg:<17} {cb.state:<7} "
f"next cooldown {cb.cooldown} s")
t=0 failed (timeout) CLOSED next cooldown 2 s
t=1 failed (timeout) CLOSED next cooldown 2 s
t=2 failed (timeout) OPEN next cooldown 2 s
t=4 failed (timeout) OPEN next cooldown 4 s
t=8 failed (timeout) OPEN next cooldown 8 s
t=16 failed (timeout) OPEN next cooldown 16 s
t=32 paid CLOSED next cooldown 2 s
t=33 paid CLOSED next cooldown 2 s
t=34 paid CLOSED next cooldown 2 s
The test calls come at t = 4, 8, 16, 32: backoff, but for the breaker. A long outage gets fewer and fewer test calls. The price: the gateway was back at t = 20, but we only noticed at t = 32. That's why the cooldown needs a cap (max_cooldown); adding jitter to the cooldown also stops many servers from testing at the same moment.
4. Wait random.uniform(0, min(cap, base * 2 ** attempt)) between tries (and don't wait after the last one). This version prints the wait instead of sleeping, so it runs instantly:
import random
random.seed(2)
def charge(order_id, amount): # the payment gateway, far away
if random.random() < 0.3: # 3 in 10 calls: the network fails
raise ConnectionError("gateway did not answer")
return f"{order_id}: paid Rs {amount}"
def charge_with_retry(order_id, amount, tries=4, base=0.1, cap=2.0):
for attempt in range(1, tries + 1):
try:
return charge(order_id, amount)
except ConnectionError as e:
if attempt == tries:
break
wait = random.uniform(0, min(cap, base * 2 ** attempt))
print(f" {order_id} attempt {attempt} failed: {e}; "
f"waiting {wait:.2f} s")
# time.sleep(wait) # a real client sleeps here
raise ConnectionError(f"{order_id}: gave up after {tries} tries")
for order_id in ["A1", "A2", "A3", "A4", "A5"]:
print(charge_with_retry(order_id, 240))
A1: paid Rs 240
A2: paid Rs 240
A3 attempt 1 failed: gateway did not answer; waiting 0.02 s
A3: paid Rs 240
A4: paid Rs 240
A5: paid Rs 240
A3 now fails once instead of twice: the jitter itself uses random numbers, which shifts the seeded sequence, so the simulated failures land differently. With base=0.1, the waits are at most 0.2, 0.4, 0.8 s for attempts 1, 2, 3, never above the 2 s cap. (This is the same randint(0, base * 2 ** attempt) idea from the AWS summary, using a float instead of an integer.)
5. "Get my orders" (a GET) and "set address to Home" are idempotent: repeating them changes nothing more. "Cancel order 57" is idempotent too: cancelling an already-cancelled order leaves it cancelled (like DELETE). "Add item to cart" is not: each repeat adds another item. It needs an idempotency key, or a redesign as "set the quantity of item X to 2", which is idempotent.
6. Each layer multiplies: 2 × 2 × 2 × 2 = 16 calls. 5 × 5 × 5 = 125 calls. Even "small" retry counts explode when every layer does them; retry at one layer.
7. 1 in 100 = 1% false timeouts → the p99: 300 ms. 1 in 1,000 = 0.1% → the p99.9: 450 ms. A lower false-timeout rate means a longer timeout.
Quick quiz
# Which call is safe to retry
# WITHOUT an idempotency key?
A charge(card, 240)
B send_sms(phone, "Order ready")
C set_status(order, "delivered")
D add_to_cart(item)
- A. charge(card, 240)
- B. send_sms(phone, "Order ready")
- C. set_status(order, "delivered")
- D. add_to_cart(item)
Answer
C. Setting a status to "delivered" twice leaves it "delivered": the same effect as once. Charging, sending an SMS and adding to a cart all happen again on every repeat: a second charge, a second SMS, a second item.
Interview questions
- Why do network calls fail, and which failures should you retry? (Dropped packets, restarts, overload; retry transient ones such as timeouts and 503s, never permanent ones such as "card declined" or "invalid input".)
- What does idempotent mean? Give an idempotent and a non-idempotent example. (Same effect once or many times: set a status, GET, DELETE; vs charge a card, send an SMS, add to cart.)
- A payment call timed out. Did the charge happen? How do you retry safely? (You can't know: the request or the reply may have been lost. Retry with an idempotency key made once per payment; the server stores key → result and replays it.)
- How would you implement idempotency keys on the server? (Store key → request parameters + saved response in a database, with a unique constraint on the key; replay the saved response for a repeat; reject the same key with different parameters; expire old keys, e.g. Stripe allows removing them after 24 hours.)
- What happens if two retries with the same key arrive at the same moment? (Both could miss the "seen it?" check. Insert the key first with a unique constraint or a lock, so only one does the work; the other waits or gets an error and retries.)
- What is a retry storm, and how do exponential backoff and jitter help? (Many clients retrying together keep a failing service down; backoff reduces the number of retries, jitter spreads them so they don't arrive in waves.)
- Why is retrying at every layer dangerous? (Retries multiply: 3 layers × 3 tries = 27 calls per user action.)
- How do you choose a timeout? (From the downstream's latency percentiles: pick an acceptable false-timeout rate such as 0.1% and use the matching percentile, p99.9.)
- Explain a circuit breaker and its three states. (Closed: calls flow, failures counted; open after a threshold: fail fast with a fallback; half-open after a cooldown: one test call decides closed or open.)
- What should happen when the breaker is open? (A fallback: cached data, queue the work for later, a default, or a clear "try again" message; never a long wait.)
- How would you make ChaiNow's checkout reliable? (Timeout from the gateway's p99.9, retries with backoff and jitter at one layer, an idempotency key per payment, a circuit breaker with a "cash on delivery" fallback, and monitoring to see when the breaker opens: next episode.)
Glossary
| Term | Meaning |
|---|---|
| Network call | Sending a message to another computer and waiting for its reply, e.g. checkout asking the gateway to charge Rs 240 |
| Payment gateway | The company or service that actually moves money from the customer to the shop |
| Request / reply | The message you send / the answer that comes back |
ConnectionError |
Python's built-in error for "couldn't talk to the other side" |
| Transient failure | A failure that fixes itself a moment later: a dropped packet, a restart, a brief overload |
| Permanent failure | A failure that gives the same answer every time, e.g. "card declined" |
| Retry | Trying a failed call again |
| Maximum tries | The limit after which a retry loop gives up (here 3) |
| HTTP 500 | "Internal Server Error": something broke on the server |
| HTTP 503 | "Service Unavailable": the server can't take the request right now; a typical transient error |
| Status code | The number at the start of an HTTP reply that says how it went (200 OK, 500, 503, ...) |
| HTTP header | A labelled extra line sent with a request, e.g. Idempotency-Key: <uuid> |
| Side effect | A change a call makes to the world: moving money, sending an SMS, adding to a cart |
| Idempotent | Doing it twice has the same effect as doing it once, e.g. pressing a lift button, set_status, GET, DELETE |
| Idempotency key | A unique id the client makes once per operation and sends with every retry, so the server can do the work only once |
| Replay | The server returning the saved result for a key it has already seen, instead of doing the work again |
| UUID (v4) | A random 128-bit id, practically never repeated; Stripe suggests V4 UUIDs for idempotency keys (episode 23) |
| Stripe | A payment company whose API documentation we use as the idempotency example |
| Retry storm | Many clients retrying at once, piling load on a failing service and keeping it down |
| Exponential backoff | Waiting longer after each failure, doubling: 2, 4, 8 seconds |
| Cap | An upper limit on the backoff wait (here 8 s), so it doesn't grow forever |
| Jitter | Adding randomness to the wait (a random time between 0 and the backoff), so clients don't retry together |
| Wave | Many retries arriving at exactly the same moment |
Seed (random.seed) |
A starting number that makes "random" numbers repeat exactly, run after run |
| Layer | One service in a chain of calls: app → order service → payment service → gateway |
| Timeout | The longest you'll wait for a reply before treating the call as failed |
| False timeout | Cutting off a healthy call that was just slower than the timeout |
| Latency | How long a call takes to get its reply |
| Percentile (p50, p99, p99.9) | The latency that 50% / 99% / 99.9% of calls finish under (episode 4); p99.9: only 1 in 1,000 is slower |
| Downstream | The service you call (the gateway is downstream of checkout) |
| Circuit breaker | A wrapper around calls that stops calling a failing service after a few failures, and tests it again later |
| Closed (state) | Normal: calls go through and failures are counted |
| Open (state) | Tripped: calls are not sent; they fail fast with a fallback |
| Half-open (state) | After the cooldown: one test call decides whether to close or open again |
| Threshold | The number of failures that trips the breaker (here 3) |
| Cooldown | How long the breaker stays open before letting a test call through (here 5 s) |
| Fail fast | Answering "failed" instantly instead of waiting for a timeout |
| Fallback | A backup answer when the real call can't be made: cached data, a queue, a default, "cash on delivery" |
| Cascading failure | One failing service slowing down or breaking the services that depend on it, one after another |
| Martin Fowler | Software author who popularized the circuit breaker pattern |
| "Release It!" | Michael Nygard's book that the circuit breaker pattern comes from |
| AWS Builders' Library | Amazon's articles on how it builds systems; "Timeouts, retries and backoff with jitter" is the one used here |
Summary
- Network calls fail; most failures are transient, so retry them, a few times, never permanent errors.
- A failed call may have done the work and lost the reply; retrying a non-idempotent call (a payment) can charge twice.
- An idempotency key made once per operation lets the server replay the saved result. Stripe:
Idempotency-Keyheader, V4 UUIDs suggested, up to 255 characters, saved responses replayed (even 500 errors), keys removable after 24 hours, an error if the parameters differ. - Fixed-interval retries cause retry storms (20,000 retries); exponential backoff cuts the count (3,000), jitter breaks the waves (busiest 100 ms: 1,000 → 88). Retry at one layer (3 × 3 × 3 = 27).
- Every call needs a timeout, chosen from the downstream's latency percentiles (0.1% false timeouts → p99.9).
- A circuit breaker (closed → open → half-open) stops calling a dead service, fails fast with a fallback, and tests recovery with one call after a cooldown.
Next episode: the breaker opened at 2 a.m. Which service broke, and why? Your users know something is slow, but you can't see inside: observability with logs, metrics and traces.
Sources: Stripe API documentation, "Idempotent requests" ("safely retrying requests without accidentally performing the same operation twice"; the Idempotency-Key header; "we suggest using V4 UUIDs"; keys up to 255 characters; API v1 saves the status code and body of the first request, "Retries with the same key return the saved response, including 500 errors"; keys can be removed "after they're at least 24 hours old"; "errors if they're not the same"; GET and DELETE "are idempotent by definition"); Lumigo (Yan Cui), summary of the Amazon Builders' Library article "Timeouts, retries and backoff with jitter" ("retrying at all layers ... multiplies the number of retries"; "inject jitter to the backoff" with randint(0, base * 2 ** attempt); use "the corresponding percentile latency for the downstream as the timeout"; "APIs with side effects aren't safe to retry unless they provide idempotency"); Martin Fowler's circuit breaker article as summarised on aleburrascano.github.io (Closed, Open, Half-Open; popularized from Michael Nygard's "Release It!"; fallbacks such as cached data, a queue or a default).