These are the notes for System Design episode 17. ChaiNow started as one program: the menu, orders, payments and loyalty points are just modules calling each other. That is a monolith, and it's a great way to start. Then one bug takes the whole program down, the Diwali rush forces you to copy everything, and every small change waits for everyone else's. You'll split ChaiNow into microservices, then discover their price: network calls are slow and can fail, so every call needs a timeout and a fallback. Finally, the honest part: when splitting is worth it, and when it isn't. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). RAM sizes, copy counts, the 5 ms network time and the 99.9% uptime are made-up numbers.
The problems
| Problem | What it teaches |
|---|---|
| One bug, whole app down | Monolith, process, shared fate, fault isolation |
| Diwali: only the menu is busy | Scaling a monolith means copying all of it |
| Every change waits for everyone | Tied change cycles, release trains, independent deploys |
| Split it up | Microservices: own process, own database, own team; Conway's law |
| Network calls are slow and flaky | Latency, compound availability, timeouts, fallbacks |
| Was it worth it? | Distributed data, operations cost, Prime Video's 90% cut, Monolith First, modular monolith |
The start: a monolith
A monolith is an application "built as a single unit" (Lewis & Fowler): one codebase, built, deployed and run as one program. Think of a small dhaba where one kitchen does everything.
- Modules call each other with ordinary function calls, in the same memory: instant and never lost.
- There is usually one shared database: any module can read any table, and a SQL
JOINacross orders and points is easy. - For a small team it is simple to build, test, run and debug. This is not a mistake. It's the right place to start.
A process is one running program, with its own memory, managed by the operating system. A monolith runs as one process (or several identical copies of it).
Problem 1: one bug, whole app down
Three teams work in one program. The points team ships a "streak bonus": average points per visit.
menu = {"chai": 20, "samosa": 15} # menu team's code
points = {"asha": 40, "ravi": 0} # points team's code
visits = {"asha": 4, "ravi": 0}
def price(items): # menu team
return sum(menu[i] for i in items)
def add_points(user, amount): # points team, new "streak bonus"
streak = points[user] // visits[user] # average points per visit
points[user] += amount // 10 + streak
def place_order(user, items): # orders team
total = price(items)
add_points(user, total) # a normal function call
print(f"{user}: order of Rs {total} placed")
place_order("asha", ["chai", "samosa"])
place_order("ravi", ["chai"]) # ravi's first visit
print("menu still works?", price(["chai"]))
Output:
asha: order of Rs 35 placed
Traceback (most recent call last):
File "main.py", line 18, in <module>
place_order("ravi", ["chai"]) # ravi's first visit
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "main.py", line 14, in place_order
add_points(user, total) # a normal function call
~~~~~~~~~~^^^^^^^^^^^^^
File "main.py", line 9, in add_points
streak = points[user] // visits[user] # average points per visit
~~~~~~~~~~~~~^^~~~~~~~~~~~~~
ZeroDivisionError: integer division or modulo by zero
Ravi has zero visits, so // 0 raises ZeroDivisionError. The order fails because of loyalty points, the least important feature. And the last line, the menu check, never runs: the whole program stopped. In one process every module shares one fate. (A real web server usually turns an exception into a failed request rather than exiting, but a crash, a memory leak or an infinite loop in any module still takes the whole process, and every feature in it, down.)
Fault isolation means one part failing doesn't take the others with it. A monolith has little of it.
Problem 2: Diwali, only the menu is busy
In episode 5 we scaled out by running more copies behind a load balancer. A monolith is one unit, so every copy carries every module.
ram_gb = {"menu": 0.5, "orders": 0.5, "payments": 0.5, "points": 0.5}
need = {"menu": 10, "orders": 2, "payments": 2, "points": 1} # copies for the Diwali rush
one_app = sum(ram_gb.values()) # a monolith copy carries every module
mono = max(need.values()) * one_app # copy ALL of it for the busiest part
print(f"monolith: {max(need.values())} copies x {one_app} GB = {mono} GB")
split = sum(need[s] * ram_gb[s] for s in ram_gb)
for s in ram_gb:
print(f" {s:8} {need[s]:2} copies x {ram_gb[s]} GB")
print(f"separate services: {split} GB")
Output:
monolith: 10 copies x 2.0 GB = 20.0 GB
menu 10 copies x 0.5 GB
orders 2 copies x 0.5 GB
payments 2 copies x 0.5 GB
points 1 copies x 0.5 GB
separate services: 7.5 GB
Lewis & Fowler: in a monolith, "Scaling requires scaling of the entire application rather than parts of it that require greater resource."
Problem 3: every change waits for everyone
With forty developers in one codebase, a one-word typo fix on the menu still means rebuilding, retesting and redeploying the whole app. Lewis & Fowler: "Change cycles are tied together - a change made to a small part of the application, requires the entire monolith to be rebuilt and deployed." Teams batch changes into a weekly release train; one team's failing test makes everyone miss it.
The fix: microservices
From one dhaba to a food court: separate stalls, each with its own store room and its own staff.
Lewis & Fowler (2014): "the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API."
| Idea | Meaning |
|---|---|
| Service | A separate program, in its own process, doing one business job (menu, orders, points) |
| Own database | Each service manages its own data ("letting each service manage its own database"); others ask through its API |
| Independent deploys | A team ships its service without waiting for others |
| Independent scaling | 10 copies of menu, 1 of points |
| Fault isolation | If points crashes, orders keep working (if orders handles that well, see below) |
Conway's law: "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." Microservices use it on purpose: one team, one service.
Problem 4: network calls are slow and flaky
A function call becomes a network call (a remote call): a request sent to another machine and a reply sent back. It takes time (latency) and can be lost.
import time
def points_local(user): # inside the monolith
return 40
def points_remote(user): # points is now another service
time.sleep(0.005) # ~5 ms there and back (made up)
return 40
for call in [points_local, points_remote]:
start = time.perf_counter()
for _ in range(100):
call("asha")
ms = (time.perf_counter() - start) * 1000
print(f"{call.__name__:13}: 100 calls in {ms:7.2f} ms")
print("each service is up 99.9% of the time:")
for hops in [1, 5, 10, 20]:
up = 0.999 ** hops # every hop must work
print(f" order needs {hops:2} services -> works {up:.1%} of the time")
Output (the remote timing varies a little per run):
points_local : 100 calls in 0.01 ms
points_remote: 100 calls in 515.38 ms
each service is up 99.9% of the time:
order needs 1 services -> works 99.9% of the time
order needs 5 services -> works 99.5% of the time
order needs 10 services -> works 99.0% of the time
order needs 20 services -> works 98.0% of the time
Availability (the share of time something works) multiplies along a chain: if every hop must work, the chain works 0.999 × 0.999 × ... of the time. Twenty services at 99.9% each give 98.0%: 1 order in 50 fails. A chatty design, where one request hops through many services, gets slow and fragile.
Timeouts and fallbacks
The worst failure is the service that never answers. A timeout is the longest you'll wait for a reply. A fallback is what you do instead when the call fails.
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError
pool = ThreadPoolExecutor()
later = [] # a queue, like episode 16
def points_service(user, amount): # another team's service
time.sleep(2) # it's stuck today
return "ok"
def place_order(user, total, timeout=None):
start = time.perf_counter()
answer = pool.submit(points_service, user, total) # send the request
try:
answer.result(timeout=timeout) # wait for the reply
except TimeoutError:
print(" points: no reply, add them later")
later.append((user, total)) # fallback
took = time.perf_counter() - start
print(f"{user}: order placed in {took:.1f} s")
place_order("asha", 35) # wait as long as it takes
place_order("ravi", 20, timeout=0.3) # wait at most 0.3 s
print("points to add later:", later)
Output:
asha: order placed in 2.0 s
points: no reply, add them later
ravi: order placed in 0.3 s
points to add later: [('ravi', 20)]
ThreadPoolExecutor.submit runs the call on a background thread and returns a future (a placeholder for the result); result(timeout=...) waits at most that long and raises TimeoutError otherwise. Rule: every network call gets a timeout and a plan for failure; otherwise one slow service makes all its callers slow too (a cascading failure). For points, the plan is a queue (episode 16): the order succeeds and points are added later.
Problem 5: was it worth it?
The bill for microservices:
- Data is split. No SQL
JOINacross services; keeping orders and payments consistent across two databases is hard (later episodes cover patterns for this). - Debugging spreads out. One order touches many services; you follow it through many logs (episode 25: observability).
- Operations multiply. Twenty programs to deploy, monitor, secure and version.
Real example. In 2023 Amazon's Prime Video team described its audio/video quality-monitoring service, built from distributed serverless components (AWS Step Functions, Lambda, S3, SNS). Costs came from "a high volume of read/writes to the S3 bucket storing intermediate work items" and "several state transitions for each second of the analyzed audio/video stream". They moved the work into a single application process on ECS/EC2, and operational costs fell by 90% (InfoQ, May 2023). For that workload, fewer network hops won.
When to split
Martin Fowler, "MonolithFirst" (2015): "Almost all the successful microservice stories have started with a monolith that got too big and was broken up", and "Almost all the cases where I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble."
| Situation | Choice |
|---|---|
| Small team, new product | A monolith |
| Growing codebase | A modular monolith: one deploy, clean modules that only talk through clear interfaces |
| One part keeps breaking others | Split that part out |
| One part needs 10x the scale | Split that part out |
| Teams keep blocking each other's releases | Split along team lines |
| One request hops through many services | You split too far; merge or use a queue |
The whole chain
- Monolith: one program, fast function calls, one database. A good start.
- One bug (
ZeroDivisionError) → the whole program stops. - Diwali → copy everything: 20 GB vs 7.5 GB.
- Every change waits for the release train.
- Microservices: own process, own database, own team; fault isolation, independent scaling and deploys.
- Network calls: ~5 ms each, 99.9%^20 = 98.0% → timeouts and fallbacks.
- Real cost (Prime Video: −90% after moving to one process) → Monolith First, split when you can name the pain.
Common mistakes (and the real errors)
| Mistake | Code | Error / result | Fix |
|---|---|---|---|
| Risky code in the same process as everything | points[user] // visits[user] |
ZeroDivisionError: integer division or modulo by zero (and the program stops) |
Test edge cases; isolate risky or low-priority parts |
| Calling a service with no timeout | answer.result() |
asha: order placed in 2.0 s |
result(timeout=0.3) + a fallback |
| Long chains of service calls | 20 services in a row | works 98.0% of the time |
Fewer hops; move side jobs to a queue |
| Microservices sharing one database | every service reads every table | a schema change breaks other teams | Each service owns its data; others use its API |
| Splitting a new product on day one | 20 services for 3 developers | slow delivery, many moving parts | Monolith first; keep it modular |
Mini project: a safe call to a flaky service
A timeout on every call, one retry, and a fallback list that is drained once the service is healthy.
# Mini project: an order service that calls a flaky points service safely:
# a timeout on every call, one retry, and a fallback list that is drained later.
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError
pool = ThreadPoolExecutor(max_workers=8)
pending, points = [], {}
calls = {"n": 0}
HANGS = {1, 3, 4} # which calls hang (made up)
def points_service(user, amount): # another team's service
calls["n"] += 1
if calls["n"] in HANGS:
time.sleep(1.0) # stuck, then gives up
raise ConnectionError("points service reset the connection")
points[user] = points.get(user, 0) + amount // 10
return "ok"
def call_with_timeout(fn, *args, timeout=0.2, retries=1):
for attempt in range(1, retries + 2):
try:
return pool.submit(fn, *args).result(timeout=timeout)
except TimeoutError:
print(f" attempt {attempt}: no reply in {timeout} s")
raise TimeoutError("points service unavailable")
def place_order(user, total):
start = time.perf_counter()
try:
call_with_timeout(points_service, user, total)
except TimeoutError:
pending.append((user, total)) # never fail the order for points
print(f"{user}: order placed in {time.perf_counter() - start:.1f} s")
for user, total in [("asha", 35), ("ravi", 20), ("meena", 50)]:
place_order(user, total)
print("pending:", pending)
HANGS.clear() # later: the service is healthy again
while pending:
call_with_timeout(points_service, *pending.pop(0))
print("points:", points)
Output:
attempt 1: no reply in 0.2 s
asha: order placed in 0.2 s
attempt 1: no reply in 0.2 s
attempt 2: no reply in 0.2 s
ravi: order placed in 0.4 s
meena: order placed in 0.0 s
pending: [('ravi', 20)]
points: {'asha': 3, 'meena': 5, 'ravi': 2}
No order ever waited more than 0.4 s, and Ravi's points arrived later. Careful: here a hung call fails without adding points. If a "timed-out" call could still succeed later, a retry would add points twice, so real retries need the idempotency keys from episode 16. Extend it: add an order id to every call and skip ids already applied.
Practice
- Homework: fix the streak bonus in Demo 1 so new users don't crash it. What does the program print now?
- Homework: in Demo 3, what is the success rate with 50 services in a row? And with 20 services at 99.99% each?
- Homework: in Demo 4, set Ravi's timeout to 3 s. What changes?
- ChaiNow has 3 developers and 200 orders a day. Monolith or microservices? Why?
- Two services both read and write the same
orderstable. Which microservice benefit did you lose? - Give two reasons Prime Video's monitoring workload got cheaper in one process.
Solutions and hints
1. Guard the division: streak = points[user] // visits[user] if visits[user] else 0:
asha: order of Rs 35 placed
ravi: order of Rs 20 placed
menu still works? 20
2. print(f"{0.999**50:.1%}", f"{0.9999**20:.1%}") prints 95.1% 99.8%. Fifty hops at 99.9% fail 1 order in 20; making each service ten times more reliable (99.99%) brings 20 hops back to 99.8%.
3. With timeout=3 the reply (2 s) arrives in time:
asha: order placed in 2.0 s
ravi: order placed in 2.0 s
points to add later: []
A timeout longer than the slow case protects nothing. Pick it from how fast the service normally answers, and what the caller can afford.
4. A monolith (keep it modular). None of the pains exist yet: no team blocking, no part needing separate scale. Microservices would add network calls and operations work for nothing.
5. Independence: a schema change by one team can break the other, and they can't deploy or scale their data separately. Each service should own its data.
6. No more reading and writing intermediate frames to S3 between components (data passed in memory), and no more orchestration state transitions for every second of video. Network hops and per-call charges disappeared.
Quick quiz
# every service is up 99.9% of the time
# an order needs these, in a row:
# gateway, menu, orders, payments, points
print(f"{0.999 ** 5:.1%}")
- A. 99.9%
- B. 99.5%
- C. 95.0%
- D. 100.0%
Answer
B. Every hop must work, so multiply: 0.999⁵ ≈ 0.995, printed as 99.5%.
Interview questions
- What is a monolith, and when is it the right choice? (One deployable unit; small teams and new products.)
- What do microservices give you? (Independent deploys, independent scaling, fault isolation, team ownership.)
- What do they cost? (Network latency and failures, distributed data, harder debugging, more operations.)
- What is Conway's law, and how does it relate to microservices? (Systems mirror the organisation's communication; one team per service.)
- Why must every remote call have a timeout? (Otherwise one stuck service blocks every caller: cascading failure.)
- What is a modular monolith? (One deploy, but strictly separated modules; easy to split later.)
- Should each microservice have its own database? Why? (Yes: otherwise schema changes and load couple the services again.)
- What happened at Prime Video, and what's the lesson? (A distributed design moved into one process cut cost 90%: pick the architecture per workload.)
Glossary
| Term | Meaning |
|---|---|
| Monolith | An application built, deployed and run as a single unit |
| Module | A part of a program with one job (menu, points), inside the same codebase |
| Process | One running program with its own memory |
| Function call | Calling code in the same process: in memory, instant |
| Shared fate | When one part fails, everything in the same process fails with it |
| Fault isolation | One part failing doesn't take down the others |
| Scale out | Run more copies of a program behind a load balancer (episode 5) |
| Release train | A fixed, shared release (e.g. weekly) that all changes wait for |
| Deploy | Put a new version of a program into production |
| Microservice | A small service in its own process, with its own data, owned by one team |
| Microservice architecture | Building one application as a suite of small services talking over the network |
| API | Application programming interface: how one program asks another for something |
| Conway's law | A system's structure copies the organisation's communication structure |
| Network call / remote call | A request to another machine and its reply |
| Latency | The time a request takes to get its answer |
| Availability | The share of time a system works (99.9% ≈ 43 minutes down a month) |
| Compound availability | Chained parts multiply: 0.999²⁰ = 98.0% |
| Chatty design | One request hopping through many services |
| Timeout | The longest a caller waits for a reply before giving up |
| Fallback | What you do instead when a call fails (a default, a queue, a cached value) |
| Cascading failure | One slow or failed service making its callers slow or fail too |
| Future | A placeholder for a result that isn't ready yet (concurrent.futures) |
ThreadPoolExecutor |
Python's pool of threads that runs functions in the background |
| Retry | Trying a failed call again (safe only if the call is idempotent, episode 16) |
| JOIN | A SQL operation combining rows from two tables; impossible across separate service databases |
| Modular monolith | One deployable program with strictly separated modules |
| Monolith First | Fowler's advice: start with a monolith, split when it gets too big |
| Serverless | Cloud functions and workflows you don't run servers for (e.g. AWS Lambda, Step Functions) |
Summary
- A monolith is one program: simple, fast function calls, one database. A great start.
- Its pains grow with size: one bug can stop it all (
ZeroDivisionError), you scale it as a whole (20 GB vs 7.5 GB), and every change waits for one release. - Microservices give each part its own process, data and team: fault isolation, independent scaling and deploys.
- The price is the network: slow (≈5 ms vs 0.0001 ms here), and failures multiply (99.9%²⁰ = 98.0%). Every call needs a timeout and a fallback.
- Splitting costs real money and effort (Prime Video cut 90% by going back to one process for one workload). Monolith first, keep it modular, split when you can name the pain.
Next episode: our services talk over the network, but in what language? API design: REST, GraphQL and gRPC.
Sources: James Lewis & Martin Fowler, "Microservices" (martinfowler.com, 2014); Martin Fowler, "MonolithFirst" (2015); InfoQ, "Prime Video Switched from Serverless to EC2 and ECS to Save Costs" (2023), on the Prime Video Tech blog post "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%".