These are the notes for System Design episode 37. Neha orders a masala dosa on a food app. First she logs in with an OTP, which must arrive in seconds; then come "order placed" and "out for delivery"; and sometimes an offer, which can wait and should come only if she wants it. Today you design the system behind every buzz on her phone. You'll write the requirements, watch an order that sends its own messages make Neha wait 2.0 s and then crash with ConnectionError: push is down, and fix it with a message queue and workers (0.0 s). You'll get past a flaky SMS gateway with retries, exponential backoff (wait 1, 2, 4 s) and a dead letter queue, stop a crashed worker from sending the OTP twice with message ids, and rescue an OTP stuck 17 minutes behind a 10 lakh offer campaign with an urgent queue. Then you'll keep offers away at 11 pm with preferences, quiet hours (promotional SMS only from 9 am to 9 pm) and a daily cap of 3, drop a 2-hour-old OTP with a TTL, and turn 5 crore users into 250 million notifications a day: 2,894 a second, 11,574 at the evening peak. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). Neha, Ravi and the other people are fictional. The provider delays, the failure counts, 1,000 messages a second, the 10 lakh campaign, the daily cap of 3, the TTL values, 5 crore users × 5 notifications a day and the 4x peak are our own assumptions; the real-world facts come only from the sources at the end.
A notification is a short message a system sends to tell a person that something happened or needs their attention. Example: "Your masala dosa is out for delivery" appearing on Neha's phone.
The problems
| Problem | What it teaches |
|---|---|
| Neha orders a masala dosa | Requirements: many channels, any service can ask, respect choices, OTPs in seconds, never slow the app |
| The order waits for messages | Synchronous sending fails; a message queue plus workers (asynchronous) |
| A flaky SMS gateway | Retries, exponential backoff, a dead letter queue |
| The OTP arrives twice | At-least-once delivery; message ids and idempotency |
| OTP stuck behind offers | Priority: an urgent queue the workers empty first |
| Offers at 11 pm | Preferences, quiet hours (TRAI: promotional SMS 9 am to 9 pm), a daily cap |
| A 2-hour-old OTP | TTL (time to live): stale messages are dropped |
| The scale | 5 crore users → 250 million a day → 11,574 a second at peak |
| Put it all together | Services → notification API → queues → workers → providers, plus a delivery log |
Problem 0: Neha orders a masala dosa (requirements)
Neha's phone gets three very different kinds of messages from one food order:
| Message | Example | Rule |
|---|---|---|
| Login OTP | 482913 |
Urgent: needed in seconds |
| Order updates | "Order placed", then "out for delivery" | Important: push, maybe SMS too |
| An offer | "50% off this week" | Can wait, and only if she wants it |
A few words first:
- An OTP (one-time password) is a short code sent to your phone that works for one login or one payment, usually for only a few minutes (general knowledge). Example:
Your OTP is 482913. - A push notification is a message an app's server sends to your phone that pops up even when the app is closed. Example: "Order placed!" on Neha's lock screen.
- An SMS (short message service) is a plain text message sent over the phone network; it reaches any phone, even without the app or the internet. Example: the OTP arriving as a text message.
- An email is a message sent to an email address. Example: the order receipt.
- A channel is one way of reaching a user: push, SMS or email. Example: the OTP goes by the SMS channel.
Functional requirements (what the system must do):
- Send on many channels: push notifications, SMS and email.
- Any service in the company can ask for a notification: login, orders, marketing.
- Respect each user's choices.
Non-functional requirements (how well it must do it):
- OTPs in seconds, even while millions of offers are going out.
- Reliable: failures are retried, and no message is sent twice.
- The most important one: a slow message must never slow down the app itself.
Problem 1: the order waits for messages (a queue and workers)
The obvious way: when Neha places her order, save it, then send the SMS, the email and the push right there, inside the order.
- A provider is an outside company or service that actually delivers messages on one channel. Example: an SMS provider that sends texts to phone networks.
- A gateway (here, an SMS gateway) is a provider's entry point that our system calls to hand over a message. Example: our code calls the SMS gateway with "Order placed!" and a phone number.
- Synchronous means the caller waits until the work is finished before moving on. Example:
place_orderwaits forsend_smsto return.
Our pretend SMS provider takes 0.8 seconds, email takes 1.2 seconds, and the push provider is down today (all our own numbers):
import time
def send_sms(text): time.sleep(0.8) # a slow SMS provider
def send_email(text): time.sleep(1.2) # a slow email provider
def send_push(text): raise ConnectionError("push is down")
def place_order(item):
start = time.time()
print("order saved:", item)
send_sms("Order placed!") # the order waits for
send_email("Order placed!") # every message...
print(f"customer waited {time.time() - start:.1f} s")
send_push("Order placed!") # one failure...
return "order confirmed" # ...breaks the order
print(place_order("masala dosa"))
Output (file paths in the traceback shortened to main.py):
order saved: masala dosa
customer waited 2.0 s
Traceback (most recent call last):
File "main.py", line 16, in <module>
print(place_order("masala dosa"))
~~~~~~~~~~~^^^^^^^^^^^^^^^
File "main.py", line 13, in place_order
send_push("Order placed!") # one failure...
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "main.py", line 5, in send_push
def send_push(text): raise ConnectionError("push is down")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionError: push is down
- Order saved. The customer waited 2.0 s, 0.8 + 1.2, just for messages.
- Then
ConnectionError: push is down, andplace_ordernever returns "order confirmed". A ConnectionError is the Python error for a failed network connection; a traceback is Python's report of where the error happened, innermost call last. - Why? The order is tied to every provider. Their slowness becomes our slowness, and their outage becomes our outage. Neha's dosa order failed because a push service was down.
Real life: the restaurant order slip
| In a restaurant | In our system |
|---|---|
| The waiter writes a slip and goes to the next table | Put a message in a queue: takes no time |
| Cooks pick up the slips in order, as fast as they can | Workers send SMS, email, push in the background |
| Busy night? Add cooks; the waiter is no slower | Add workers: they scale separately |
- A message queue is a waiting line of messages: one part of a system adds messages at the back, another takes them from the front later (episode 16). Example:
outboxbelow, holding("sms", "Order placed!"). - A worker is a program that runs separately, takes messages from a queue and does the slow work. Example:
worker()below, which sends each message. - Asynchronous means the caller hands the work over and moves on without waiting for it to finish. Example: the order drops three notes in the queue and confirms at once.
The fix: drop a note and move on
import time
from collections import deque
outbox = deque() # the message queue
def place_order(item):
start = time.time()
print("order saved:", item)
for channel in ["sms", "email", "push"]: # just drop a note
outbox.append((channel, "Order placed!"))
print(f"customer waited {time.time() - start:.1f} s")
return "order confirmed"
def worker(): # runs separately
while outbox:
channel, text = outbox.popleft()
print(f" worker sends {channel}: {text}")
print(place_order("masala dosa"))
worker()
Output:
order saved: masala dosa
customer waited 0.0 s
order confirmed
worker sends sms: Order placed!
worker sends email: Order placed!
worker sends push: Order placed!
- The order only drops three notes into the queue (one per channel): customer waited 0.0 s, and the order is confirmed.
- The worker sends SMS, email and push afterwards. If a provider is down now, only the worker waits and retries; the order is already confirmed.
- A
deque(double-ended queue, from Python'scollections) is a list that is fast to add to at one end and take from at the other:appendadds at the back,poplefttakes from the front.
You just invented asynchronous notifications.
Problem 2: a flaky SMS gateway (retries, backoff, dead letters)
Workers still have to deal with failures. Our pretend SMS gateway times out: twice for Neha's OTP, and nine times for an offer to Ravi (our numbers).
- A timeout is when a caller stops waiting for a reply after a fixed time. Python's TimeoutError is the error for it. Example:
TimeoutError("SMS gateway timed out"). - A retry is sending the same message again after a failure (episode 24 covered retries).
- Exponential backoff means waiting longer after each failure, usually doubling: 1 s, 2 s, 4 s. Example:
wait *= 2below. - A dead letter queue is a separate place where messages that kept failing are parked, so they stop blocking the rest and a person can look at them later. Example:
dead_lettersbelow.
send tries up to four times. After each failure it doubles the wait; in this demo we print the wait instead of really sleeping. If all four tries fail, the message goes to the dead letter list:
failures = {"OTP for Neha": 2, "Offer for Ravi": 9} # times each fails
def provider(msg): # a flaky SMS gateway
if failures[msg] > 0:
failures[msg] -= 1
raise TimeoutError("SMS gateway timed out")
return "delivered"
dead_letters = []
def send(msg, max_tries=4):
wait = 1
for attempt in range(1, max_tries + 1):
try:
return f"{msg}: {provider(msg)} on try {attempt}"
except TimeoutError:
if attempt < max_tries:
print(f" {msg}: try {attempt} failed, wait {wait} s")
wait *= 2 # 1, 2, 4...: exponential backoff
dead_letters.append(msg) # give up: park it for a human
return f"{msg}: moved to the dead letter queue"
print(send("OTP for Neha"))
print(send("Offer for Ravi"))
Output:
OTP for Neha: try 1 failed, wait 1 s
OTP for Neha: try 2 failed, wait 2 s
OTP for Neha: delivered on try 3
Offer for Ravi: try 1 failed, wait 1 s
Offer for Ravi: try 2 failed, wait 2 s
Offer for Ravi: try 3 failed, wait 4 s
Offer for Ravi: moved to the dead letter queue
- Neha's OTP: failed, failed, delivered on try 3.
- Ravi's offer: failed four times (waits of 1, 2 and 4 s between the tries), then moved to the dead letter queue.
- Why double the wait? A struggling provider gets room to recover instead of a flood of instant retries from every worker (episode 24).
- Why a dead letter queue? Retrying forever would keep a worker busy with one hopeless message. Parking it keeps everything else moving, and nothing is lost: someone can fix the cause and resend it.
Problem 3: the OTP arrives twice (message ids)
A worker sends Neha's OTP, then crashes before telling the queue it's done. The queue thinks the message was never handled, so it hands it out again.
- An ack (acknowledgement) is the worker telling the queue "done, you can delete this message".
- At-least-once delivery means the queue promises every message is handed out at least once, and maybe more than once: if no ack arrives, it hands the message out again. That's the queue doing the safe thing, because losing a message would be worse.
- A message id is a unique name given to a message when it is created, kept on every retry and redelivery. Example:
N101. - Idempotency means doing the same thing twice has the same effect as doing it once (episode 34: the UPI transaction id). Example: delivering
N101twice puts one OTP on the phone. - Dedupe (de-duplicate) means dropping repeats of something already handled.
We deliver the queue twice: once ignoring ids, once skipping ids already sent:
queue = [("N101", "Your OTP is 482913")]
queue.append(queue[0]) # the worker crashed before saying "done",
# so the queue hands the message out again
def deliver(queue, check_ids):
sent_ids, phone = set(), []
for msg_id, text in queue:
if check_ids and msg_id in sent_ids:
continue # already sent this id: skip it
phone.append(text)
sent_ids.add(msg_id)
return phone
print("without ids:", deliver(queue, check_ids=False))
print("with ids: ", deliver(queue, check_ids=True))
Output:
without ids: ['Your OTP is 482913', 'Your OTP is 482913']
with ids: ['Your OTP is 482913']
- Without ids: the same OTP, twice. With ids: once.
- Two identical OTPs confuse people; two payment alerts make them panic. So workers keep a record of sent ids, in a fast store with an expiry, and skip repeats.
You just invented duplicate-free delivery.
Problem 4: OTP stuck behind offers (an urgent queue)
Real life: the ambulance lane
| On the road | In the queues |
|---|---|
| A traffic jam at 7 pm: thousands of cars | A big offer campaign: 10 lakh messages queued |
| An ambulance stuck in it: every minute matters | An OTP behind them: it expires |
| A lane only for the ambulance: always clear | An urgent queue: workers empty it first |
- A campaign is a batch of marketing messages sent to many users at once. Example: "50% off!" to 10 lakh users at 7 pm.
- A lakh is 100,000 and a crore is 10 million (Indian number words). 10 lakh = 1,000,000.
- A priority queue here means more than one queue, with workers always taking from the more important one first. Example: an urgent queue for OTPs and a bulk queue for offers.
Our workers send 1,000 messages a second, and 10 lakh offers are queued (both our numbers):
rate = 1_000 # messages our workers send a second
campaign = 1_000_000 # "50% off!" offers, queued at 7 pm
# one queue: Neha's OTP joins the line behind the whole campaign
wait = campaign / rate
print(f"one queue: OTP waits {wait:,.0f} s = {wait / 60:.0f} minutes")
# two queues: workers always empty the urgent queue first
urgent, bulk = ["OTP for Neha"], ["offer"] * campaign
order = urgent + bulk # what the workers pick next
pos = order.index("OTP for Neha") + 1
print(f"two queues: OTP is message number {pos}")
print("the offers still go out, just after anything urgent")
Output:
one queue: OTP waits 1,000 s = 17 minutes
two queues: OTP is message number 1
the offers still go out, just after anything urgent
- One queue: the OTP waits 1,000,000 ÷ 1,000 = 1,000 s = 17 minutes. An OTP usually expires in a few minutes (general knowledge), so it's useless by then.
- Two queues: the OTP is message number 1. The offers are not dropped, only sent after anything urgent.
- Real systems often go further: separate queues and separate workers for urgent, normal and bulk messages, so a campaign can never use up the workers OTPs need.
You just invented priority queues for notifications.
Problem 5: offers at 11 pm (preferences, quiet hours, a daily cap)
Next: being polite. Neha turned off offers in settings. Ravi allows offers, but he has already had three today.
- A preference is a choice a user has made about which messages they want, and how. Example: Neha's
"offers": False. - Quiet hours are times of day when non-urgent messages are held instead of sent. Example: an offer at 11 pm waits until 9 am.
- TRAI (Telecom Regulatory Authority of India) is the government body that sets rules for telecom in India, including commercial SMS (general knowledge).
- Promotional messages advertise something (offers, sales); transactional messages are ones a user needs because of something they did (OTPs, order updates, payment alerts). General knowledge; the time rule below is about promotional SMS.
- A rate limit caps how many requests or messages are allowed in a period (episode 19); a daily cap is a rate limit per user per day. Example: at most 3 offers a day (our own number).
Under TRAI rules, as MyOperator puts it: "Promotional SMS messages can only be delivered between 9:00 AM and 9:00 PM." We decide for every message, before it's sent:
prefs = {"Neha": {"offers": False}, "Ravi": {"offers": True}}
sent_today = {"Ravi": 3} # offers Ravi already got today
DAILY_CAP = 3 # our own limit
def decide(user, kind, hour):
if kind == "otp":
return "send now" # OTPs and alerts always go
if not prefs[user]["offers"]:
return "skip: turned off offers"
if not 9 <= hour < 21: # promotional SMS: 9 am to 9 pm
return "hold until 9 am"
if sent_today.get(user, 0) >= DAILY_CAP:
return "skip: daily limit reached"
return "send now"
print("Neha, OTP, 11 pm: ", decide("Neha", "otp", 23))
print("Neha, offer, 7 pm: ", decide("Neha", "offer", 19))
print("Ravi, offer, 11 pm:", decide("Ravi", "offer", 23))
print("Ravi, offer, 7 pm: ", decide("Ravi", "offer", 19))
Output:
Neha, OTP, 11 pm: send now
Neha, offer, 7 pm: skip: turned off offers
Ravi, offer, 11 pm: hold until 9 am
Ravi, offer, 7 pm: skip: daily limit reached
- Neha's OTP at 11 pm: send now. OTPs and alerts always go: the check order matters (see Common mistakes).
- Neha's offer: skip, she turned offers off.
- Ravi's offer at 11 pm: hold until 9 am.
9 <= hour < 21allows 9:00 am up to 8:59 pm. - Ravi's offer at 7 pm: skip, daily limit reached (3 of 3).
- Too many messages and people turn notifications off, or uninstall the app. So a preference service (the part that stores and checks these choices) is asked before every non-urgent send.
You just invented user preferences and quiet hours.
Problem 6: a 2-hour-old OTP (TTL)
Neha's phone is switched off. Push messages wait at the push service until the phone comes back.
- FCM (Firebase Cloud Messaging) is Google's push service, used to send push notifications to Android phones (and other apps); APNs (Apple Push Notification service) is Apple's push service for iPhones (general knowledge). Our workers hand push messages to them; they deliver to the phone.
- By default, FCM keeps a message for a long time. Firebase's docs: "Requests that don't contain this field default to the maximum period of four weeks".
- TTL (time to live) is how long a message stays useful; after that it is dropped instead of delivered. Example: 5 minutes for the OTP, an hour for the delivery update, a week for the offer (our numbers).
Two hours later, the phone comes back:
# Neha's phone is switched off; messages wait at the push service
waiting = [("OTP 482913", 5), # (text, minutes
("Your dosa is out for delivery", 60), # it stays useful)
("Offer: 50% off this week", 7 * 24 * 60)]
def phone_comes_online(after_minutes, use_ttl):
for text, ttl in waiting:
if use_ttl and after_minutes > ttl:
print(f" dropped (expired): {text}")
else:
print(f" shown: {text}")
print("2 hours later, no TTL:"); phone_comes_online(120, False)
print("2 hours later, with TTL:"); phone_comes_online(120, True)
Output:
2 hours later, no TTL:
shown: OTP 482913
shown: Your dosa is out for delivery
shown: Offer: 50% off this week
2 hours later, with TTL:
dropped (expired): OTP 482913
dropped (expired): Your dosa is out for delivery
shown: Offer: 50% off this week
- No TTL: a two-hour-old OTP, and an "out for delivery" message for a dosa she has already eaten.
- With TTL: both are dropped (expired); only the offer (still within its week) is shown.
- Firebase lets you set this lifespan on every message: "The value must be a duration from 0 to 2,419,200 seconds (28 days)". 2,419,200 ÷ 86,400 = 28 days.
You just invented message expiry.
Problem 7: the scale
How big does this get? Say an app has 5 crore users (50 million), each gets 5 notifications a day, and evenings are 4x busier than average. All three are our assumptions.
users = 50_000_000 # an app's users (our assumption)
per_user = 5 # notifications each, per day (our guess)
peak = 4 # evenings are 4x busier (our guess)
per_day = users * per_user
per_second = per_day / 86_400
print(f"per day: {per_day / 1e6:,.0f} million notifications")
print(f"per second, on average: {per_second:,.0f}")
print(f"at the evening peak: {per_second * peak:,.0f}")
Output:
per day: 250 million notifications
per second, on average: 2,894
at the evening peak: 11,574
- 50,000,000 × 5 = 250 million a day; ÷ 86,400 seconds = 2,894 a second on average; × 4 = 11,574 a second at the evening peak.
- That's why the queues are split into partitions and many workers run side by side for each channel. A partition is one slice of a big queue, held and read separately, so many workers can read different slices at the same time (general knowledge). Example: OTP messages split into 8 partitions by user id, with one worker reading each.
- The app never talks to Apple's or Google's push services directly. Workers do.
The full design
services notification API queues workers providers
──────── ──────────────── ────── ─────── ─────────
login ─┐ check preferences urgent ─┐ push ────────▶ FCM / APNs
orders ─┼─ notify ────▶ quiet hours, daily cap ──▶ normal ─┼──────────▶ SMS ────────▶ SMS gateway
offers ─┘ (user,kind) fill template, give id bulk ─┘ (urgent email ────────▶ email provider
set TTL first) │
├─ retry with backoff
├─ after the last try → dead letter queue
├─ sent ids → skip duplicates
└─ every attempt → delivery log
- Any service (login, orders, marketing) calls the notification API: "notify this user, this kind of message".
- The API checks preferences, quiet hours and the daily cap, fills in a template, gives the message a message id and a TTL, and puts it in the urgent, normal or bulk queue. It returns at once: the app drops a note and moves on.
- Workers for each channel take messages, urgent first, and call the providers: FCM for Android, APNs for iPhones, an SMS gateway, an email provider.
- Failures are retried with backoff; after the last try, the message goes to the dead letter queue.
- Sent ids are stored so nothing goes twice. Expired messages are dropped.
- Every attempt is written to the delivery log, so support can answer: "did my OTP go out?"
- A template is a stored message with blanks that are filled in for each user. Example:
"Your OTP is {code}"becomes"Your OTP is 482913". - The notification API is the single entry point other services call to send a notification.
- A delivery log is a record of every message and what happened to it. Example: "N1, Neha, SMS, try 3, delivered, 19:00:03".
The whole chain
- The customer waited 2.0 s, then
ConnectionError→ a queue and workers: 0.0 s. - A flaky gateway → retries with exponential backoff, then a dead letter queue.
- A crash sent the OTP twice → message ids, skip ids already sent.
- The OTP was 17 minutes late behind 10 lakh offers → an urgent queue.
- Offers at 11 pm → preferences, quiet hours (promotional SMS 9 am to 9 pm), a daily cap.
- A 2-hour-old OTP → TTL.
- 11,574 a second at peak → partitioned queues, many workers per channel.
Common mistakes (and the real outputs)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
send_push() inside place_order() |
ConnectionError: push is down (and customer waited 2.0 s) |
The order depends on every provider | Put a message in a queue; workers send |
| Retry without remembering ids | without ids: ['Your OTP is 482913', 'Your OTP is 482913'] |
At-least-once queues redeliver | Skip message ids already sent |
| Give the retry a new id | with ids: ['Your OTP is 482913', 'Your OTP is 482913'] (Practice 7) |
A new id looks like a new message | Create the id once; reuse it on every retry |
| OTPs and offers in one queue | one queue: OTP waits 1,000 s = 17 minutes |
Urgent waits behind bulk | Separate urgent and bulk queues |
Retry without backoff (wait *= 2 removed) |
Offer for Ravi: try 3 failed, wait 1 s |
A struggling gateway gets hammered at the same pace | Double the wait each time |
| Check quiet hours before "is it an OTP?" | Neha, OTP, 11 pm: hold until 9 am |
The quiet-hours rule caught an urgent message | Let OTPs and alerts through first |
| No TTL on push messages | shown: OTP 482913 two hours later |
The push service keeps messages up to four weeks by default | Set a TTL per kind of message |
The "check quiet hours first" row is the PREFS demo with the quiet-hours check moved above the OTP check:
prefs = {"Neha": {"offers": False}, "Ravi": {"offers": True}}
sent_today = {"Ravi": 3} # offers Ravi already got today
DAILY_CAP = 3 # our own limit
def decide(user, kind, hour):
if not 9 <= hour < 21: # quiet hours checked FIRST, for all
return "hold until 9 am"
if kind == "otp":
return "send now" # OTPs and alerts always go
if not prefs[user]["offers"]:
return "skip: turned off offers"
if sent_today.get(user, 0) >= DAILY_CAP:
return "skip: daily limit reached"
return "send now"
print("Neha, OTP, 11 pm: ", decide("Neha", "otp", 23))
Output:
Neha, OTP, 11 pm: hold until 9 am
Mini project: a tiny notification service
An urgent and a bulk queue for each channel, one worker per channel (one message a second each), retries with exponential backoff and a dead letter queue, message-id dedupe after a worker crash, preferences, quiet hours, a daily cap, templates, and a push service that keeps messages for an off phone only until their TTL. Time is simulated: the clock jumps forward, nothing really sleeps.
# Mini project: a tiny notification service. An urgent and a bulk queue for
# each channel, one worker per channel, retries with exponential backoff and
# a dead letter queue, message-id dedupe, preferences, quiet hours, a daily
# cap and TTL. Time is simulated: the clock jumps, nothing really sleeps.
from collections import deque
MAX_TRIES, DAILY_CAP = 4, 3 # our own numbers
TTL = {"otp": 300, "order": 3600, "offer": 7 * 86400} # seconds (ours)
TEMPLATES = {"otp": "Your OTP is {code}", "order": "Your {item} is {status}",
"offer": "{pct}% off this week"}
CHANNEL = {"otp": "sms", "order": "push", "offer": "sms"}
prefs = {"Neha": False, "Ravi": True, "Asha": True, "Kabir": True,
"Meera": True} # does the user want offers?
def clock(t):
day = "day 2 " if t >= 86_400 else ""
return f"{day}{t // 3600 % 24:02d}:{t // 60 % 60:02d}:{t % 60:02d}"
class Provider: # a fake SMS gateway / FCM
def __init__(self, name, failures=None):
self.name, self.failures = name, failures or {} # msg id -> fails
self.offline, self.stored, self.inbox = set(), [], []
def send(self, msg):
if self.failures.get(msg["id"], 0) > 0:
self.failures[msg["id"]] -= 1
raise TimeoutError(f"{self.name} timed out")
if msg["user"] in self.offline: # kept until its TTL ends
self.stored.append((msg, msg["created"] + msg["ttl"]))
return "stored, phone is off"
self.inbox.append((msg["user"], msg["text"]))
return "delivered"
def phone_online(self, user, now):
self.offline.discard(user)
for msg, expires in [s for s in self.stored if s[0]["user"] == user]:
self.stored.remove((msg, expires))
fate = "shown" if now <= expires else "dropped (expired)"
print(f"{clock(now)} {user}'s phone is back: {fate}: {msg['text']}")
class NotificationService:
def __init__(self, providers):
self.providers, self.now = providers, 0
self.queues = {ch: {"urgent": deque(), "bulk": deque()} for ch in providers}
self.later = [] # (ready time, msg)
self.sent_ids, self.dead_letters, self.sent_today = set(), [], {}
self.crash_once = set() # ids whose worker crashes
def log(self, text): print(f"{clock(self.now)} {text}")
def notify(self, user, kind, msg_id, **fields): # the notification API
msg = {"id": msg_id, "user": user, "kind": kind, "tries": 0,
"text": TEMPLATES[kind].format(**fields),
"created": self.now, "ttl": TTL[kind]}
if kind == "offer": # OTPs and orders always go
day, hour = self.now // 86_400, self.now // 3600 % 24
if not prefs[user]:
return self.log(f"{msg_id}: skip, {user} turned off offers")
if self.sent_today.get((user, day), 0) >= DAILY_CAP:
return self.log(f"{msg_id}: skip, {user}'s daily limit reached")
self.sent_today[(user, day)] = self.sent_today.get((user, day), 0) + 1
if not 9 <= hour < 21: # promotional: 9 am to 9 pm
release = (day + (hour >= 21)) * 86_400 + 9 * 3600
self.later.append((release, msg))
return self.log(f"{msg_id}: hold until 9 am")
self.enqueue(msg)
def enqueue(self, msg):
lane = "bulk" if msg["kind"] == "offer" else "urgent"
self.queues[CHANNEL[msg["kind"]]][lane].append(msg)
def work(self, channel): # one message per second
q = self.queues[channel]
lane = q["urgent"] if q["urgent"] else q["bulk"] # urgent first
if not lane:
return
msg = lane.popleft()
if msg["id"] in self.sent_ids:
return self.log(f"{channel} worker: {msg['id']} already sent, skip")
if self.now > msg["created"] + msg["ttl"]:
return self.log(f"{channel} worker: {msg['id']} expired, drop")
msg["tries"] += 1
try:
result = self.providers[channel].send(msg)
except TimeoutError:
if msg["tries"] < MAX_TRIES:
wait = 2 ** (msg["tries"] - 1) # 1, 2, 4 s: backoff
self.later.append((self.now + wait, msg))
return self.log(f"{channel} worker: {msg['id']} try "
f"{msg['tries']} failed, retry in {wait} s")
self.dead_letters.append(msg)
return self.log(f"{channel} worker: {msg['id']} failed "
f"{MAX_TRIES} times -> dead letter queue")
self.sent_ids.add(msg["id"]) # remember it, then ack
self.log(f"{channel} worker: {msg['id']} {result}: {msg['text']}")
if msg["id"] in self.crash_once: # crash before the ack
self.crash_once.discard(msg["id"])
self.log(f"{channel} worker crashed; queue hands out {msg['id']} again")
lane.appendleft(msg)
def run_until(self, end): # the simulated clock
while self.now < end:
for item in [x for x in self.later if x[0] <= self.now]:
self.later.remove(item)
self.enqueue(item[1])
for channel in self.queues:
self.work(channel)
busy = any(lane for q in self.queues.values() for lane in q.values())
ready = min([t for t, _ in self.later] + [end])
self.now = self.now + 1 if busy else max(self.now + 1, ready)
self.now = end
sms = Provider("SMS gateway", failures={"N1": 2, "C1-Kabir": 9})
push = Provider("push service")
svc = NotificationService({"sms": sms, "push": push})
svc.sent_today[("Ravi", 0)] = 3 # Ravi already got 3 today
svc.crash_once.add("N1")
push.offline.add("Neha") # Neha's phone is off
svc.run_until(19 * 3600) # 7 pm: a big campaign...
for user in prefs:
svc.notify(user, "offer", f"C1-{user}", pct=50)
svc.notify("Neha", "otp", "N1", code=482913) # ...then Neha's OTP
svc.run_until(19 * 3600 + 5)
svc.notify("Neha", "order", "O1", item="masala dosa", status="out for delivery")
svc.run_until(20 * 3600 + 30 * 60)
svc.notify("Neha", "order", "O2", item="refund", status="processed")
svc.run_until(21 * 3600)
push.phone_online("Neha", svc.now) # 9 pm: phone back on
svc.run_until(23 * 3600)
svc.notify("Asha", "offer", "C2-Asha", pct=20) # 11 pm: too late
svc.run_until(86_400 + 9 * 3600 + 5) # next morning
print("Neha's SMS inbox:", [t for u, t in sms.inbox if u == "Neha"])
print("dead letter queue:", [m["id"] for m in svc.dead_letters])
Output:
19:00:00 C1-Neha: skip, Neha turned off offers
19:00:00 C1-Ravi: skip, Ravi's daily limit reached
19:00:00 sms worker: N1 try 1 failed, retry in 1 s
19:00:01 sms worker: N1 try 2 failed, retry in 2 s
19:00:02 sms worker: C1-Asha delivered: 50% off this week
19:00:03 sms worker: N1 delivered: Your OTP is 482913
19:00:03 sms worker crashed; queue hands out N1 again
19:00:04 sms worker: N1 already sent, skip
19:00:05 sms worker: C1-Kabir try 1 failed, retry in 1 s
19:00:05 push worker: O1 stored, phone is off: Your masala dosa is out for delivery
19:00:06 sms worker: C1-Meera delivered: 50% off this week
19:00:07 sms worker: C1-Kabir try 2 failed, retry in 2 s
19:00:09 sms worker: C1-Kabir try 3 failed, retry in 4 s
19:00:13 sms worker: C1-Kabir failed 4 times -> dead letter queue
20:30:00 push worker: O2 stored, phone is off: Your refund is processed
21:00:00 Neha's phone is back: dropped (expired): Your masala dosa is out for delivery
21:00:00 Neha's phone is back: shown: Your refund is processed
23:00:00 C2-Asha: hold until 9 am
day 2 09:00:00 sms worker: C2-Asha delivered: 20% off this week
Neha's SMS inbox: ['Your OTP is 482913']
dead letter queue: ['C1-Kabir']
- Urgent first: the OTP was queued after the campaign, yet it was the first message the SMS worker touched at 19:00:00. While it waited for its retries (1 s, then 2 s), the worker kept sending offers (C1-Asha at 19:00:02), so backoff blocked nobody.
- Preferences and the daily cap: Neha (offers off) and Ravi (3 already today) were skipped at the API, before anything was queued.
- Dedupe: the worker crashed after sending N1 but before the ack; the queue handed N1 out again, and the worker skipped it. Neha's SMS inbox has one OTP.
- Dead letter queue: C1-Kabir failed 4 times (waits 1, 2, 4 s) and was parked; the other offers still went out.
- TTL: Neha's phone was off. At 9 pm the "out for delivery" push (TTL 1 hour, created 19:00:05) was dropped (expired); the refund update from 20:30 was shown.
- Quiet hours: Asha's 11 pm offer was held and sent at 09:00:00 the next morning.
Try extending it: add a normal queue between urgent and bulk; give the SMS worker a rate of 2 messages a second; make the email channel and let users choose channels; and write every attempt to a delivery_log list, then answer "did N1 go out, and on which try?".
Practice
- Homework: in the RETRY demo, try
max_tries = 6. What happens to the OTP and to Ravi's offer? - Homework: in the PREFS demo, add a "no SMS, push only" choice.
- Homework: in the TTL demo, bring the phone back after 3 minutes instead of 2 hours.
- Homework (comment): which messages need both a push and an SMS?
- In the PRIORITY demo, the workers get faster: 5,000 a second. How long does the OTP wait with one queue? Is one queue now fine?
- In the SCALE demo, use 10 crore users instead of 5 crore.
- In the DEDUPE demo, the retry creates a new message id (
N102) instead of reusingN101. What does the phone get? - Why is a dead letter queue better than retrying a failing message forever?
Solution
1. Change def send(msg, max_tries=4): to def send(msg, max_tries=6):. Output:
OTP for Neha: try 1 failed, wait 1 s
OTP for Neha: try 2 failed, wait 2 s
OTP for Neha: delivered on try 3
Offer for Ravi: try 1 failed, wait 1 s
Offer for Ravi: try 2 failed, wait 2 s
Offer for Ravi: try 3 failed, wait 4 s
Offer for Ravi: try 4 failed, wait 8 s
Offer for Ravi: try 5 failed, wait 16 s
Offer for Ravi: moved to the dead letter queue
The OTP is unchanged (it only needed 3 tries). Ravi's offer fails 9 times, so even 6 tries aren't enough: it waits 1, 2, 4, 8, 16 s (31 s in all) and still ends in the dead letter queue. More tries only help for short outages; past a point, park the message.
2. Add an "sms" choice per user and pass the channel to decide:
prefs = {"Neha": {"offers": False, "sms": True},
"Ravi": {"offers": True, "sms": False}} # Ravi: no SMS, push only
sent_today = {"Ravi": 1} # offers Ravi already got today
DAILY_CAP = 3 # our own limit
def decide(user, kind, hour, channel):
if channel == "sms" and not prefs[user]["sms"]:
return "send by push instead" # respect "push only", even for OTPs
if kind == "otp":
return "send now"
if not prefs[user]["offers"]:
return "skip: turned off offers"
if channel == "sms" and not 9 <= hour < 21: # the rule is for SMS
return "hold until 9 am"
if sent_today.get(user, 0) >= DAILY_CAP:
return "skip: daily limit reached"
return "send now"
print("Neha, OTP, SMS, 11 pm: ", decide("Neha", "otp", 23, "sms"))
print("Ravi, OTP, SMS, 11 pm: ", decide("Ravi", "otp", 23, "sms"))
print("Ravi, offer, SMS, 7 pm: ", decide("Ravi", "offer", 19, "sms"))
print("Ravi, offer, push, 7 pm:", decide("Ravi", "offer", 19, "push"))
Output:
Neha, OTP, SMS, 11 pm: send now
Ravi, OTP, SMS, 11 pm: send by push instead
Ravi, offer, SMS, 7 pm: send by push instead
Ravi, offer, push, 7 pm: send now
Ravi said no SMS, so even his OTP is sent by push instead. Neha still gets her OTP by SMS. The 9 am to 9 pm check applies to SMS here, because the TRAI rule is about promotional SMS; many apps also hold offer pushes at night (our choice, not a rule from our sources).
3. Change both calls to phone_comes_online(3, ...) and the labels. Output:
3 minutes later, no TTL:
shown: OTP 482913
shown: Your dosa is out for delivery
shown: Offer: 50% off this week
3 minutes later, with TTL:
shown: OTP 482913
shown: Your dosa is out for delivery
shown: Offer: 50% off this week
After 3 minutes nothing has expired (the OTP's TTL is 5 minutes), so TTL changes nothing: it only drops what is really stale.
4. Messages where missing them is costly and time matters: OTPs (SMS works without the app or the internet), payment and bank alerts, security alerts ("new login from another phone"), and "your delivery is at the door". A push alone fails if the app is uninstalled, notifications are off or the phone has no internet; an SMS costs money for each message, so offers and routine updates usually go by push only (general knowledge).
5. Change rate = 1_000 to rate = 5_000. Output:
one queue: OTP waits 200 s = 3 minutes
two queues: OTP is message number 1
the offers still go out, just after anything urgent
Still 3 minutes, for an OTP that may be valid for only a few. And campaigns grow too: double the campaign and the wait doubles again. Faster workers shrink the wait; only a separate urgent queue removes it.
6. Change users = 50_000_000 to users = 100_000_000. Output:
per day: 500 million notifications
per second, on average: 5,787
at the evening peak: 23,148
Twice the users, twice the load: design for over 23,000 a second at the peak.
7.
queue = [("N101", "Your OTP is 482913")]
queue.append(("N102", queue[0][1])) # the retry made a NEW id
# so the queue hands out a "new" message
def deliver(queue, check_ids):
sent_ids, phone = set(), []
for msg_id, text in queue:
if check_ids and msg_id in sent_ids:
continue # already sent this id: skip it
phone.append(text)
sent_ids.add(msg_id)
return phone
print("with ids: ", deliver(queue, check_ids=True))
Output:
with ids: ['Your OTP is 482913', 'Your OTP is 482913']
Even with id checks on, the phone gets two OTPs: N102 looks like a brand-new message. The id must be created once, when the message is created, and reused on every retry and redelivery.
8. A message that will never succeed (a wrong phone number, a gateway that's down for hours) would keep a worker busy, and keep hitting the provider, for ever. A dead letter queue stops after the last try, frees the worker for everything else, and keeps the message so a person can see why it failed, fix the cause and resend it. Nothing is lost, and nothing is stuck.
Quick quiz
message: "Your order is delivered"
try 1: timed out
try 2: timed out
try 3: timed out
The SMS gateway keeps failing. What should the worker do now?
- A. Retry at once, forever
- B. Crash the worker
- C. Back off, then a dead letter queue
- D. Send it with a new id
Answer
C. Wait longer after each failure so the gateway can recover; after the last try, park the message in the dead letter queue for later. Retrying at once floods the gateway, and a new id could send it twice.
Interview questions: "Design a notification system"
1. Clarify requirements. Which channels (push, SMS, email)? Who sends (any internal service)? Which messages are urgent (OTPs, payment alerts) and which can wait (offers)? How many users and messages a day; what peak? Do users choose what they get? Any legal rules (in India, promotional SMS only from 9 am to 9 pm)? Then state: many channels, any service can ask, respect preferences, OTPs in seconds even during campaigns, retry failures, never send twice, and never slow down the app.
2. Estimate. "5 crore users × 5 notifications a day (our assumptions) is 250 million a day, about 2,900 a second on average; I'll design for a 4x evening peak, about 11,600 a second. If each delivery log record is about 500 bytes (our assumption), that's about 125 GB of logs a day."
3. Components.
- Notification API: one entry point for all services; checks preferences, quiet hours and caps; fills templates; assigns the message id and TTL; returns at once.
- Preference service: per-user choices (offers on or off, channels), read on every non-urgent send.
- Queues: urgent, normal and bulk, per channel, split into partitions.
- Workers per channel: call FCM, APNs, the SMS gateway and the email provider; retry with backoff; dead letter queue.
- Dedupe store: sent message ids, in a fast store with an expiry.
- Delivery log: every attempt and its result, for support and analytics.
4. The hard parts. Never blocking the caller (asynchronous by design); at-least-once queues plus idempotent sending by message id; keeping OTPs fast during a 10 lakh campaign (separate queues and workers); retries that don't flood a struggling provider (backoff); stale messages (TTL); not spamming users (preferences, quiet hours, caps).
5. Scale and reliability (general knowledge). Stateless API servers behind a load balancer; partition queues (for example by user id); add workers per channel as load grows; replicate the queues so a crashed machine loses nothing; rate-limit calls to each provider to stay within its limits; alert when a provider's failure rate or the dead letter queue jumps; keep a second SMS provider to switch to during an outage.
Common follow-ups:
- What if a worker crashes after sending but before the ack? (The queue redelivers; the worker finds the message id in the sent store and skips it.)
- How do you stop a campaign from delaying OTPs? (Separate urgent and bulk queues, with their own workers.)
- The user is offline for a day; what happens to the OTP? (Its TTL, a few minutes, expires; it's dropped rather than shown late. The offer, with a longer TTL, is still delivered.)
- How do you avoid waking people at 3 am? (Quiet hours from the user's time zone; hold non-urgent messages until morning; for promotional SMS in India, 9 am to 9 pm.)
- How does support answer "did my OTP go out?" (Look up the delivery log by user and message id.)
Glossary
| Term | Meaning |
|---|---|
| Notification | A short message a system sends to tell a person something happened, e.g. "out for delivery" |
| Push notification | A message from an app's server that pops up on the phone even when the app is closed |
| SMS | A plain text message over the phone network; works without the app or the internet |
| A message sent to an email address | |
| OTP | One-time password: a short code for one login or payment, valid for a few minutes, e.g. 482913 |
| Channel | One way of reaching a user: push, SMS or email |
| Provider | An outside service that delivers messages on one channel |
| Gateway / SMS gateway | A provider's entry point our system calls to hand over a message |
| Functional / non-functional requirements | What the system must do / how well it must do it (speed, reliability, scale) |
| Synchronous | The caller waits until the work is finished |
| Asynchronous | The caller hands the work over and moves on without waiting |
| ConnectionError | Python's error for a failed network connection |
| Traceback | Python's report of where an error happened, innermost call last |
| Message queue | A waiting line of messages: added at the back, taken from the front later |
| Outbox | The name of the queue in our QUEUE demo |
| deque | Python's double-ended queue: fast append at the back, popleft from the front |
| Worker | A separate program that takes messages from a queue and does the slow work |
| Ack (acknowledgement) | A worker telling the queue "done, delete this message" |
| Timeout / TimeoutError | Giving up waiting for a reply / Python's error for it |
| Retry | Sending the same message again after a failure |
| Exponential backoff | Waiting longer after each failure, usually doubling: 1, 2, 4 s |
| Dead letter queue | Where messages that kept failing are parked for a person to look at |
| At-least-once delivery | The queue hands out every message at least once, maybe more if no ack arrives |
| Message id | A unique name for one message, reused on every retry, e.g. N101 |
| Idempotency | Doing the same thing twice has the same effect as once |
| Dedupe | Dropping repeats of something already handled |
| Campaign | A batch of marketing messages sent to many users at once |
| Lakh / crore | 100,000 / 10 million |
| Priority queue | Several queues, with workers always taking from the most important first |
| Urgent / normal / bulk queue | Queues for OTPs and alerts / ordinary updates / campaigns |
| Backlog | Messages waiting in a queue that haven't been sent yet |
| Preference | A user's choice about which messages they want and how, e.g. offers off |
| Preference service | The part that stores and checks users' preferences |
| Quiet hours | Times of day when non-urgent messages are held, e.g. until 9 am |
| TRAI | Telecom Regulatory Authority of India, which sets telecom rules including for commercial SMS |
| Promotional / transactional | Messages that advertise / messages a user needs because of something they did |
| Rate limit / daily cap | A cap on messages per period / per user per day, e.g. 3 offers a day |
| TTL (time to live) | How long a message stays useful; after that it is dropped |
| FCM | Firebase Cloud Messaging: Google's push service (Android and more) |
| APNs | Apple Push Notification service: Apple's push service for iPhones |
| Push service | A service like FCM or APNs that delivers push messages to phones |
| Template | A stored message with blanks filled in per user, e.g. "Your OTP is {code}" |
| Notification API | The single entry point other services call to send a notification |
| Partition | One slice of a big queue, read separately so many workers run in parallel |
| Delivery log | A record of every message attempt and what happened to it |
| Peak | The busiest period's load, here assumed 4x the average |
| Load balancer | A server that spreads incoming requests across many servers |
Summary
- A notification system must never slow down the app: services drop a note in a message queue and move on (customer waited 2.0 s → 0.0 s), and workers per channel do the sending.
- Providers fail. Workers retry with exponential backoff (1, 2, 4 s) and, after the last try, park the message in a dead letter queue.
- Queues deliver at least once, so a crash can redeliver. A message id, created once and checked before sending, makes delivery idempotent: one OTP, not two.
- Separate urgent and bulk queues: behind a 10 lakh campaign at 1,000 a second, an OTP waits 17 minutes; with an urgent queue it's message number 1.
- Preferences, quiet hours and a daily cap decide before every non-urgent send; under TRAI rules, promotional SMS only from 9 am to 9 pm.
- A TTL drops stale messages; FCM otherwise keeps a message for up to four weeks (settable from 0 to 28 days).
- 5 crore users × 5 a day = 250 million a day = 2,894 a second, 11,574 at a 4x peak: partitioned queues and many workers.
Next episode: you save a photo on your phone and it's on your laptop a second later. Design a file storage service, like Google Drive.
Sources: Firebase docs, "About FCM messages" (fetched 5 Nov 2026), on the message lifespan: "Requests that don't contain this field default to the maximum period of four weeks"; "The value must be a duration from 0 to 2,419,200 seconds (28 days)". MyOperator support, on TRAI rules (fetched 5 Nov 2026): "Promotional SMS messages can only be delivered between 9:00 AM and 9:00 PM." Our own stated assumptions: the provider delays, the failure counts, 1,000 and 5,000 messages a second, the 10 lakh campaign, the daily cap of 3, the TTL values, 5 crore (and 10 crore) users × 5 notifications a day, the 4x peak, the 500-byte log record, and all names. What OTPs, FCM, APNs, TRAI, partitions, promotional vs transactional messages, backoff, dead letter queues, at-least-once delivery and idempotency are is general textbook knowledge.