SyntaxPilot
Courses/System Design/Lesson 16

System Design · Lesson 16 of 40 · 11 min

Message queues and pub/sub

queued

These are the notes for System Design episode 16. Until now every request waited for its answer. Today a ChaiNow order needs four other jobs (SMS, kitchen, loyalty points, invoice email), and calling them one after another makes orders slow and fragile. You'll invent the message queue, scale it with competing consumers, make it survive crashes with acks and the visibility timeout, handle the duplicates that brings with idempotency, and then invent pub/sub, the idea behind Kafka's topics, offsets and consumer groups. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). The sleep times are made-up numbers.

The six problems

Problem What it teaches
Every order waits for email Synchronous calls; the message queue (producer, consumer)
Dinner rush Buffering (load levelling), competing consumers
A worker crashes mid-job Acks, the visibility timeout (SQS: 30 s default)
The same job runs twice At-least-once delivery, idempotency keys
Many services, one event Pub/sub: topics, offsets, consumer groups (Kafka)
Events out of order Partitions; order only within a partition; partition keys

Problem 1: every order waits

Asha orders a chai. The order service calls four services synchronously (one after another, waiting for each): send an SMS, tell the kitchen, add loyalty points, email the invoice. Today the email server is slow (1 s per invoice) and sometimes down.

Demo 1: calling every service in a row

import time
def send_sms(order):     time.sleep(0.1)
def tell_kitchen(order): time.sleep(0.1)
def add_points(order):   time.sleep(0.1)
def email_invoice(order):
    time.sleep(1.0)                     # the email server is slow today
    if order == 3:
        raise ConnectionError("email server not responding")

def place_order(order):
    start = time.perf_counter()
    for step in [send_sms, tell_kitchen, add_points, email_invoice]:
        step(order)                     # wait for each one, in turn
    took = time.perf_counter() - start
    print(f"order {order} confirmed in {took:.1f} s")

for order in [1, 2, 3]:
    place_order(order)

Output:

order 1 confirmed in 1.3 s
order 2 confirmed in 1.3 s
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    place_order(order)
    ~~~~~~~~~~~^^^^^^^
  File "main.py", line 13, in place_order
    step(order)                     # wait for each one, in turn
    ~~~~^^^^^^^
  File "main.py", line 8, in email_invoice
    raise ConnectionError("email server not responding")
ConnectionError: email server not responding

Two problems: the customer waits for the slowest step (1.3 s, for an email she doesn't need right now), and the order fails because of a side job that has nothing to do with making chai. The kitchen was even told already.

The fix: a message queue

In a restaurant In software
The waiter writes a slip, clips it on the rail, walks away Producer: puts a message on the queue
The rail of slips: first in, first cooked Queue: holds messages in order (FIFO, first in first out)
The cook takes the next slip at the kitchen's own speed Consumer / worker: takes messages and does the work

A message is a small piece of data describing a job or an event ("send the invoice for order 3"). A message queue stores messages until a consumer takes them.

Demo 2: a queue and a worker

queue.Queue is Python's thread-safe queue; a thread is a second line of execution running at the same time inside the same program.

import queue, threading, time
jobs = queue.Queue()                    # the order desk's tray

def email_worker():                     # runs on its own, in the back
    while True:
        order = jobs.get()
        time.sleep(0.5)                 # the slow email server
        print(f"  email sent for order {order}")
        jobs.task_done()

threading.Thread(target=email_worker, daemon=True).start()

def place_order(order):
    start = time.perf_counter()
    jobs.put(order)                     # drop a note in the tray
    took = time.perf_counter() - start
    print(f"order {order} confirmed in {took:.2f} s")

for order in [1, 2, 3]:
    place_order(order)
jobs.join()                             # wait, only so the demo can finish

Output:

order 1 confirmed in 0.00 s
order 2 confirmed in 0.00 s
order 3 confirmed in 0.00 s
  email sent for order 1
  email sent for order 2
  email sent for order 3

Orders are confirmed instantly; emails go out in the background. If the email server is down, the notes wait in the queue until it's back. The order service and the email service are now decoupled: neither waits for the other. Work done later, in the background, is called asynchronous processing. (daemon=True means the thread won't keep the program alive on its own; jobs.join() waits until every job is marked task_done().)

Problem 2: dinner rush

Twelve orders land at once. The queue happily holds them all: that's buffering, also called load levelling (a spike of work is spread out over time). But with one worker, the last email waits a long time.

Demo 3: more workers on one queue

import queue, threading, time

def rush(workers, orders=12):
    jobs = queue.Queue()
    for order in range(orders):         # dinner rush: 12 at once
        jobs.put(order)

    def email_worker():
        while True:
            try:
                order = jobs.get_nowait()
            except queue.Empty:
                return                  # tray is empty, go home
            time.sleep(0.2)             # one email takes 0.2 s

    start = time.perf_counter()
    team = [threading.Thread(target=email_worker)
            for _ in range(workers)]
    for t in team: t.start()
    for t in team: t.join()
    took = time.perf_counter() - start
    print(f"{workers} worker(s): 12 emails in {took:.1f} s")

rush(1)
rush(4)

Output:

1 worker(s): 12 emails in 2.4 s
4 worker(s): 12 emails in 0.6 s

Each message goes to exactly one of the workers, so no email is sent twice. This pattern is competing consumers: several workers taking from one queue, like four cooks taking slips from one rail. When the queue grows, add workers (horizontal scaling, episode 5).

Problem 3: a worker crashes mid-job

If a worker removes a message and then crashes, the message is gone forever:

from collections import deque
queue = deque(["order-1", "order-2"])
points = {"asha": 0}

def handle(crash=False):
    msg = queue.popleft()           # removed the moment we take it
    if crash:
        print(f"{msg}: CRASH")      # ...and gone with the worker
        return
    points["asha"] += 10
    print(f"{msg}: +10")

handle(crash=True)
handle()
print("left in queue:", list(queue), "| asha's points:", points["asha"], "(2 orders = 20)")
order-1: CRASH
order-2: +10
left in queue: [] | asha's points: 10 (2 orders = 20)

Better: when a worker takes a message, the queue only hides it for a while. The worker deletes it only when the job is finished. That delete is an acknowledgement (ack). If no ack arrives in time, the message reappears for another worker.

From the Amazon SQS Developer Guide ("Amazon SQS visibility timeout"):

  • "The default visibility timeout for a queue is 30 seconds", with "a maximum limit of 12 hours".
  • "During this period, you're expected to process and delete the message."
  • "If you don't delete it before the timeout expires, the message becomes visible again in the queue and can be retrieved by another consumer."

Demo 4: acks and the visibility timeout

queue maps each message to the time it becomes visible again. receive hides a message for 30 s; handle gives Asha 10 points, and only then deletes the message.

TIMEOUT = 30                        # SQS default visibility timeout: 30 s

def receive(now):                   # take a message, hide it for 30 s
    for msg, visible_at in queue.items():
        if visible_at <= now:
            queue[msg] = now + TIMEOUT
            return msg

def handle(now, crash=False, careful=False):
    msg = receive(now)
    if careful and msg in done:
        print(f"  t={now:2}: {msg} already done, skip")
    else:
        points["asha"] += 10        # give loyalty points
        done.add(msg)
        if crash:
            print(f"  t={now:2}: {msg} +10, then CRASH before delete")
            return
        print(f"  t={now:2}: {msg} +10")
    del queue[msg]                  # delete = "I'm finished with it"

for careful in [False, True]:
    print("skip orders already done:", careful)
    queue = {"order-1": 0, "order-2": 0}   # message -> visible at (s)
    points, done = {"asha": 0}, set()
    handle(0, crash=True, careful=careful)
    handle(1, careful=careful)
    handle(31, careful=careful)     # order-1 is visible again
    print("  asha's points:", points["asha"], "(2 orders = 20)")

Output:

skip orders already done: False
  t= 0: order-1 +10, then CRASH before delete
  t= 1: order-2 +10
  t=31: order-1 +10
  asha's points: 30 (2 orders = 20)
skip orders already done: True
  t= 0: order-1 +10, then CRASH before delete
  t= 1: order-2 +10
  t=31: order-1 already done, skip
  asha's points: 20 (2 orders = 20)

First block: nothing is lost (order-1 came back at t = 31), but it ran twice: 30 points for 2 orders. This is unavoidable: the worker crashed after adding points but before deleting, and the queue can't know how far it got. SQS's guide: "due to the at-least-once delivery model of Amazon SQS, there's no absolute guarantee that a message won't be delivered more than once."

Delivery guarantee Meaning
At-most-once Never twice, but may be lost (delete on receive)
At-least-once Never lost, but may come twice (delete after work)
Exactly-once (effectively) At-least-once + an idempotent consumer

Problem 4: the same job runs twice (idempotency)

At home In a consumer
The electricity bill is delivered twice Duplicate message (at-least-once)
Check: bill no. 4417 paid? Yes, on Monday Idempotency key: a unique id per job, like the order id
Don't pay it again: twice = once Idempotent consumer: doing it twice has the effect of once

Second block of Demo 4: with careful=True, the worker skips order ids in done, and Asha has the correct 20 points. In a real system, store done in the database, in the same transaction as the points, so the two can't get out of step.

At-least-once delivery + an idempotent consumer behaves like exactly once. That's the standard recipe.

Problem 5: many services, one event (pub/sub)

Now SMS, kitchen and email all need every order. If they share one queue, each message goes to only one of them (that was the point of competing consumers). We need every subscriber to get every message, like a newspaper where every subscriber gets every edition: publish/subscribe (pub/sub).

Demo 5: one queue vs one topic

A topic keeps a log (an append-only list) of every event, plus an offset for each group: that group's place in the log.

from collections import deque
events = ["order 1", "order 2", "order 3"]

print("one queue, three services take turns:")
q = deque(events)
for service in ["sms", "kitchen", "email"]:
    print(f"  {service:7} got", q.popleft())

class Topic:                        # a log that keeps every event
    def __init__(self):
        self.log, self.offset = [], {}   # offset: each group's place
    def publish(self, event):
        self.log.append(event)
    def poll(self, group):
        start = self.offset.get(group, 0)
        self.offset[group] = len(self.log)
        return self.log[start:]

print("one topic, every group reads it all:")
orders = Topic()
for e in events:
    orders.publish(e)
for group in ["sms", "kitchen", "email"]:
    print(f"  {group:7} got", orders.poll(group))
orders.publish("order 4")
print("  kitchen then got", orders.poll("kitchen"))

Output:

one queue, three services take turns:
  sms     got order 1
  kitchen got order 2
  email   got order 3
one topic, every group reads it all:
  sms     got ['order 1', 'order 2', 'order 3']
  kitchen got ['order 1', 'order 2', 'order 3']
  email   got ['order 1', 'order 2', 'order 3']
  kitchen then got ['order 4']

How Apache Kafka does it

From the Kafka documentation (Introduction):

  • "Kafka is a distributed, partitioned, replicated commit log service." "Messaging traditionally has two models: queuing and publish-subscribe."
  • "Consumers label themselves with a consumer group name, and each message published to a topic is delivered to one consumer instance within each subscribing consumer group."
  • Same group: "this works just like a traditional queue", balancing load over the consumers. Different groups: "this works like publish-subscribe".
  • Each message has "a sequential id number called the offset".
  • "The Kafka cluster retains all published messages, whether or not they have been consumed, for a configurable period of time."

So within a group it's a queue (competing consumers), and across groups it's pub/sub. Because the log is kept, a group that joins late can read from the start (practice 3).

Problem 6: events out of order (partitions and keys)

One log can't carry all of ChaiNow's events, so a topic is split into partitions: several logs read in parallel. Kafka: "Each partition is an ordered, immutable sequence of messages that is continually appended to, a commit log." But: "Kafka only provides a total order over messages within a partition, not between different partitions in a topic."

If one order's events (placed, paid, delivered) are spread across partitions, a consumer can see delivered before paid. The fix: the producer chooses the partition by key. Kafka's docs: the producer can choose "in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the message)." Key = order id → one order's events land in one partition → in order. Different orders still run in parallel. One more limit from the docs: "there cannot be more consumer instances in a consumer group than partitions."

order-1: placed, paid, delivered  -> partition 1 (in order)
order-4: placed, paid             -> partition 0 (in order)

The whole chain

  1. Calling services in a row → slow (1.3 s), and one failure breaks the order.
  2. Queue + worker → confirmed in 0.00 s; services decoupled.
  3. Rush → the queue buffers; competing consumers drain it (2.4 s → 0.6 s).
  4. Crash → ack after the work; the visibility timeout (SQS default 30 s) brings the message back → duplicates.
  5. Duplicates → idempotency key: skip if done. At-least-once + idempotent = effectively once.
  6. Everyone needs it → pub/sub topics with offsets and consumer groups; partition keys keep one order's events in order.

Common mistakes (and the real errors)

Mistake Code Error / result Fix
Side jobs called in a row for step in [...]: step(order) ConnectionError: email server not responding (order fails) Put side jobs on a queue
Deleting on receive msg = queue.popleft() then crash left in queue: [], points 10 for 2 orders Delete only after the work (ack)
Consumer not idempotent points["asha"] += 10, no check asha's points: 30 (2 orders = 20) Idempotency key: skip if done
Several services share one queue sms, kitchen, email on one queue kitchen got order 2 (sms never saw it) A topic, one group per service
Visibility timeout shorter than the job 30 s timeout, 45 s job the message is processed twice Timeout longer than the job; stay idempotent
No partition key one order's events round-robin delivered before paid Key by order id

Mini project: a tiny Kafka-style broker

Partitions chosen by key (zlib.crc32, a stable hash), offsets per (group, partition), commit only after processing, and an idempotent kitchen.

# Mini project: a tiny Kafka-style broker: partitions by key, consumer groups with offsets, idempotent consumers
import zlib

class Topic:
    def __init__(self, partitions=2):
        self.parts = [[] for _ in range(partitions)]   # each partition: an append-only log
        self.offsets = {}                              # (group, partition) -> next index to read

    def publish(self, key, event):
        p = zlib.crc32(key.encode()) % len(self.parts)  # same key -> same partition
        self.parts[p].append((key, event))
        return p

    def poll(self, group, p):
        start = self.offsets.get((group, p), 0)
        return start, self.parts[p][start:]

    def commit(self, group, p, next_offset):          # "I've finished up to here"
        self.offsets[(group, p)] = next_offset

orders = Topic(partitions=2)
for key, event in [("order-1", "placed"), ("order-4", "placed"), ("order-1", "paid"),
                   ("order-4", "paid"), ("order-1", "delivered")]:
    print(f"{key} {event:9} -> partition {orders.publish(key, event)}")

done = set()                                          # the kitchen's idempotency keys
def kitchen(p, crash_after=None):
    start, batch = orders.poll("kitchen", p)
    for i, (key, event) in enumerate(batch):
        if (key, event) in done:
            print(f"  p{p}: skip {key} {event} (already done)")
            continue
        print(f"  p{p}: {key} {event}")
        done.add((key, event))
        if crash_after is not None and i == crash_after:
            print("  CRASH before commit")
            return
    orders.commit("kitchen", p, start + len(batch))

print("kitchen reads partition by partition:")
for p in range(2):
    kitchen(p, crash_after=1 if p == 1 else None)
print("kitchen restarts and reads again:")
for p in range(2):
    kitchen(p)
print("email group, reading for the first time:", [e for p in range(2) for e in orders.poll("email", p)[1]])

Output:

order-1 placed    -> partition 1
order-4 placed    -> partition 0
order-1 paid      -> partition 1
order-4 paid      -> partition 0
order-1 delivered -> partition 1
kitchen reads partition by partition:
  p0: order-4 placed
  p0: order-4 paid
  p1: order-1 placed
  p1: order-1 paid
  CRASH before commit
kitchen restarts and reads again:
  p1: skip order-1 placed (already done)
  p1: skip order-1 paid (already done)
  p1: order-1 delivered
email group, reading for the first time: [('order-4', 'placed'), ('order-4', 'paid'), ('order-1', 'placed'), ('order-1', 'paid'), ('order-1', 'delivered')]

Each order's events stay in order inside its partition. The crash before the commit made partition 1 be read again from the start, and the idempotency keys skipped what was already done. The email group, joining late, still reads everything. Extend it: give the kitchen two consumers, one per partition.

Practice

  1. Homework: in Demo 3, try 2, 8 and 12 workers.
  2. Homework: in Demo 4, make order-2 crash too (at t = 1). What happens with and without careful?
  3. Homework: in Demo 5, add a points group after order 4 is published. What does its first poll return?
  4. A worker takes 45 s per message on a queue with a 30 s visibility timeout. What goes wrong, and give two fixes.
  5. Which needs a queue and which needs pub/sub: (a) resizing every uploaded photo, (b) telling search, analytics and recommendations that a new restaurant joined?
  6. Why can't a Kafka consumer group usefully have more consumers than partitions?
Solutions and hints

1. Replace the last two lines with for w in [2, 8, 12]: rush(w):

2 worker(s): 12 emails in 1.2 s
8 worker(s): 12 emails in 0.4 s
12 worker(s): 12 emails in 0.2 s

8 workers take 0.4 s, not 0.3: 12 jobs over 8 workers is two rounds. Doubling workers stops helping once each has one job.

2. Change handle(1, careful=careful) to handle(1, crash=True, careful=careful) and add handle(32, careful=careful):

skip orders already done: False
  t= 0: order-1 +10, then CRASH before delete
  t= 1: order-2 +10, then CRASH before delete
  t=31: order-1 +10
  t=32: order-2 +10
  asha's points: 40 (2 orders = 20)
skip orders already done: True
  t= 0: order-1 +10, then CRASH before delete
  t= 1: order-2 +10, then CRASH before delete
  t=31: order-1 already done, skip
  t=32: order-2 already done, skip
  asha's points: 20 (2 orders = 20)

3. Add print(" points (joined late) got", orders.poll("points")) at the end:

  points (joined late) got ['order 1', 'order 2', 'order 3', 'order 4']

A new group starts at offset 0 and reads the whole kept log: you can add a new service later and replay history. A plain queue can't do this, because messages are deleted once consumed.

4. At t = 30 the message becomes visible again and a second worker starts it: it's processed twice, and both may fail to delete in time, forever. Fixes: set the visibility timeout longer than the job (SQS allows up to 12 hours), extend it while working (SQS's ChangeMessageVisibility), and keep the consumer idempotent anyway.

5. (a) A queue: each photo needs resizing once, by any worker. (b) Pub/sub: three different services each need every event.

6. Within a group, each partition is read by one consumer, so an extra consumer gets no partition and sits idle (Kafka's docs: "there cannot be more consumer instances in a consumer group than partitions").

Quick quiz

# SQS queue, visibility timeout = 30 s
# a worker takes 'order-9' at t = 0
# and is still working at t = 45

# what happens at t = 30?
  • A. Nothing, it's in progress
  • B. order-9 is deleted
  • C. order-9 is visible again
  • D. The worker gets an error
Answer

C. No delete within 30 s, so the message becomes visible again and another worker can take it: a duplicate. Set the timeout longer than the job, and keep the consumer idempotent.

Interview questions

  • Why put a queue between two services? (Decoupling: the caller doesn't wait; buffering spikes; the consumer can be down or slow.)
  • What are competing consumers? (Several workers on one queue; each message goes to one of them.)
  • How does a queue avoid losing a message when a worker crashes? (Ack after processing; a visibility timeout makes unacked messages reappear.)
  • At-most-once vs at-least-once vs exactly-once? (May lose / may duplicate / at-least-once + idempotency.)
  • What is an idempotency key, and where do you store it? (A unique job id; in the database, in the same transaction as the effect.)
  • Queue vs pub/sub? (One consumer per message vs every subscriber gets every message.)
  • How do Kafka consumer groups combine both? (Queue within a group, pub/sub across groups.)
  • How do you keep one order's events in order in Kafka? (Same key → same partition; order is only guaranteed within a partition.)

Glossary

Term Meaning
Synchronous call The caller waits for the answer before moving on
Asynchronous processing Work done later, in the background, without the caller waiting
Message A small piece of data describing a job or event
Message queue Stores messages until a consumer takes them (usually first in, first out)
Producer Puts messages on a queue or topic
Consumer / worker Takes messages and does the work
FIFO First in, first out
Decoupled Two services that don't wait for or depend on each other directly
Thread A second line of execution inside the same program
queue.Queue Python's thread-safe queue: put, get, task_done, join
Buffering / load levelling A queue absorbing a spike so work is spread out over time
Competing consumers Several workers on one queue; each message goes to one
Ack (acknowledgement) The consumer telling the queue "done", so it can delete the message
Visibility timeout How long a received message stays hidden before it reappears (SQS default 30 s, max 12 h)
At-most-once A message is never delivered twice but may be lost
At-least-once A message is never lost but may be delivered more than once
Exactly-once (effectively) At-least-once delivery plus an idempotent consumer
Idempotent Doing it twice has the same effect as doing it once
Idempotency key A unique id per job (like the order id) used to skip repeats
Pub/sub (publish/subscribe) Every subscriber gets every message
Topic A named stream of messages in pub/sub (Kafka's word)
Log / commit log An append-only list of events, kept in order
Offset A message's position in a partition; a group's offset is its place in the log
Consumer group Kafka: consumers sharing a group name share the work; each group gets every message
Partition One of several logs a topic is split into, read in parallel
Partition key The value (like order id) used to choose the partition, so related events stay together
Retention How long Kafka keeps messages, whether or not they've been read
Commit (offset) A consumer recording how far it has processed
zlib.crc32 A fast, stable hash; same input, same number on every run
Amazon SQS Amazon Simple Queue Service, a managed message queue
Apache Kafka A distributed, partitioned, replicated commit log used for pub/sub and streaming

Summary

  • Calling side jobs in a row made every order wait 1.3 s and fail when email was down.
  • A queue decouples services: orders confirmed in 0.00 s, emails sent in the background.
  • Queues buffer rushes; competing consumers drain them (12 emails: 2.4 s → 0.6 s).
  • Ack after the work and a visibility timeout (SQS: 30 s default) mean no lost messages, but at-least-once delivery brings duplicates.
  • Idempotency keys make duplicates harmless: at-least-once + idempotent = effectively once.
  • Pub/sub gives every group every event; Kafka is a queue within a consumer group and pub/sub across groups, keeps its log for a set time, and orders messages only within a partition, so key by order id.

Next episode: SMS, kitchen, points, email: should they really be separate programs? Monolith vs microservices.