SyntaxPilot
Courses/System Design/Lesson 6

System Design · Lesson 6 of 40 · 11 min

Load balancers

queued

These are the notes for System Design episode 6. Last episode, we scaled out to many stateless servers, but we picked s1, s2 and s3 by hand. Here you invent the thing that does it for real: the load balancer. We try handing out every address through DNS (it fails), build one front door, teach it to choose with round robin and least connections, skip dead servers with health checks, meet sticky sessions, protect the balancer itself with a standby pair, and compare layer 4 and layer 7 balancers. Every Python demo below is self-contained and runs as-is; the outputs shown are real (Python 3.13). The 203.0.113.x addresses are from the range reserved for documentation (RFC 5737), not real servers.

The six problems

Problem What it teaches
Give users every address? Why DNS alone isn't enough
One front door The load balancer
Take turns Round robin
Equal turns, unequal work Least connections
A server dies quietly Health checks
The balancer dies A standby pair; layer 4 vs layer 7

Problem 1: give users every address?

ChaiNow now runs on three stateless servers, each with its own IP address. But users only know one name: chainow.in. In episode 3, DNS turned a name into an address. So which of the three should it give? The obvious idea: let DNS hand out all three, in turns, and each phone uses the one it got.

Demo 1: DNS only, one server dead

# plan A: no balancer. DNS hands out all three addresses, in turns
servers = ["203.0.113.11", "203.0.113.12", "203.0.113.13"]
alive = {"203.0.113.11": True, "203.0.113.12": False, "203.0.113.13": True}

for phone in range(1, 7):
    ip = servers[(phone - 1) % 3]     # this phone got this address...
    status = "ok" if alive[ip] else "FAILED"
    print(f"phone {phone} -> {ip} {status}  (cached for 300 s)")

Output:

phone 1 -> 203.0.113.11 ok  (cached for 300 s)
phone 2 -> 203.0.113.12 FAILED  (cached for 300 s)
phone 3 -> 203.0.113.13 ok  (cached for 300 s)
phone 4 -> 203.0.113.11 ok  (cached for 300 s)
phone 5 -> 203.0.113.12 FAILED  (cached for 300 s)
phone 6 -> 203.0.113.13 ok  (cached for 300 s)
  • (phone - 1) % 3 gives 0, 1, 2, 0, 1, 2: the addresses handed out in turns.
  • Phones 2 and 5 got the dead server .12, and they fail.
  • The real problem is the cache. Remember TTL from episode 3? Those phones cached that answer, here for 300 seconds. So they keep trying the dead server for five minutes, and DNS can't take it back. One in three users is stuck.
  • Why did it fail? The decision was made once, far away, and then frozen inside every phone. We need someone who decides fresh, for every single request.

Problem 2: one front door (the load balancer)

At a hospital In your system
One main entrance: everyone walks in here One public address: DNS points only here
The token counter: "go to room 3" Load balancer: picks a server for every request, fresh each time
The doctors' rooms: patients never pick Backend servers: hidden behind the balancer; users never see them

If one room closes, the counter just stops sending people there.

users (phones)  →  load balancer (one address)  →  s1 · s2 · s3 (servers)
  • Users only ever talk to one address: the balancer's.
  • It picks a server for every request.
  • Servers can be added or removed behind it without users noticing.

The AWS Elastic Load Balancing docs put it simply: "A load balancer distributes incoming traffic across targets." (A target is one of the servers behind it.)

Now the real question: how should it pick?

Problem 3: take turns (round robin)

The simplest rule: take turns. s1, then s2, then s3, then back to s1.

Demo 2: round robin

from itertools import cycle

servers = ["s1", "s2", "s3"]
next_server = cycle(servers)      # s1, s2, s3, s1, s2, s3, ...

count = {s: 0 for s in servers}
for request in range(9000):
    count[next(next_server)] += 1

print("round robin:", count)

Output:

round robin: {'s1': 3000, 's2': 3000, 's3': 3000}
  • itertools.cycle(servers) repeats the list for ever; next(...) takes the next item each time.
  • 9,000 requests, exactly 3,000 each. Perfectly fair. This rule is called round robin.
  • It's so common that nginx, a very popular load balancer, uses it by default: "When the load balancing method is not specifically configured, it defaults to round-robin."
  • But fair counts aren't the same as fair work. Not every request is the same size.

Problem 4: equal turns, unequal work

A bank has three counters, and a guard sends customers to them in turns. By bad luck, every third customer has a big job (a loan application that takes 30 minutes), and they all land on counter 3. Counter 3's line keeps growing, while counters 1 and 2 finish quickly and sit idle.

Demo 3: round robin vs least connections

from itertools import cycle

# how long each request takes, in seconds: every 3rd one is a big report
jobs = [1, 1, 9] * 4
servers = ["s1", "s2", "s3"]

# round robin: take turns, no matter how busy a server is
busy = {s: 0 for s in servers}
for job, s in zip(jobs, cycle(servers)):
    busy[s] += job
print("round robin      :", busy)

# least connections: give the job to the server with the least work
busy = {s: 0 for s in servers}
for job in jobs:
    s = min(busy, key=busy.get)
    busy[s] += job
print("least connections:", busy)

Output:

round robin      : {'s1': 4, 's2': 4, 's3': 36}
least connections: {'s1': 20, 's2': 13, 's3': 11}
  • [1, 1, 9] * 4 makes 12 jobs: most take 1 second, every third is a 9-second report.
  • zip(jobs, cycle(servers)) pairs each job with the next server in turn.
  • Round robin: s1 and s2 get 4 seconds of work each, s3 gets 36. Every big report landed on the same server, so its users wait nine times longer.
  • min(busy, key=busy.get) picks the server whose value in busy is smallest: the least busy one.
  • Smarter rule: the busiest server has 20 seconds of work instead of 36.
  • Real balancers measure "busy" by counting open connections, so this rule is called least connections. nginx describes it as: the "next request is assigned to the server with the least number of active connections". A connection that stays open usually means work in progress, so fewer connections is a good sign of a less busy server.
  • To keep the demo short, busy only ever grows (jobs never finish). A real balancer's count goes down again as each request completes.

Problem 5: a server dies quietly

Back to round robin. But now server s2 has crashed, and the balancer doesn't know.

Demo 4: round robin sends users to a dead server

from itertools import cycle

servers = ["s1", "s2", "s3"]
alive = {"s1": True, "s2": False, "s3": True}    # s2 has crashed

def send(server):
    if not alive[server]:
        raise ConnectionError(f"{server} is not responding")
    return "200 OK"

for request, server in zip(range(1, 7), cycle(servers)):
    try:
        print(f"request {request} -> {server}: {send(server)}")
    except ConnectionError as e:
        print(f"request {request} -> {server}: 502 Bad Gateway ({e})")

Output:

request 1 -> s1: 200 OK
request 2 -> s2: 502 Bad Gateway (s2 is not responding)
request 3 -> s3: 200 OK
request 4 -> s1: 200 OK
request 5 -> s2: 502 Bad Gateway (s2 is not responding)
request 6 -> s3: 200 OK
  • send raises a ConnectionError if the server is down; try / except catches it and shows what the user would get.
  • 200 OK is the HTTP status for success. 502 Bad Gateway is the status a balancer returns when the server behind it fails.
  • Requests 2 and 5 hit s2 and fail. Round robin keeps faithfully sending one in three users to a dead server, because taking turns doesn't care whether a server is alive. The balancer needs to check.

The nurse who checks the rooms

At the hospital In the balancer
A nurse checks each room every few minutes Health check: ask every server "are you OK?", often on a special address like /health, every few seconds
Doctor not answering: stop sending patients Mark unhealthy: take it out of the list
Doctor back: the room reopens Mark healthy again: put it back in

Demo 5: health checks

from itertools import cycle

servers = ["s1", "s2", "s3"]
alive = {"s1": True, "s2": False, "s3": True}    # s2 has crashed

def health_check(server):        # the balancer pings /health
    return alive[server]

healthy = [s for s in servers if health_check(s)]
print("healthy:", healthy)

for request, server in zip(range(1, 7), cycle(healthy)):
    print(f"request {request} -> {server}: 200 OK")

Output:

healthy: ['s1', 's3']
request 1 -> s1: 200 OK
request 2 -> s3: 200 OK
request 3 -> s1: 200 OK
request 4 -> s3: 200 OK
request 5 -> s1: 200 OK
request 6 -> s3: 200 OK
  • Before sending anything, the balancer checks each server and keeps only the healthy ones. Then it does round robin over the healthy list.
  • A crashed server, and no user noticed.
  • Real balancers do this all the time. AWS Elastic Load Balancing "monitors the health of its registered targets and ensures that it routes traffic only to healthy targets". nginx, by default, marks a server failed after 1 failed attempt (max_fails "By default ... set to 1") and skips it for 10 seconds (fail_timeout "By default, the parameter is set to 10 seconds") before trying again.

Problem 6: sticky sessions

Some older apps aren't stateless, so they want each user to always land on the same server. That's called a sticky session. A simple way: hash the user and take it modulo (%) the number of servers. A hash turns any text into a number, always the same number for the same text. (nginx offers a method like this, called ip-hash, which uses the user's IP address.)

Demo 6: sticky by hashing, then remove a server

import zlib

users = [f"user{i}" for i in range(1, 1001)]

def pick(user, n):            # sticky: hash, then mod n
    return zlib.crc32(user.encode()) % n

before = {u: pick(u, 3) for u in users}   # 3 servers
after = {u: pick(u, 2) for u in users}    # one server removed

moved = sum(before[u] != after[u] for u in users)
print("user1 goes to server", before["user1"], "every time")
print(f"after removing a server, {moved} of 1000 users moved")

Output:

user1 goes to server 1 every time
after removing a server, 649 of 1000 users moved
  • zlib.crc32 is a quick hash function; it needs bytes, so user.encode() turns the text into bytes first. % n turns the big number into a server number from 0 to n−1.
  • user1 goes to server 1 every time. Sticky.
  • But after removing one server, 649 of 1000 users (about 65%) moved. If their session lived on the old server, they're logged out.
  • That's why last episode's fix, stateless servers with a shared store, is the better answer. There's also a smarter way to hash so only a few users move: consistent hashing, which gets its own episode later in the course.

Problem 7: the balancer dies

Now every request passes through one load balancer. If the token counter closes, the doctors are all fine, but nobody gets in. We just built a new single point of failure, the very thing episode 5 warned us about.

The fix: run balancers in pairs. A standby balancer watches the main one, and if it dies, the standby takes over the same public address.

Layer 4 vs layer 7

Layer 4 · transport Layer 7 · application
Example AWS Network Load Balancer AWS Application Load Balancer, nginx
Sees IP address + port The whole HTTP request
Reads Not the HTTP request Path, headers, cookies
Speaks TCP, UDP, TLS HTTP, HTTPS
Chooses By connection By content, e.g. /api → API servers
Good at Raw speed, any protocol Smart routing
Like A guard who only sees your ticket number A receptionist who reads your slip

From the AWS docs: the Network Load Balancer "Operates at the transport layer (layer 4) and supports TCP, TLS, UDP, and QUIC"; the Application Load Balancer "Operates at the application layer (layer 7) and supports HTTP and HTTPS". A layer 7 balancer can send /api to API servers and /images somewhere else.

The whole chain

  1. DNS hands out all IPs → cached, stuck on dead ones
  2. One front door → the load balancer
  3. Take turns → round robin
  4. Big jobs pile up → least connections
  5. Dead server, 502s → health checks
  6. Same server every time → sticky sessions (prefer stateless)
  7. Balancer dies → a standby pair

Common mistakes (and the real errors)

Mistake Code Error Fix
Every server failed its health check healthy = [] then next(cycle(healthy)) StopIteration Alert a human, and return a clear 503 instead of crashing
Least connections with zero servers to pick min({}, key=busy.get) # no servers ValueError: min() iterable argument is empty Check the server list isn't empty first
Hashing text instead of bytes zlib.crc32("user1") % 3 TypeError: a bytes-like object is required, not 'str' Encode first: zlib.crc32("user1".encode())

The first one in full:

from itertools import cycle
healthy = []
next(cycle(healthy))
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    next(cycle(healthy))
    ~~~~^^^^^^^^^^^^^^^^
StopIteration

Misconceptions

  • "DNS can balance the load on its own." It can spread users out, but answers are cached, so phones stay stuck on a dead server until the TTL runs out.
  • "Round robin is fair, so it's balanced." It gives equal turns, not equal work. Big jobs can pile up on one server.
  • "A balancer knows when a server dies." Only if it runs health checks. Without them, it keeps sending users to the dead server (502s).
  • "Sticky sessions are a good way to keep users logged in." Removing or adding a server moves most users. Stateless servers with a shared store are better.
  • "Adding a load balancer removes every single point of failure." One balancer is itself a single point of failure. Run a standby pair.

Mini project: a tiny load balancer

A LoadBalancer class that combines the episode's ideas: health checks, two ways to pick (round robin or least connections), and a clear 503 when nothing is healthy.

# a tiny load balancer: health checks + round robin or least connections
from itertools import cycle

class LoadBalancer:
    def __init__(self, servers):
        self.servers = servers
        self.alive = {s: True for s in servers}
        self.busy = {s: 0 for s in servers}      # seconds of work in progress

    def health_check(self):
        return [s for s in self.servers if self.alive[s]]

    def route(self, jobs, method):
        healthy = self.health_check()
        if not healthy:
            return "503 Service Unavailable: no healthy servers"
        self.busy = {s: 0 for s in healthy}
        turns = cycle(healthy)
        for job in jobs:
            if method == "round robin":
                s = next(turns)
            else:                                  # least connections
                s = min(self.busy, key=self.busy.get)
            self.busy[s] += job
        return self.busy

lb = LoadBalancer(["s1", "s2", "s3", "s4"])
jobs = [1, 1, 1, 9] * 3

print("round robin      :", lb.route(jobs, "round robin"))
print("least connections:", lb.route(jobs, "least connections"))

lb.alive["s2"] = False                             # s2 crashes
print("s2 down, healthy :", lb.health_check())
print("least connections:", lb.route(jobs, "least connections"))

for s in lb.servers:                               # everything is down
    lb.alive[s] = False
print("all down         :", lb.route(jobs, "round robin"))

Output:

round robin      : {'s1': 3, 's2': 3, 's3': 3, 's4': 27}
least connections: {'s1': 11, 's2': 4, 's3': 12, 's4': 9}
s2 down, healthy : ['s1', 's3', 's4']
least connections: {'s1': 10, 's3': 15, 's4': 11}
all down         : 503 Service Unavailable: no healthy servers
  • With four servers and every fourth job big, round robin piles all the big jobs (27 seconds) on s4. Least connections keeps the busiest at 12.
  • When s2 crashes, the health check drops it and the work is shared by the other three.
  • With no healthy server, route checks first and returns a clear 503 instead of crashing with StopIteration or ValueError.

Practice

  1. In Demo 3, change jobs to [9, 1, 1] * 4. Predict which server round robin overloads, then run it.
  2. In Demo 5, bring s2 back to life. What changes?
  3. In Demo 6, change pick(u, 2) to pick(u, 4) (adding a fourth server instead of removing one). How many users move?
  4. Round robin with health checks over s1, s2, s3, s4; s2 is unhealthy. Write code that prints where requests 1 to 5 go.
  5. Write a route(requests, healthy) function that returns a 503 for every request when healthy is empty, instead of raising StopIteration.
  6. Would you pick a layer 4 or a layer 7 balancer for a video call app? Why?
Solutions and hints

1. Now the big job is first in each group of three, so round robin gives every 9 to s1.

from itertools import cycle

# how long each request takes, in seconds: every 3rd one is a big report
jobs = [9, 1, 1] * 4
servers = ["s1", "s2", "s3"]

# round robin: take turns, no matter how busy a server is
busy = {s: 0 for s in servers}
for job, s in zip(jobs, cycle(servers)):
    busy[s] += job
print("round robin      :", busy)

# least connections: give the job to the server with the least work
busy = {s: 0 for s in servers}
for job in jobs:
    s = min(busy, key=busy.get)
    busy[s] += job
print("least connections:", busy)
round robin      : {'s1': 36, 's2': 4, 's3': 4}
least connections: {'s1': 13, 's2': 19, 's3': 12}

Round robin's worst server has 36 seconds; least connections' has 19.

2. Change the alive line to alive = {"s1": True, "s2": True, "s3": True} # s2 is back:

healthy: ['s1', 's2', 's3']
request 1 -> s1: 200 OK
request 2 -> s2: 200 OK
request 3 -> s3: 200 OK
request 4 -> s1: 200 OK
request 5 -> s2: 200 OK
request 6 -> s3: 200 OK

s2 passes its health check again, so it goes back into the list and gets its share.

3. Change the after line to after = {u: pick(u, 4) for u in users} # one server added (and the message to "after adding a server"):

user1 goes to server 1 every time
after adding a server, 742 of 1000 users moved

Adding a server is just as bad: about three quarters of users move. Plain hash % n reshuffles almost everyone whenever n changes. That is the problem consistent hashing solves.

4.

from itertools import cycle

servers = ["s1", "s2", "s3", "s4"]
alive = {"s1": True, "s2": False, "s3": True, "s4": True}

healthy = [s for s in servers if alive[s]]
for request, server in zip(range(1, 6), cycle(healthy)):
    print(f"request {request} -> {server}")
request 1 -> s1
request 2 -> s3
request 3 -> s4
request 4 -> s1
request 5 -> s3

This is the quiz below: request 4 goes to s1.

5.

from itertools import cycle

def route(requests, healthy):
    if not healthy:
        return ["503 Service Unavailable"] * requests
    turns = cycle(healthy)
    return [f"{next(turns)}: 200 OK" for _ in range(requests)]

print(route(3, ["s1", "s3"]))
print(route(3, []))
['s1: 200 OK', 's3: 200 OK', 's1: 200 OK']
['503 Service Unavailable', '503 Service Unavailable', '503 Service Unavailable']

if not healthy is true for an empty list, so we never call next on an empty cycle.

6. There's no single right answer; reason from what each layer sees and speaks. First ask: what protocol does the call traffic use? If it isn't HTTP (for example UDP), a layer 7 HTTP balancer can't read it, while a layer 4 balancer (such as AWS's Network Load Balancer, which supports TCP, TLS, UDP and QUIC) works per connection without reading the content. For the app's ordinary web and API requests (/api, /login), a layer 7 balancer is the better fit, because reading the path lets it route smartly. An app can use both.

Quick quiz

Round robin with health checks over servers s1, s2, s3, s4. s2 fails its health check. Requests 1, 2, 3 went to s1, s3, s4. Where does request 4 go?

  • A. s2
  • B. s1
  • C. s3
  • D. Nowhere: 502
Answer

B, s1. s2 is out of the healthy list, so round robin cycles s1 → s3 → s4 → s1.

Interview questions

  • What is a load balancer, and why not just give users all the server addresses through DNS?
  • Explain round robin and least connections. When does round robin go wrong?
  • What is a health check? What happens without one? (Users get 502s from a dead server.)
  • What are sticky sessions? Why are stateless servers usually better? What happens to sticky users when you add or remove a server?
  • The load balancer is a single point of failure. How do you fix that? (A standby pair that takes over the same address.)
  • What is the difference between a layer 4 and a layer 7 load balancer? Give an example of each.
  • What does 502 Bad Gateway mean? What would you return if every server is unhealthy? (503.)

Glossary

Term Meaning
Load balancer The "front door" that receives every request and picks a server to handle it
Backend server / target One of the servers behind the balancer that does the real work
IP address A number that identifies a machine on the network, like 203.0.113.11
DNS The internet's phone book: turns a name like chainow.in into an IP address (episode 3)
TTL (time to live) How long a cached answer may be kept before it must be fetched again (episode 3)
Cache / cached A saved copy of an answer, reused instead of asking again
Round robin Take turns: s1, s2, s3, s1, ...
Least connections Send the next request to the server with the fewest open connections (the least busy)
Connection An open line of communication between a client and a server while work is in progress
Health check The balancer regularly asking each server "are you OK?" and skipping those that don't answer
/health A special address on a server that only answers "I'm OK"; used by health checks
Healthy / unhealthy A server that passes / fails its health checks
max_fails / fail_timeout nginx settings: how many failed attempts mark a server failed (default 1) and how long it is skipped (default 10 s)
HTTP status code A number in a web reply that says what happened
200 OK The HTTP status for success
502 Bad Gateway The HTTP status a balancer returns when the server behind it fails
503 Service Unavailable The HTTP status for "no server can handle this right now"
Sticky session Always sending the same user to the same server
Hash A function that turns any text into a number, always the same number for the same text
CRC32 (zlib.crc32) A quick hash function in Python's zlib module; it works on bytes
Modulo (%) The remainder after division; x % 3 is always 0, 1 or 2
Bytes / .encode() Raw data a computer stores; .encode() turns text into bytes
Consistent hashing A smarter way to hash so only a few users move when servers change (later episode)
Single point of failure One part whose failure takes down the whole system (episode 5)
Standby A spare that watches the main one and takes over if it dies
Layer 4 (transport layer) The level of networking that moves data between machines; sees IP addresses and ports
Layer 7 (application layer) The level of the actual request, like HTTP; sees paths, headers and cookies
Port A number that says which program on a machine a connection is for
TCP / UDP Transport protocols: TCP delivers data reliably and in order; UDP sends quickly without that guarantee
TLS Encryption for a connection (the "S" in HTTPS)
QUIC A newer transport protocol built on UDP
HTTP / HTTPS The language of web requests; HTTPS is HTTP encrypted with TLS
Path / headers / cookies Parts of an HTTP request: the address on the site (/api), extra information, and small saved values the browser sends back
nginx A very popular web server and load balancer
AWS Elastic Load Balancing Amazon's managed load balancers, including the Network (layer 4) and Application (layer 7) Load Balancers
itertools.cycle Python tool that repeats a list for ever: s1, s2, s3, s1, ...
StopIteration Python's signal that an iterator has nothing more to give
ValueError / TypeError Python errors for a wrong value (e.g. empty input to min) / a wrong type (e.g. text instead of bytes)
RFC 5737 The internet standard that reserves 203.0.113.x addresses for documentation examples

Summary

  • Handing out every address through DNS isn't enough: answers are cached, so phones stay stuck on dead servers.
  • A load balancer is one front door: one public address that picks a server for every request.
  • Round robin gives equal turns, not equal work. Least connections sends work to the least busy server.
  • Health checks keep dead servers out of the list, so users don't get 502s.
  • Sticky sessions by hash % n move most users when the server count changes; prefer stateless servers.
  • The balancer itself must not be a single point of failure: run a standby pair.
  • Layer 4 balancers see IPs and ports (fast, any protocol); layer 7 balancers read the HTTP request (smart routing).

Next episode: our servers now share the load, but every one of them still asks the database the same questions, again and again. Caching: the fastest request is the one you never make.