These are the notes for System Design episode 40, the final episode of the course. Every episode of this course ended with a code editor where you ran a program and saw the output. Today you design the system behind that editor. A college holds a national coding contest. At 7 pm the problems open and 50,000 students click Run, almost together. Some of their code is buggy, and some might be trying to break things. Everyone wants a result in seconds, and the contest must be fair. You'll write the requirements, watch one student's infinite loop crash a server that runs code itself (TimeoutError: the code ran for over 2 s), and fix it with a sandbox, limits and verdicts (Time Limit Exceeded, Runtime Error). Then the last student waits 1,000 s, so you add a queue and workers and use Little's law to size the pool: 1,667 busy workers. You'll see that most students first run the same starter template and cache results by hash (102 sandbox runs for 1,000 clicks, 90% saved), stop Ravi's 40 clicks with a token bucket (9 accepted, 31 rejected with 429), and cut the bill with autoscaling (89% cheaper). Then you put every idea of the course in one picture, build a runnable mini code runner, and finish with your roadmap. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). Asha, Ravi, Meera, Neha, Kabir and the other people are fictional. Every number in this episode is our own assumption: the 50,000 clicks in the first minute, 2 s per run, 50 normal workers, Rs 4 per worker hour, the 700/200/100 click mix, the 2-hour peak. The video states no facts about how real companies or real code runners work. Little's law is standard queueing theory.
An online code runner (also called an online judge or an online compiler) is a website where you type a program, click Run, and see its output. Example: the compiler you have watched in every episode of this course.
The problems
| Problem | What it teaches |
|---|---|
| The college coding contest | Requirements: many languages, verdicts, safe, fast, 50,000 runs in the first minute, never trust what you run |
| One loop crashes the server | Running code on the web server fails; a sandbox with limits and verdicts |
| 50,000 clicks, one runner | A job queue plus workers, and Little's law for how many workers |
| Everyone runs Hello World | A result cache keyed by a SHA-256 hash of language, code and input |
| Ravi clicks Run 40 times | A token bucket rate limiter at the gateway (429) |
| Peak size all day | Autoscaling on queue length: 89% cheaper |
| Put it all together | The full design: every box is an episode you already know |
| The whole course, and what next | A course map of all 40 episodes and a 5-step roadmap |
Problem 0: the college coding contest (requirements)
Last week Asha solved a contest problem on her laptop. In the real contest, the same site must serve 50,000 students at once, in Python, Java and C++, and any of them can send any code.
A few words first:
- A coding contest is a timed event where students solve programming problems; each submission is run on hidden test inputs and judged. Example: "print the sum of 1 to n" with
n = 9, answer45. - A verdict is the short result a code runner gives for one run, such as
OK,Time Limit Exceeded,Runtime ErrororWrong Answer. Example: Ravi's endless loop getsTime Limit Exceeded. - Untrusted code is code written by someone you don't know, which you must assume may be buggy or hostile. Example: every student's submission.
- A stdin (standard input) is the text a program reads while running; its stdout (standard output) is the text it prints. Example: the contest feeds
9on stdin, and the program prints45on stdout.
Functional requirements (what the system must do):
- Run code in many languages, with input, and show the output and a verdict.
- Verdicts:
OK,Time Limit Exceeded,Runtime Error(and, in the homework,Wrong Answer).
Non-functional requirements (how well it must do it):
- Safe: one program must never hurt another program or the server.
- Fast: results in seconds, even at 7 pm.
- Scale: 50,000 runs in the first minute (our guess).
- The rule that makes this system special: we run code from strangers. Never trust it.
Problem 1: one loop crashes the server (a sandbox, limits and verdicts)
The obvious design: the web server receives the code and runs it itself. We even add a 2 second timeout. Asha sends a sum, Ravi sends a loop he forgot to stop, Neha sends hello.
- A subprocess is a separate program started by your program. Python's
subprocess.runstarts one, waits for it and collects its output. Example:subprocess.run([sys.executable, "-c", code])starts a new Python that runs the student'scode. - A timeout is a limit on how long you wait; when it passes, you give up. Example:
timeout=2below. - A web server is the program that receives requests from browsers and sends replies. Example: the server behind the Run button.
import signal
def too_slow(signum, frame):
raise TimeoutError("the code ran for over 2 s")
signal.signal(signal.SIGALRM, too_slow)
def run(name, code): # the web server runs the code itself
print(name, end=": ")
signal.alarm(2) # a 2 second timer
exec(code)
signal.alarm(0)
run("Asha", "print(sum(range(10)))")
run("Ravi", "while True: pass") # forgot to stop the loop
run("Neha", "print('hello')") # never reached
Output (file paths shortened to main.py):
Asha: 45
Ravi: Traceback (most recent call last):
File "main.py", line 14, in <module>
run("Ravi", "while True: pass") # forgot to stop the loop
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "main.py", line 10, in run
exec(code)
~~~~^^^^^^
File "<string>", line 1, in <module>
File "main.py", line 4, in too_slow
raise TimeoutError("the code ran for over 2 s")
TimeoutError: the code ran for over 2 s
- Asha gets 45 (0 + 1 + ... + 9).
- Ravi's loop runs for 2 s, the timer fires, and we get
TimeoutError: the code ran for over 2 s. The error was never caught, so the whole program crashed, and Neha never gets her answer. A traceback is Python's report of where an error happened, innermost call last. - Why? Student code runs inside our own server's program (
exec(code)runs text as Python code), and one bad program takes everyone down. Without the timer it would hang for ever. Worse, that code could read our files or use our network. signal.alarm(2)asks the operating system to send our program a signal after 2 s (works on Linux and macOS);too_slowis the function that runs when it arrives. A signal is a message the operating system sends to a running program. An exception such asTimeoutErroris Python's way of stopping normal flow when something goes wrong.
Real life: the chemistry lab
| In a school lab | In a code runner |
|---|---|
| Experiments happen in a fume hood, not in the classroom | A sandbox: a sealed container for each run |
| "Ten minutes, and only these chemicals" | Limits: time, memory, no network |
| The teacher writes a result for each student: pass, spilled, overtime | A verdict: OK, Time Limit Exceeded, Runtime Error |
- A sandbox is a sealed environment where a program runs with strict limits and cannot touch anything outside it. Example: a fresh, empty box per student run, thrown away afterwards.
- A resource limit caps what a program may use: CPU time, memory, disk, number of processes, network. Example: "at most 2 s of CPU and 256 MB of memory" (our numbers).
The fix: catch what goes wrong, and name it
Every run happens in a sandbox with limits, and we catch what goes wrong: a timeout becomes the verdict Time Limit Exceeded; a crash becomes Runtime Error plus the last line of the error; Meera divides by zero.
import subprocess, sys
def run(code, seconds=2): # in a sandbox, with limits
try:
result = subprocess.run([sys.executable, "-c", code],
capture_output=True, text=True, timeout=seconds)
except subprocess.TimeoutExpired:
return "Time Limit Exceeded"
if result.returncode != 0:
return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
return "OK: " + result.stdout.strip()
print("Asha:", run("print(sum(range(10)))"))
print("Ravi:", run("while True: pass"))
print("Meera:", run("print(10 / 0)"))
print("Neha:", run("print('hello')"))
Output:
Asha: OK: 45
Ravi: Time Limit Exceeded
Meera: Runtime Error: ZeroDivisionError: division by zero
Neha: OK: hello
- Asha: OK, 45. Ravi: Time Limit Exceeded. Meera: Runtime Error,
ZeroDivisionError: division by zero. Neha: OK, hello. Everyone gets an answer. - A returncode is the number a finished program hands back:
0means success, anything else means it crashed. AZeroDivisionErroris Python's error for dividing by zero;stderr(standard error) is where Python writes error messages, sosplitlines()[-1]takes the error's last line. - The
try/exceptcatches the timeout insiderun, so Ravi's loop becomes one line of output instead of a crash. You just invented verdicts.
Real life limits, in general (not any product's design)
In a real system each run usually goes into a container or a small virtual machine, with a memory limit, a CPU limit, no network, and a fresh, empty disk that is thrown away afterwards (general knowledge, described here as general techniques and not as any product's design). That is security from episode 26 and reliability from episode 24.
- A container is an isolated group of processes that sees its own files and has its own limits, while sharing the machine's operating system. Example: one run inside one container.
- A virtual machine (VM) is a whole pretend computer running inside a real one; it is a stronger wall than a container but heavier. Example: a small VM per run.
An important warning: a subprocess with a timeout is NOT a security sandbox
The demo above protects us from infinite loops and crashes. It does not protect us from hostile code. The student's program still runs as the same user on the same machine, so it can read our files. Here the "student" reads a file it should never see:
import subprocess, sys, tempfile, os
secret = tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False)
secret.write("the server's private key") # pretend this is a secret file
secret.close()
def run(code, seconds=2): # the SANDBOX demo's run(): only a timeout
result = subprocess.run([sys.executable, "-c", code],
capture_output=True, text=True, timeout=seconds)
return result.stdout.strip()
evil = f"print(open({secret.name!r}).read())" # student code reads a file it shouldn't
print("student code:", evil.replace(secret.name, "<secret file>"))
print("output:", run(evil))
os.remove(secret.name)
Output:
student code: print(open('<secret file>').read())
output: the server's private key
- The student's code read the secret file and printed it. A timeout limits time; it does not limit what the program can see or do.
- That is why the demos in this episode are teaching models. A real runner needs the walls described above: containers or VMs, resource limits, no network, a fresh disk per run (general techniques). Never run strangers' code like this on a real server.
Problem 2: 50,000 clicks, one runner (a queue, workers and Little's law)
Next, speed. 50,000 clicks in the first minute, and say each run takes 2 seconds of CPU (our guess). How long does the last student wait with 100, 1,000 or 2,000 sandboxes?
- A worker is a program that runs separately, takes a job from a queue and does the slow work. Example: one worker that takes a submission, runs it in a sandbox, and saves the verdict.
- CPU time is how long a processor actually spends computing for a program. Example: a run that needs 2 s of CPU keeps one core busy for 2 s.
- Arrival rate is how many jobs arrive per second. Example: 50,000 clicks in 60 s is about 833 a second.
runs = 50_000 # Run clicks in the contest's first minute
run_time = 2 # seconds of CPU for one run (our guess)
def last_wait(workers): # how long the last student waits
return runs * run_time / workers
for workers in [100, 1_000, 2_000]:
print(f"{workers:>5,} workers: last result after {last_wait(workers):,.0f} s")
arrival = runs / 60 # runs arriving per second
needed = arrival * run_time # Little's law: busy workers = rate x time
print(f"to keep up: {arrival:,.0f} runs a second x {run_time} s = {needed:,.0f} busy workers")
Output:
100 workers: last result after 1,000 s
1,000 workers: last result after 100 s
2,000 workers: last result after 50 s
to keep up: 833 runs a second x 2 s = 1,667 busy workers
- 50,000 × 2 s = 100,000 seconds of work. With 100 workers that is 100,000 ÷ 100 = 1,000 s, about 17 minutes, in a contest! With 1,000 workers: 100 s. With 2,000: 50 s.
- The last line is Little's law from queueing theory (the mathematics of waiting lines). In its usual form: the average number of things in a system = the arrival rate × the average time each one spends in it. Here: busy workers = arrival rate × time per job = 833 runs a second × 2 s = 1,667 workers busy at once. Example: a shop where 3 customers arrive each minute and each stays 4 minutes holds about 12 customers at a time. The 833 and the 2 s are our assumptions; the law itself is standard.
Real life: the exam hall counter
| At a college exam hall | In our system |
|---|---|
| Students hand in answer sheets at a counter; clerks check them one by one | Students' runs are put in a job queue |
| More clerks mean a shorter line | More workers mean a shorter wait |
| A student does not stand inside the clerk's room | The web server never runs code itself |
So the web server never runs code. It puts each run in a message queue (episode 16), and a pool of workers, each with sandboxes, takes jobs from it.
- A message queue is a waiting line of messages: one part of a system adds jobs at the back, another takes them from the front later. Example:
("run #812", code, input). - A worker pool is a group of workers taking jobs from the same queue. Example: 1,667 workers busy at 7 pm.
- The student's page waits for the result, being told when it is ready (the video compares it with the change log push in Drive, episode 38). You just invented a job queue for code.
Problem 3: everyone runs Hello World (a result cache)
Look at what students actually run. Most first click Run on the starter template, unchanged. Say (our assumption) 700 of 1,000 clicks run the same Hello World, 200 run print(2 + 2), and 100 are different.
- A cache is a fast store of results already computed, so the same question doesn't need the work twice (episodes 7 and 8). Example: the answer to "run
print("Hello, World!")" kept for next time. - A hash is a fixed-size fingerprint made from data; the same data always gives the same hash. SHA-256 is a standard hash function that makes a 64-character fingerprint (episode 38). Example:
hashlib.sha256(b"abc").hexdigest(). - A cache key is the name you look a cached result up by. Here: the hash of language + code + input.
import hashlib
cache = {} # hash(language + code + input) -> result
real_runs = 0
def run(lang, code, stdin):
global real_runs
key = hashlib.sha256(f"{lang}\0{code}\0{stdin}".encode()).hexdigest()
if key not in cache:
real_runs += 1 # only now do we use a sandbox
cache[key] = f"output of {code!r}"
return cache[key]
starter = 'print("Hello, World!")' # everyone runs the template first
clicks = [starter] * 700 + ["print(2 + 2)"] * 200 + [f"print({i})" for i in range(100)]
for code in clicks:
run("python", code, "")
print(f"Run clicks: {len(clicks):,}")
print(f"sandbox runs: {real_runs:,}")
print(f"saved: {1 - real_runs / len(clicks):.0%}")
Output:
Run clicks: 1,000
sandbox runs: 102
saved: 90%
- 1,000 clicks, only 102 sandbox runs: 1 for the template, 1 for
print(2 + 2), and 100 different programs. 1 - 102/1,000 = 0.898, shown as 90% saved. - The key includes the language, the code and the input, because changing any of them can change the output. The
\0(a zero byte) between them keeps"ab" + "c"from looking the same as"a" + "bc". - In this demo the "run" just builds a string; in a real runner
cache[key] = ...would hold the verdict and output a worker produced. - The catch: caching is only safe for programs that give the same output every time (they are deterministic). Code that prints the time or a random number must skip the cache (see Common mistakes). You just invented a result cache.
Problem 4: Ravi clicks Run 40 times (a token bucket)
Ravi is nervous, so he clicks Run four times a second for ten seconds: 40 runs. Multiply that by a few thousand nervous students and the queue fills with junk. We use a token bucket (episode 19): a bucket holds 5 tokens, each run costs one, and one token comes back every 2 seconds.
- A rate limiter caps how often one user may call a service. Example: at most a few runs per second per student.
- A token bucket is a rate limiter that lets short bursts through but holds a steady average: you may spend a token only if the bucket has one, and tokens refill at a fixed pace. Example: 5 runs at once, then one more every 2 s.
- HTTP 429 (Too Many Requests) is the web's standard reply that says "slow down". Example: Ravi's 10th click gets 429.
import time
class TokenBucket: # episode 19: 5 runs, refilled 1 every 2 s
def __init__(self, size=5, refill_every=2):
self.size, self.tokens, self.every = size, size, refill_every
self.last = time.time()
def allow(self, now):
self.tokens = min(self.size, self.tokens + (now - self.last) / self.every)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
bucket = TokenBucket()
start = bucket.last
results = [bucket.allow(start + t * 0.25) for t in range(40)] # 4 clicks a second, 10 s
print("Ravi clicks Run 40 times in 10 s")
print("accepted:", results.count(True), " rejected (429):", results.count(False))
Output:
Ravi clicks Run 40 times in 10 s
accepted: 9 rejected (429): 31
- Accepted: 9. Rejected with 429: 31. The bucket starts with 5 tokens (5 runs at once), and over the 9.75 s of clicks about 4.9 more tokens refill (one per 2 s), so 5 + 4 whole tokens spent = 9.
- The clicks are given fake times (
start + t * 0.25), so the result is the same every run; a real limiter would pass the current time. - Ravi still gets his results, and his clicks can't starve everyone else. The limit lives in the API gateway (episode 20), before anything reaches the queue. You just invented fairness.
- An API gateway is the front door of a system: it checks who you are and how often you call, then passes the request on. Example: the box in front of the run service.
Problem 5: peak size all day (autoscaling)
Last problem: money. We need 1,667 workers at 7 pm, but maybe 50 on a normal evening (our guess). Say a worker costs Rs 4 an hour (a made-up number). Compare keeping the peak size all day with autoscaling for a 2 hour peak.
- Horizontal scaling means adding more machines instead of making one bigger (episode 5). Example: going from 50 workers to 1,667.
- Autoscaling means adding and removing machines automatically as load changes. Example: more workers when the queue grows, fewer when it empties.
- Queue length (backlog) is how many jobs are waiting. Example: 20,000 runs waiting is a signal to add workers.
peak_workers = 1_667 # from the contest minute (our estimate)
normal_workers = 50 # a normal evening (our guess)
cost_per_worker_hour = 4 # rupees, our made-up number
always_peak = peak_workers * 24 * cost_per_worker_hour
autoscale = (normal_workers * 22 + peak_workers * 2) * cost_per_worker_hour
print(f"peak size all day: Rs {always_peak:,} a day")
print(f"autoscaling (2 peak hours): Rs {autoscale:,} a day")
print(f"saved: {1 - autoscale / always_peak:.0%}")
Output:
peak size all day: Rs 160,032 a day
autoscaling (2 peak hours): Rs 17,736 a day
saved: 89%
- Peak size all day: 1,667 × 24 × Rs 4 = Rs 160,032 a day (about 1.6 lakh). Autoscaling: (50 × 22 h + 1,667 × 2 h) × Rs 4 = Rs 17,736. 89% cheaper. A lakh is 100,000 (an Indian number word).
- Autoscaling watches the queue length and adds workers when it grows. And because the contest start time is known, we start workers about ten minutes before 7 pm, not after the queue has exploded (new workers take time to start).
- You just invented autoscaling. All the money numbers are made up.
The full design
students API gateway run service queue workers sandboxes
──────── ─────────── ─────────── ───── ─────── ─────────
browser ───▶ login, rate limit ──▶ check result cache ──▶ per language ──▶ autoscaled ───▶ time, memory,
(Run) (token bucket, hit → answer now (job id) (take jobs) no network,
▲ 429) miss → job id │ fresh disk
│ ▼
└──────── result pushed over a WebSocket ◀───────────────────────── result store (cache + DB)
logs, metrics, traces (episode 25) watch every box
- Students' browsers talk to an API gateway that checks login and the rate limit.
- The run service checks the result cache. On a hit it answers at once. On a miss it gives the run a job id and puts it in a queue per language.
- Autoscaled workers take jobs, run each in a fresh sandbox with time and memory limits and no network, and save the result to the result store and the cache.
- Results are pushed back over a WebSocket to the student's page. Logs, metrics and traces (episode 25) show where time is going on contest night.
- Look again: a gateway, rate limiting, caching, queues, workers, scaling, security, observability. Every box is an episode you already know. That is the whole point of this course.
- A WebSocket is a long-lived connection between browser and server over which the server can push messages whenever it likes. Example: "your result is ready" arrives without the page asking.
- A job id is a unique name for one run, so the page and the worker can find each other's result. Example:
run-812. - A result store is a database that keeps finished verdicts and outputs. Example: where "Meera: Runtime Error" is saved.
- Observability means being able to see what a system is doing from the outside, using logs, metrics and traces (episode 25).
The whole chain
TimeoutError: the code ran for over 2 stook the server down → a sandbox with limits and verdicts.- The last result after 1,000 s → a queue and workers; Little's law says 1,667 busy workers.
- 700 identical runs → a result cache keyed by hash: 90% saved.
- 40 clicks from Ravi → a token bucket: 9 accepted, 31 rejected (429).
- Peak size all day → autoscaling: 89% cheaper.
Common mistakes (and the real outputs)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Run student code on the web server | TimeoutError: the code ran for over 2 s |
One bad program takes everyone down | Sandbox + limits + verdicts, on workers |
| Too few workers for the rush | 100 workers: last result after 1,000 s |
Arrival rate × time > workers | Little's law, a queue, autoscaling |
| Cache every program's output | time shown by the second run minus the first: 0 (2 s passed) |
Not every program gives the same output each time | Cache only programs with the same output every time |
| Think a timeout makes a sandbox | output: the server's private key |
A timeout limits time, not what the code can read | Containers or VMs, no network, fresh disk |
No Wrong Answer verdict |
Kabir: OK: 36 (the right answer is 45) |
The runner only checks that the program did not crash | Compare the output with the expected output |
The "cache everything" row is a cache with the key made from the code only, applied to a program that prints the current time:
import hashlib, subprocess, sys, time
cache = {}
def run(code):
key = hashlib.sha256(code.encode()).hexdigest()
if key not in cache: # cache EVERYTHING: the mistake
cache[key] = subprocess.run([sys.executable, "-c", code],
capture_output=True, text=True).stdout.strip()
return cache[key]
code = "import time; print(int(time.time()))" # prints the current time
first = int(run(code))
time.sleep(2)
second = int(run(code))
print("seconds between the two clicks: 2")
print("time shown by the second run minus the first:", second - first)
Output:
seconds between the two clicks: 2
time shown by the second run minus the first: 0
The second click came 2 seconds later, but the student sees the old time: the cache returned the first answer.
The "no Wrong Answer verdict" row is the verdict demo with an input and a buggy answer (range(n) stops one short, so it adds 0 to 8 and gets 36, not 45):
import subprocess, sys
def run(code, stdin="", seconds=2): # the SANDBOX demo's run(), plus input
try:
result = subprocess.run([sys.executable, "-c", code], input=stdin,
capture_output=True, text=True, timeout=seconds)
except subprocess.TimeoutExpired:
return "Time Limit Exceeded"
if result.returncode != 0:
return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
return "OK: " + result.stdout.strip()
# the task: print 1 + 2 + ... + n. For n = 9 the right answer is 45.
print("Kabir:", run("n = int(input()); print(sum(range(n)))", "9"))
Output:
Kabir: OK: 36
Mini project: a tiny online code runner
A job queue, three worker threads, a subprocess sandbox with a 1 s timeout, four verdicts (OK, Wrong Answer, Time Limit Exceeded, Runtime Error), a result cache keyed by a SHA-256 hash of language, code and input, and a per-user token bucket (3 runs, one more every 2 s). Time for the token bucket is passed in (simulated), so the output is the same on every run. The task is "read n, print 1 + 2 + ... + n"; the test input is 9 and the expected output is 45.
A thread is a path of execution inside a program; several threads can run at the same time. Example: three worker threads taking jobs from one queue. A lock lets only one thread at a time change shared data. Example: with lock: around the cache.
Remember the warning: this runner's sandbox is a subprocess with a timeout, a teaching model, not a security boundary.
# Mini project: a tiny online code runner. A queue, three worker threads, a
# subprocess "sandbox" with a timeout, verdicts (OK, Wrong Answer, Time Limit
# Exceeded, Runtime Error), a result cache and a per-user token bucket.
# NOTE: a subprocess with a timeout is NOT a security sandbox (see the notes).
import hashlib, queue, subprocess, sys, threading
TIMEOUT = 1 # seconds a run may use (ours)
WORKERS = 3 # worker threads (ours)
BUCKET, REFILL = 3, 2 # 3 runs, +1 every 2 s (ours)
class TokenBucket: # episode 19, one bucket per user
def __init__(self):
self.tokens, self.last = BUCKET, 0.0
def allow(self, now):
self.tokens = min(BUCKET, self.tokens + (now - self.last) / REFILL)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
def sandbox(code, stdin, expected): # run one program, return a verdict
try:
r = subprocess.run([sys.executable, "-c", code], input=stdin,
capture_output=True, text=True, timeout=TIMEOUT)
except subprocess.TimeoutExpired:
return "Time Limit Exceeded"
if r.returncode != 0:
return "Runtime Error: " + r.stderr.strip().splitlines()[-1]
if r.stdout.strip() != expected:
return "Wrong Answer"
return "OK"
jobs = queue.Queue() # the job queue
results, cache, buckets = {}, {}, {}
sandbox_runs = 0
lock = threading.Lock()
def worker(): # takes jobs until it gets None
global sandbox_runs
while True:
job = jobs.get()
if job is None:
return
sub_id, key, code, stdin, expected = job
verdict = sandbox(code, stdin, expected)
with lock:
sandbox_runs += 1
cache[key] = verdict
results[sub_id] = verdict
jobs.task_done()
def submit(sub_id, user, now, code, stdin="9", expected="45"): # the API
if not buckets.setdefault(user, TokenBucket()).allow(now):
results[sub_id] = "429 Too Many Requests"
return
key = hashlib.sha256(f"python\0{code}\0{stdin}".encode()).hexdigest()
with lock:
hit = cache.get(key)
if hit: # same language+code+input: no sandbox
results[sub_id] = hit + " (cached)"
return
jobs.put((sub_id, key, code, stdin, expected))
threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
for t in threads:
t.start()
GOOD = "n = int(input()); print(sum(range(1, n + 1)))" # sum 1..n
WAVE1 = [("Asha", GOOD), ("Ravi", "while True: pass"), ("Meera", "print(10 / 0)"),
("Kabir", "n = int(input()); print(sum(range(n)))"), ("Neha", GOOD)]
order = []
for i, (user, code) in enumerate(WAVE1, 1): # wave 1: five students click Run
order.append((f"#{i}", user))
submit(f"#{i}", user, now=0.0, code=code)
jobs.join() # wait until wave 1 is done
for i in range(6, 12): # wave 2: Ravi hammers Run, 0.5 s apart
order.append((f"#{i}", "Ravi"))
submit(f"#{i}", "Ravi", now=1.0 + (i - 6) * 0.5, code=GOOD)
order.append(("#12", "Asha")) # and Asha reruns her good code
submit("#12", "Asha", now=4.0, code=GOOD)
jobs.join()
for _ in threads:
jobs.put(None)
for t in threads:
t.join()
for sub_id, user in order:
print(f"{sub_id:<4}{user:<6}{results[sub_id]}")
print(f"clicks: {len(order)}, sandbox runs: {sandbox_runs}, "
f"cached: {sum('(cached)' in v for v in results.values())}, "
f"rejected: {sum(v.startswith('429') for v in results.values())}")
Output:
#1 Asha OK
#2 Ravi Time Limit Exceeded
#3 Meera Runtime Error: ZeroDivisionError: division by zero
#4 Kabir Wrong Answer
#5 Neha OK
#6 Ravi OK (cached)
#7 Ravi OK (cached)
#8 Ravi OK (cached)
#9 Ravi 429 Too Many Requests
#10 Ravi 429 Too Many Requests
#11 Ravi 429 Too Many Requests
#12 Asha OK (cached)
clicks: 12, sandbox runs: 5, cached: 4, rejected: 3
- Four verdicts in wave 1: Asha
OK, RaviTime Limit Exceeded(killed after 1 s), MeeraRuntime Error: ZeroDivisionError: division by zero, KabirWrong Answer(his code printed 36, not 45). Three workers ran the five jobs side by side, so the whole program took about one second, not five. - Cache: Ravi's good code (#6 to #8) and Asha's rerun (#12) came from the cache, with no sandbox. Neha's #5 was not cached: Asha's and Neha's identical code arrived in the same wave, before the first result was saved, so both ran (5 sandbox runs for 12 clicks). A real system would also remember jobs that are in flight so the second click waits for the first (see "Try extending it").
- Token bucket: Ravi's bucket held 3 tokens. His loop (#2) used one, and #6 to #8 used the rest as tokens trickled back; #9 to #11 got 429. The limit is checked before the cache, so even a cached rerun costs a token.
- Queue + workers + results by id: the API only puts jobs in the queue; the workers write
results[sub_id], which the student's page would read.
Try extending it: remember in-flight jobs so duplicate clicks in the same wave share one run; skip the cache for code that imports time or random; add a Memory Limit Exceeded verdict with the resource module (on Linux); run a separate queue per language; and make the workers add themselves when jobs.qsize() grows (autoscaling).
The whole course, in one map
Look how far you've come. Here are all 40 episodes, grouped as in the video.
| Group | Episodes | Titles |
|---|---|---|
| How the web works | 1-4 | 1 What happens when you open a website; 2 Client–server and HTTP; 3 DNS deep dive; 4 Latency vs throughput |
| Scaling and caching | 5-8 | 5 Scaling: vertical vs horizontal; 6 Load balancers; 7 Caching; 8 Cache strategies |
| Data at scale | 9-15 | 9 Databases: SQL vs NoSQL; 10 Indexing; 11 Replication; 12 Sharding; 13 Consistent hashing; 14 CAP and PACELC; 15 Consistency models |
| Services talking | 16-20 | 16 Message queues and pub/sub; 17 Monolith vs microservices; 18 API design: REST, GraphQL, gRPC; 19 Rate limiting; 20 Proxies and API gateways |
| Building blocks | 21-26 | 21 Storage and CDNs; 22 Search: the inverted index; 23 Unique IDs at scale; 24 Reliability patterns: retries, idempotency, circuit breakers; 25 Observability: logs, metrics, traces; 26 Security basics: auth, OAuth, JWT, encryption |
| Real designs + interviews | 27-39 | 27 Design a URL shortener; 28 Design a rate limiter service; 29 Design WhatsApp chat; 30 Design the Instagram feed; 31 Design Hotstar live streaming for IPL; 32 Design Uber/Ola ride matching; 33 Design Swiggy/Zomato delivery; 34 Design UPI payments; 35 Design IRCTC Tatkal booking; 36 Design search autocomplete; 37 Design a notification system; 38 Design a file storage service (Drive); 39 The system design interview framework |
| Capstone | 40 | 40 Capstone and roadmap (this episode) |
And where this episode used them:
| Box in the code runner | Episode(s) |
|---|---|
| Queue between the run service and workers | 16 |
| Result cache, cache by hash | 7, 8, 38 |
| Rate limit (token bucket) | 19 |
| API gateway | 20 |
| Horizontal scaling and autoscaling | 5 |
| Retries and reliability | 24 |
| Logs, metrics, traces | 25 |
| Security (sandbox, no network) | 26 |
Your roadmap
- Redo three designs from memory, out loud, in 45 minutes each. Pick three designs from episodes 27 to 38 and explain them to an empty room, or to a friend.
- Do mock interviews with a friend, and swap roles. Being the interviewer teaches you a lot.
- Build one for real. A small URL shortener with a cache and a database. Then load test it (send it many requests at once) and watch what breaks.
- Read company engineering blogs, where teams explain how they really scaled. Treat each as one team's story, not a universal rule.
- Go deeper into databases, distributed systems and networks.
And the habit that makes a great designer: for every app you use, ask "what breaks at 10x the users?"
Practice
- Homework: in the SANDBOX demo, add a Wrong Answer verdict by comparing the output with the expected one.
- Homework: in the WORKERS demo, runs take 5 s. How many workers? What does the SCALE bill become?
- In the CACHE demo, 102 of 1,000 clicks reach a sandbox. How many busy workers do we need now at the contest minute?
- In the LIMIT demo, the bucket holds 10 runs instead of 5. How many of Ravi's 40 clicks are accepted?
- Little's law: runs arrive at 100 a second and take 0.5 s; or 2,000 a second and 4 s. How many busy workers?
- Why must code that prints the time or a random number skip the cache?
- Why is a subprocess with a timeout not a security sandbox? What does a real runner add (in general)?
- Why start the workers ten minutes before 7 pm, instead of waiting for the queue to grow?
Solution
1. Give run an input and an expected output, and add a third check after the crash check:
import subprocess, sys
def run(code, stdin="", expected=None, seconds=2): # in a sandbox, with limits
try:
result = subprocess.run([sys.executable, "-c", code], input=stdin,
capture_output=True, text=True, timeout=seconds)
except subprocess.TimeoutExpired:
return "Time Limit Exceeded"
if result.returncode != 0:
return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
if expected is not None and result.stdout.strip() != expected:
return "Wrong Answer"
return "OK: " + result.stdout.strip()
# the task: read n, print the sum 1 + 2 + ... + n. For n = 9 the answer is 45.
print("Asha:", run("n = int(input()); print(sum(range(1, n + 1)))", "9", "45"))
print("Ravi:", run("while True: pass", "9", "45"))
print("Meera:", run("print(10 / 0)", "9", "45"))
print("Kabir:", run("n = int(input()); print(sum(range(n)))", "9", "45")) # off by one: 36
Output:
Asha: OK: 45
Ravi: Time Limit Exceeded
Meera: Runtime Error: ZeroDivisionError: division by zero
Kabir: Wrong Answer
The order matters: a run that timed out or crashed has no trustworthy output, so check those first; only a clean run is compared with the expected output. Real judges also usually ignore trailing spaces and blank lines (we use .strip()).
2. Change run_time = 2 to run_time = 5. Output:
100 workers: last result after 2,500 s
1,000 workers: last result after 250 s
2,000 workers: last result after 125 s
to keep up: 833 runs a second x 5 s = 4,167 busy workers
833 × 5 = 4,167 busy workers (2.5 times more than 1,667, because each run is 2.5 times longer). In the SCALE demo, set peak_workers = 4_167. Output:
peak size all day: Rs 400,032 a day
autoscaling (2 peak hours): Rs 37,736 a day
saved: 91%
Autoscaling saves even more (91%), because the peak is further above the normal 50 workers.
3. Only the cache misses need a worker:
arrival = 50_000 / 60 # clicks a second
run_time = 2 # seconds per run
hit_rate = 1 - 102 / 1000 # from the cache demo: 102 real runs in 1,000 clicks
real_runs_per_second = arrival * (1 - hit_rate)
print(f"workers without a cache: {arrival * run_time:,.0f}")
print(f"workers with the cache: {real_runs_per_second * run_time:,.0f}")
Output:
workers without a cache: 1,667
workers with the cache: 170
About 170 workers instead of 1,667, if the same 90% hit rate holds at the contest minute (our assumption: the click mix is made up).
4. Change size=5 to size=10. Output:
Ravi clicks Run 40 times in 10 s
accepted: 14 rejected (429): 26
The bigger bucket lets 14 through (10 at once + 4 refilled). A bigger bucket allows a bigger burst but the long-run rate (one every 2 s) is unchanged.
5. Busy workers = arrival × time:
def busy(arrival, seconds): return arrival * seconds
print(busy(500, 3), busy(100, 0.5), busy(2000, 4))
Output:
1500 50.0 8000
500 × 3 = 1,500 (the quiz), 100 × 0.5 = 50, 2,000 × 4 = 8,000.
6. The cache returns the old output for the same code. A program that prints time.time() or a random number should give a new answer each time, but a cached run returns the first one (see the Common mistakes demo: the second run showed the old time, difference 0). Cache only deterministic programs, or skip the cache for code that uses time, randomness or the network.
7. The timeout only limits how long a program runs. The program still runs as the same user on the same machine, so (as the demo showed) it can read files, and it could also use the network or start many processes. A real runner adds, in general: a container or small VM per run, CPU and memory limits, no network, a fresh empty disk thrown away afterwards, and a limit on the number of processes. These are general techniques; the demos here are teaching models.
8. New workers take time to start (machines boot, programs load). If you wait for the queue to grow, students are already waiting before help arrives. The contest time is known in advance, so you can start workers early and keep scaling on queue length for surprises.
Quick quiz
arrival = 500 # runs a second
run_time = 3 # seconds each
workers = ???
Runs arrive at 500 a second and take 3 s each. How many workers are busy at once?
- A. 167
- B. 503
- C. 1,500
- D. 15,000
Answer
C. Little's law: busy workers = arrival rate × time per job = 500 × 3 = 1,500.
Interview questions: "Design an online code runner / online judge"
1. Clarify requirements. Which languages? Is it practice (a few users) or a live contest (a huge burst)? Do we only run code, or also judge it against hidden tests with a verdict? Input and output sizes? Time and memory limits per run? How fast must results come back? Then state: run untrusted code safely, give a verdict, fair under a burst, results in seconds.
2. Estimate (our assumptions, said out loud). "50,000 runs in the first minute, 2 s each. That is about 833 runs a second. By Little's law, busy workers = 833 × 2 ≈ 1,667. If 70% of clicks are the starter template, a cache cuts that sharply. The peak lasts maybe two hours, so I would autoscale instead of paying for the peak all day."
3. Components.
- API gateway: login, per-user rate limit (token bucket, 429).
- Run service: checks the result cache (key = hash of language, code, input), makes a job id, enqueues.
- Queues: per language, so one slow language doesn't block another.
- Workers, autoscaled: take jobs, run each in a fresh sandbox, compare output with the expected output, produce a verdict.
- Sandbox: container or small VM with CPU, memory and process limits, no network, fresh disk (general technique).
- Result store and cache; WebSocket push of results; logs, metrics, traces.
4. The hard parts. Running untrusted code safely (isolation, not just a timeout); a huge burst at a known time (queue + workers + pre-scaling); fairness (rate limit per user); not wasting work (cache by hash; skip it for non-deterministic code); and an honest verdict (time limit, runtime error, wrong answer).
5. Scale and reliability (general knowledge). Stateless API servers behind a load balancer; replicate the queue so a crashed machine loses no jobs; a worker that dies mid-run lets the job be handed out again (idempotent runs, episode 24); alert when the queue length or the worst wait grows; keep a dead letter queue for runs that keep crashing the sandbox itself.
Common follow-ups:
- What if a student's code forks many processes or fills the disk? (A process limit, a disk quota, and a fresh disk per run.)
- What if two identical submissions arrive at the same moment? (Remember in-flight jobs so the second waits for the first.)
- How do you keep the contest fair when some runs are slower? (Same limits for everyone; judge by CPU time, not queue waiting time.)
- How do you test the design before the contest? (A load test that sends the expected burst, and watching the queue.)
- A worker crashes after finishing but before saving the result? (The job is handed out again; the result is the same, so saving it twice is harmless.)
Glossary
| Term | Meaning |
|---|---|
| Online code runner / judge | A website where you type a program, click Run, and see its output and a verdict |
| Coding contest | A timed event where students solve programming problems and submissions are judged |
| Verdict | The short result of a run: OK, Time Limit Exceeded, Runtime Error, Wrong Answer |
| Time Limit Exceeded | The program ran longer than allowed and was stopped, e.g. an infinite loop |
| Runtime Error | The program crashed while running, e.g. ZeroDivisionError: division by zero |
| Wrong Answer | The program finished cleanly but its output differs from the expected output |
| Untrusted code | Code from someone you don't know, assumed possibly buggy or hostile |
| stdin / stdout / stderr | A program's input text / its printed output / where error messages are written |
| Returncode | The number a finished program returns; 0 means success |
| Subprocess | A separate program started by your program, e.g. subprocess.run |
| Timeout | A limit on how long you wait before giving up |
TimeoutError |
Python's error for a timeout; in the first demo, raised by our own 2 s timer |
subprocess.TimeoutExpired |
Python's error when a subprocess runs past its timeout (the sandbox demo catches it) |
exec / signal |
Python's exec runs text as code / a message the operating system sends to a running program |
| Traceback | Python's report of where an error happened, innermost call last |
| Web server | The program that receives browser requests and sends replies |
| Sandbox | A sealed environment where a program runs with strict limits and cannot touch the outside |
| Resource limit | A cap on CPU time, memory, disk, processes or network for a program |
| Container | An isolated group of processes with its own files and limits, sharing the machine's operating system |
| Virtual machine (VM) | A whole pretend computer running inside a real one; a stronger wall, but heavier |
| Message queue | A waiting line of jobs: added at the back, taken from the front later |
| Worker / worker pool | A separate program that takes jobs from the queue and does the slow work / a group of them |
| CPU time | How long a processor actually computes for a program |
| Arrival rate | How many jobs arrive per second, e.g. 833 runs a second |
| Little's law | In a stable system, average number inside = arrival rate × average time inside; here busy workers = runs per second × seconds per run |
| Queueing theory | The mathematics of waiting lines |
| Cache | A fast store of results already computed, so the work isn't repeated |
| Cache key | The name a cached result is looked up by, here the hash of language + code + input |
| Hash / SHA-256 | A fixed-size fingerprint of data / a standard hash function that gives a 64-character fingerprint |
| Deterministic | Gives the same output every time for the same input; only such programs are safe to cache |
| Rate limiter | A cap on how often one user may call a service |
| Token bucket | A rate limiter with a bucket of tokens: each request spends one, tokens refill at a fixed pace, short bursts are allowed |
| HTTP 429 | "Too Many Requests", the web's reply that says "slow down" |
| API gateway | The front door of a system: checks login and rate limits, then passes requests on |
| Horizontal scaling | Adding more machines instead of making one bigger |
| Autoscaling | Adding and removing machines automatically as load changes |
| Queue length / backlog | How many jobs are waiting; the signal for autoscaling |
| Lakh | 100,000 (Indian number word); 1.6 lakh = 160,000 |
| Job id | A unique name for one run, so the page and the worker can find each other's result |
| Result store | A database that keeps finished verdicts and outputs |
| WebSocket | A long-lived browser-server connection over which the server can push messages |
| Observability | Seeing what a system is doing from outside, using logs, metrics and traces |
| Thread / lock | A path of execution inside a program / a rule that lets one thread at a time change shared data |
| Load test | Sending a system many requests at once to see where it breaks |
| Peak | The busiest period of load, here the contest's first minutes |
| Roadmap | A plan of what to learn or do next |
Summary
- An online code runner runs untrusted code, so never run it on the web server: one
while True: passgaveTimeoutErrorand crashed everything. Run each program in a sandbox with limits and report a verdict:OK,Time Limit Exceeded,Runtime Error,Wrong Answer. A subprocess with a timeout is not a security sandbox (it read a secret file); real runners use containers or VMs, resource limits, no network and a fresh disk. - Put each run in a queue and let workers take jobs. Little's law: busy workers = arrival rate × time per job = 833 × 2 s = 1,667 (100 workers: the last student waits 1,000 s). All numbers are our assumptions.
- A result cache keyed by the SHA-256 hash of language, code and input: 102 sandbox runs for 1,000 clicks, 90% saved. Never cache programs whose output changes (time, random).
- A token bucket at the API gateway: Ravi's 40 clicks, 9 accepted, 31 rejected with 429.
- Autoscaling on queue length, started before the known 7 pm peak: Rs 17,736 vs Rs 160,032 a day, 89% cheaper (made-up prices).
- Every box in the full design is an episode of this course. Your roadmap: redo three designs from memory, mock interviews, build one for real, read engineering blogs, go deeper, and always ask "what breaks at 10x?".
This is the final episode. Forty episodes, from opening a website to designing Drive and a code runner. Keep designing, keep building.
Sources: none needed. This episode states no facts about real companies or real code runners. All numbers are our own assumptions: the 50,000 clicks in the first minute, 2 s (and 5 s) per run, 50 normal workers, Rs 4 per worker hour, the 700/200/100 click mix, the 2-hour peak, the token bucket of 5 (and 10), the mini project's limits, and all names. Little's law, queueing theory, sandboxes, containers, virtual machines, hashes, token buckets and autoscaling are general textbook knowledge.