These are the notes for System Design episode 25. It's 2 a.m., customers in Pune say ChaiNow's checkout is painfully slow, and you can't see inside any of your six services. You'll write free-text logs and watch a simple question crash with ValueError, fix that with structured JSON logs and log levels, stop reading lines and start counting with metrics (and catch the average hiding the slowest customers, which p99 shows), learn Google's four golden signals, follow one request across six services with a trace (a trace id passed in the W3C traceparent header, and spans drawn as a waterfall), decide what should wake you up (alert on symptoms, not causes), and meet OpenTelemetry, the open standard that collects all three. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). ChaiNow, its orders and its timings are made up, and the trace ids are random, so yours will differ.
Observability means being able to tell what is happening inside a running system from what the system reports about itself. Example: from its logs, metrics and traces, you can say "checkout is slow since 2:04, for about 1 in 100 customers, because the payments database is slow", without logging into any server. Monitoring is the everyday part of it: watching a few chosen numbers and getting told when they go bad.
The problems
| Problem | What it teaches |
|---|---|
| Messy text logs | Logs, why free text is easy to write but hard to query (ValueError) |
| One question, one filter | Structured logs: JSON fields, log levels, collecting logs in one place |
| Too many lines to read | Metrics: numbers over time, counters, gauges, histograms |
| The average lies | Percentiles (p50, p95, p99), tail latency |
| Which numbers first? | The four golden signals: latency, traffic, errors, saturation |
| Which service is slow? | Distributed tracing: trace id, spans, parent id, the W3C traceparent header |
| What should wake you up? | Alerting on symptoms, not causes; every alert needs an action |
| Every tool has its own plug | OpenTelemetry, OTLP, avoiding vendor lock-in |
Problem 1: messy logs
Since episode 17, ChaiNow is six services (gateway, orders, menu, payment, fraud check, the payments database) running on many servers. Customers say checkout is slow. Where do you even look?
The first idea: every service writes a line of text whenever something happens, like a diary. That diary is a log: a record of events, one line per event, with the time it happened. Example: 02:01:07 order 5512 placed by priya in Pune, took 180ms.
Four developers, four services, four styles:
logs = """\
02:01:07 order 5512 placed by priya in Pune, took 180ms
02:01:09 Payment OK for order 5513 (Delhi) 2.4 s
02:01:12 order=5514 city=pune slow!!! 2300ms
02:01:15 placed order 5515, Mumbai, 95ms"""
# 2 a.m. question: how many Pune orders took over 1 second?
count = 0
for line in logs.splitlines():
ms = int(line.split()[-1].replace("ms", ""))
if "Pune" in line and ms > 1000:
count += 1
print("slow Pune orders:", count)
Output:
Traceback (most recent call last):
File "plain.py", line 10, in <module>
ms = int(line.split()[-1].replace("ms", ""))
ValueError: invalid literal for int() with base 10: 's'
line.split()[-1]takes the last word of each line, and.replace("ms", "")strips the unit, hoping what's left is a number.- Line 1 works:
180ms→180. Line 2 ends in2.4 s, so the last word iss, andint("s")crashes. - Even if it ran, line 3 says
punein small letters, so"Pune" in linewould miss the one slow Pune order.
Free text is easy to write and painful to search. Humans can read it; programs can't, because every developer writes it differently (milliseconds or seconds, Pune or pune, order 5512 or order=5514).
Problem 1, fixed: structured logs
A hospital solved this long ago. Doctors' scribbled notes are different every time, so the hospital prints a form with boxes: name, age, blood pressure. Every patient, same fields. And a red URGENT stamp says how serious it is.
For logs, the form is JSON (JavaScript Object Notation): a text format for data with named fields, like {"city": "pune", "ms": 180}. Almost every language can write and read it. A log written as named fields is a structured log. The stamp is the log level: how serious the event is. The common levels, from least to most serious:
| Level | Meaning | Example |
|---|---|---|
| DEBUG | Detail for developers, usually turned off in production | cart has 3 items |
| INFO | Normal events | order 5512 placed |
| WARN | Something unusual, but it still worked | order 5514 took 2300 ms |
| ERROR | Something failed | payment for order 5520 failed |
import json
def log(level, **fields): # one JSON object per line
line = json.dumps({"level": level, **fields})
print(line)
return line
lines = [
log("INFO", order=5512, city="pune", ms=180),
log("INFO", order=5513, city="delhi", ms=2400),
log("WARN", order=5514, city="pune", ms=2300),
log("INFO", order=5515, city="mumbai", ms=95),
]
events = [json.loads(l) for l in lines] # what a log tool does
slow = [e["order"] for e in events
if e["city"] == "pune" and e["ms"] > 1000]
print("slow Pune orders:", slow)
Output:
{"level": "INFO", "order": 5512, "city": "pune", "ms": 180}
{"level": "INFO", "order": 5513, "city": "delhi", "ms": 2400}
{"level": "WARN", "order": 5514, "city": "pune", "ms": 2300}
{"level": "INFO", "order": 5515, "city": "mumbai", "ms": 95}
slow Pune orders: [5514]
**fieldscollects any named arguments into a dictionary, solog("INFO", order=5512, city="pune", ms=180)becomes{"order": 5512, "city": "pune", "ms": 180}.json.dumpsturns the dictionary into one line of JSON text;json.loadsturns it back into a dictionary. That read-back is what a log tool does with every line.- Every service agrees on the field names (
order,city,msalways in milliseconds, cities in lowercase), so the 2 a.m. question is one filter:city == "pune" and ms > 1000. No parsing, no guessing. - To see only the trouble, filter on the level:
e["level"] in ("WARN", "ERROR").
In a real system, log tools collect the lines from every server into one place, so you can search all of them at once instead of logging into each machine.
Problem 2: too many lines to read (metrics)
ChaiNow handles 100,000 checkouts a day. Is checkout getting slower this week? You can't read 100,000 lines to find out.
Think of driving. A diary of every trip is too long to read at the wheel, so the car gives you gauges: speed, fuel. A few numbers, at a glance. And a warning light comes on by itself when a number goes bad.
| Driving a car | Observability |
|---|---|
| A diary of every trip | Logs: one line per event, full detail |
| Speedometer, fuel gauge | Metrics: numbers measured over time |
| A warning light | Alerts: a rule on a metric that tells you when to look |
A metric is a number measured over time: requests per second, how long requests take, how much disk is used. Three basic kinds:
| Kind | What it is | Example |
|---|---|---|
| Counter | A number that only goes up (until restart) | total checkouts: 100,000; total errors: 230 |
| Gauge | A number that goes up and down; you read its current value | CPU at 47%, queue length 12 |
| Histogram | Counts of values in ranges (buckets), so you can compute percentiles | 80,000 requests under 150 ms, 2,000 over 2 s |
Which number shows "checkout is slow"? The obvious one is the average (the mean: add all values, divide by how many).
import random, statistics
random.seed(7)
# a day of checkout times: most are fast, 2% wait on a slow payment
latencies = [random.gauss(110, 20) for _ in range(98_000)]
latencies += [random.gauss(2400, 300) for _ in range(2_000)]
def pct(p): # the p-th percentile
s = sorted(latencies)
return s[int(len(s) * p / 100) - 1]
print(f"requests: {len(latencies):,}")
print(f"average: {statistics.mean(latencies):.0f} ms")
for p in (50, 95, 99):
print(f"p{p}: {pct(p):.0f} ms")
Output:
requests: 100,000
average: 156 ms
p50: 111 ms
p95: 147 ms
p99: 2395 ms
- Latency is how long a request takes, here in ms (milliseconds, thousandths of a second).
random.seed(7)makes the "random" numbers the same on every run, so you get exactly this output.random.gauss(110, 20)draws a time around 110 ms, usually within about 20 ms of it.- 98,000 fast checkouts (about 110 ms) and 2,000 slow ones (about 2,400 ms, waiting on a slow payment).
- A percentile answers "how long did the X% fastest requests take?" p99 is the time that 99% of requests beat (episode 4): sort all 100,000 times and take the one at position 99,000.
pctdoes exactly that (- 1because Python lists start at 0).
The average lies
The average says 156 ms. That looks healthy. Check the arithmetic: 0.98 × 110 + 0.02 × 2,400 = 107.8 + 48 = 155.8 ms. The 2,000 slow requests only move the average up by about 48 ms, because they are drowned out by 98,000 fast ones.
| Number | Value | What it says |
|---|---|---|
| average | 156 ms | "Everything is fine" |
| p50 (the median) | 111 ms | Half the checkouts took 111 ms or less |
| p95 | 147 ms | 95 in 100 took 147 ms or less |
| p99 | 2,395 ms | 1 in 100 waited about 2.4 seconds |
The slowest requests are called the tail (on a chart of times, they are the long thin end on the right), and their latency is tail latency. As archman.dev puts it: "Percentiles expose tail latency", and "p99 catches slowness affecting 1 in 100 users". At 100,000 checkouts a day, 1 in 100 is about 1,000 unhappy customers every day, and the average hid all of them.
Two more things to notice:
- Real dashboards show latency as percentiles, not averages.
- Five numbers summarise 100,000 requests. That's why metrics are cheap to keep for months, while keeping every log line is expensive. Metrics give the trend; logs give the detail.
Real metric systems don't keep every single latency like our list does. A histogram keeps only a count per bucket (for example "under 100 ms", "100 to 250 ms", ..., "over 2.5 s") and estimates percentiles from those counts, which costs a few numbers instead of 100,000.
The four golden signals
Which metrics do you watch first? Google's Site Reliability Engineering book (SRE: the job of keeping big systems running reliably, using software engineering) answers in chapter 6, "Monitoring Distributed Systems": "The four golden signals of monitoring are latency, traffic, errors, and saturation".
| Signal | Question it answers | ChaiNow example |
|---|---|---|
| Latency | How long do requests take? | checkout p50 111 ms, p99 2,395 ms |
| Traffic | How much demand is there? | 70 checkouts per second at lunch |
| Errors | How many requests fail? | 0.3% of checkouts return a 500 or time out |
| Saturation | How "full" is the service? | payments-db disk 85% used, queue length 400 |
- Traffic is usually requests per second (archman.dev).
- Errors include HTTP 5xx responses (the server failed, like
500 Internal Server Error), connection failures and timeouts (the caller gave up waiting) (archman.dev). - Saturation is "How 'full' your service is" (the SRE book): CPU, memory, disk, network, or queue depth (archman.dev). A service at 100% of its CPU, or with a queue that keeps growing, is about to get slow.
- The SRE book also says "it's important to track error latency, as opposed to just filtering out errors". Example: a failed checkout that times out after 30 seconds is very different from one that fails in 5 ms, and if you drop failures from your latency numbers, the 30-second ones disappear from view.
If you watch only four numbers per service, watch these.
Problem 3: which service is slow? (traces)
The p99 alarm rings. But one checkout passes through six services. Which one is slow?
It's like a parcel crossing five courier hubs, each with its own logbook. Searching every hub's book for your parcel is hopeless. Our logs have the same problem: each service's lines don't say which customer request they belong to. So the courier prints one tracking number on the parcel, and every hub scans it.
For requests, that tracking number is the trace id: a random id made once, at the front door (the gateway), and passed along with every call the request causes. Each piece of work (one service handling its part) reports a span: which trace it belongs to, when it started, and how long it took. All the spans with the same trace id together form a trace: the whole journey of one request. Following requests across many services this way is distributed tracing.
import uuid
spans = [] # what the tracing tool receives
def call(trace_id, service, start, own_ms, children=()):
t = start + own_ms # this service's own work
for child in children: # then it calls the next ones
t = child(trace_id, t)
spans.append((start, t - start, service, trace_id))
return t
def menu(tid, t): return call(tid, " menu", t, 40)
def fraud(tid, t): return call(tid, " fraud-check", t, 60)
def db(tid, t): return call(tid, " payments-db", t, 2100)
def payment(tid, t): return call(tid, " payment", t, 20, [fraud, db])
def orders(tid, t): return call(tid, " orders", t, 30, [menu, payment])
def gateway(tid, t): return call(tid, "gateway", t, 10, [orders])
trace_id = uuid.uuid4().hex # made once, at the front door
print("traceparent: 00-" + trace_id + "-" + uuid.uuid4().hex[:16] + "-01")
gateway(trace_id, 0)
for start, ms, service, tid in sorted(spans):
bar = " " * (start // 100) + "#" * max(1, ms // 100)
print(f"{service:<15}{start:>5} ms {ms:>5} ms {bar}")
Output (one run; the trace id and parent id are random, so they change every run, while the times are always the same):
traceparent: 00-810ec1e0c1c8447d99b5706222677ce1-e002ddd929304a95-01
gateway 0 ms 2260 ms ######################
orders 10 ms 2250 ms ######################
menu 40 ms 40 ms #
payment 80 ms 2180 ms #####################
fraud-check 100 ms 60 ms #
payments-db 160 ms 2100 ms #####################
How it works:
calldoes a service's own work (own_ms), then calls its children (the services it depends on) one after another, then records its span:(start, duration, service, trace_id). A span's duration includes the time spent waiting for its children.- The call tree: gateway → orders → (menu, then payment) → payment → (fraud-check, then payments-db). The spaces in the names show the depth.
uuid.uuid4().hexmakes a random 32-character hex id (hex = base 16, digits 0-9 and a-f). The sametrace_idis passed to every function, just as a real service passes it to the next one in a request header.- Each
#is 100 ms, and the spaces before it show when the span started: a waterfall chart.
Reading the waterfall:
- The whole request: 2,260 ms (the gateway's span). Check: 10 + 30 + 40 + 20 + 60 + 2,100 = 2,260.
- Menu took 40 ms, fraud check 60 ms. payments-db took 2,100 ms, about 93% of the request (2,100 / 2,260).
- Orders and payment look slow too (2,250 and 2,180 ms), but only because they wait for their children. The span with a long bar and no children under it is where the time really went.
Found it in one look: the database behind payments.
How tracing works for real: traceparent
Every service must pass the trace id to the next one in the same way, or the chain breaks. So the header is a standard: W3C Trace Context, a W3C Recommendation since 23 November 2021 (the W3C, World Wide Web Consortium, is the group that writes web standards; a "Recommendation" is its finished standard). The header is called traceparent:
traceparent: version-trace_id-parent_id-trace_flags
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
| Part | Example | Meaning |
|---|---|---|
| version | 00 |
Which version of the format |
| trace-id | 0af7651916cd43dd8448eb211c80319c |
The whole request's id: 16 bytes, written as 32 hex characters |
| parent-id | b7ad6b7169203331 |
The id of the calling span (the parent): 8 bytes, 16 hex characters |
| trace-flags | 01 |
A small field of settings for this trace |
(One byte is two hex characters, so 16 bytes = 32 characters and 8 bytes = 16 characters.) When orders calls payment, it sends the same trace-id with its own span id as the parent-id. That's how the tracing tool knows payment's span sits under orders' span in the waterfall. In our demo the parent relationship is shown by indentation; a real span records its parent's id.
Joining logs to traces
The trick that joins everything: put the trace id in every log line too (homework 1 below). Then from the slow trace you jump straight to that one request's log lines, out of millions.
| Tool | Tells you | ChaiNow at 2 a.m. |
|---|---|---|
| Metric | When something went wrong, and how bad | p99 jumped to 2.4 s at 2:04 |
| Trace | Where the time went | payments-db: 2,100 of 2,260 ms |
| Log | Why | {"level": "WARN", "service": "payments-db", "msg": "slow query", ...} |
These three (logs, metrics, traces) are often called the three pillars of observability.
Problem 4: what should wake you up? (alerts)
An alert is a rule on a metric that notifies a person (a phone call, a message) when it's true. Which rules deserve a 2 a.m. phone call?
| Alert | Good? | Why |
|---|---|---|
| CPU at 90% on one server | ✗ Usually not | Might be completely fine; customers may not notice at all |
| Checkout p99 > 1 s for 5 minutes | ✓ Yes | Customers are hurting right now |
| Error rate > 1% for 5 minutes | ✓ Yes | Real requests are failing |
| Disk 85% full, filling 1% a day | Ticket, not a call | Needs action, but this week, not tonight |
The SRE book calls this symptoms versus causes: "The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause".
- Symptom (what's broken, what users feel): slow checkout, failed payments.
- Cause (why it's broken): a slow database, a full disk, a bad deploy.
Alert on symptoms, then use traces and logs to find the cause. A cause alert fires when nothing is wrong for users (high CPU at a busy moment) and stays silent for causes you didn't think of; a symptom alert fires whenever users are hurt, whatever the cause.
Two more rules:
- "For 5 minutes" avoids alerts for a single bad moment.
- Every alert needs an action, or delete it. If an alert wakes people up and needs no action, it trains them to ignore alerts, including the real one. Fewer, better alerts means people trust them.
The mini project below runs a symptom alert over a simulated night, next to a CPU spike that a cause alert would have woken you up for.
Problem 5: every tool has its own plug (OpenTelemetry)
Every monitoring product used to come with its own code library: you'd add tool A's library to every service to send logs, metrics and traces. Like phone chargers, every brand its own plug. Switching to tool B meant rewriting that code in every service. Being stuck with one supplier because leaving is too expensive is called vendor lock-in (a vendor is a company that sells a product).
Then came one standard plug, like USB-C: OpenTelemetry. According to last9.io:
- It is "an open-source observability framework" (open source: its code is public and anyone can use it for free).
- It was formed by merging two older projects: OpenCensus (from Google) and OpenTracing.
- It collects all three: traces, metrics and logs (together called telemetry: data a system sends about itself).
- Its protocol, OTLP (the OpenTelemetry Protocol), is "a vendor-neutral, tool-agnostic protocol": your services send data the same way, and you can change where it goes (which tool stores and shows it, called the backend) without touching their code.
- It is the second most active CNCF project. The CNCF (Cloud Native Computing Foundation) is a foundation that hosts open-source projects for running software in the cloud.
| Phone chargers | Telemetry |
|---|---|
| Every brand, its own plug | Every tool, its own library: switching = rewrite |
| One USB-C for all | OpenTelemetry: one way to produce traces, metrics, logs |
| Change the charger, keep the phone | Change the backend, keep the code (OTLP) |
The whole chain
- Free-text logs: the 2 a.m. question crashed with
ValueError: invalid literal for int() with base 10: 's'. - Structured JSON logs + levels: same fields everywhere, the answer is one filter:
[5514]. - Metrics: 100,000 requests in five numbers; the average said 156 ms, p99 said 2,395 ms.
- Golden signals: latency, traffic, errors, saturation.
- Traces: one trace id from the front door, spans as a waterfall: payments-db took 2,100 of 2,260 ms.
- Alert on symptoms (what users feel), not causes; OpenTelemetry collects all three, vendor-neutral.
Common mistakes (and the real outputs)
| Mistake | Code | Output | Fix |
|---|---|---|---|
| Free-text logs | print(f"took {ms}ms"), then parse the last word |
ValueError: invalid literal for int() with base 10: 's' |
JSON fields: {"ms": 180} |
| Watching only the average | dashboard: average latency | average: 156 ms while p99: 2395 ms |
Show p50, p95, p99 |
| Logs without a trace id | {"service": "payments-db", "ms": 2100} |
which order? None |
Pass traceparent, log trace_id |
| Inconsistent field values | city is "pune" in one service, "Pune" in another |
[5516] instead of [5514, 5516] |
Agree on values (lowercase), or normalize when filtering |
| Off-by-one percentile | s[int(len(s) * p / 100)] |
IndexError: list index out of range for p100 |
s[int(len(s) * p / 100) - 1] |
| Alerting on causes | page when CPU > 90% | woken at 2:01 while p99 was 115 ms (mini project) | Alert on symptoms: p99, error rate |
The third mistake, run for real (two checkouts at the same second, logs without a trace id):
import json
lines = [ # two checkouts at the same second, no trace id
'{"service": "orders", "order": 5514, "ms": 2300}',
'{"service": "orders", "order": 5517, "ms": 90}',
'{"service": "payments-db", "ms": 2100, "msg": "slow query"}',
'{"service": "payments-db", "ms": 12, "msg": "ok"}',
]
events = [json.loads(l) for l in lines]
db = [e for e in events if e["service"] == "payments-db"]
print("db lines:", len(db), "| which order?", db[0].get("order"))
Output:
db lines: 2 | which order? None
The database knows a query was slow, but not which checkout it belonged to. dict.get returns None when a key is missing.
The fourth mistake:
import json
lines = ['{"order": 5514, "city": "pune", "ms": 2300}',
'{"order": 5516, "city": "Pune", "ms": 1900}']
events = [json.loads(l) for l in lines]
print([e["order"] for e in events if e["city"] == "Pune"])
print([e["order"] for e in events if e["city"].lower() == "pune"])
Output:
[5516]
[5514, 5516]
Structure fixes the shape of the data, not its values: services still have to agree on units and spelling.
The fifth mistake:
latencies = [120, 95, 2400, 110, 101]
def pct(p): # forgot the "- 1"
s = sorted(latencies)
return s[int(len(s) * p / 100)]
print("p50:", pct(50))
print("p100 (the max):", pct(100))
Output:
p50: 110
Traceback (most recent call last):
File "m_pct.py", line 8, in <module>
print("p100 (the max):", pct(100))
~~~^^^^^
File "m_pct.py", line 5, in pct
return s[int(len(s) * p / 100)]
~^^^^^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range
With 5 values, p100 asks for s[5], but the positions are 0 to 4 (Python lists start at 0). The episode's pct subtracts 1, so p100 is s[4], the maximum (2,400). Without the - 1, every percentile also shifts one place up: p50 here returned the 3rd value (110), where the episode's rule gives the 2nd (101). Over 100,000 values a one-place shift barely matters; over 5 it does. (Real tools use slightly different percentile rules; what matters is using one rule consistently.)
(File paths in the tracebacks are shortened to the file name.)
Mini project: a tiny observability toolkit
All three pillars in about 80 lines of plain Python: a structured logger that puts the trace id in every line, a metrics class with request and error counters and percentiles, spans for each checkout, and a symptom alert ("p99 > 1000 ms for 3 minutes in a row") over a simulated night of 3,000 checkouts. At 2:01 the CPU spikes to 95% but users are fine; at 2:04 the payments database starts making 2% of checkouts slow. The random seed makes every number repeat, including the trace ids (here made with random.getrandbits(128) instead of uuid, so they are seeded too).
# Mini project: a tiny observability toolkit (logs, metrics, traces)
import json, random
random.seed(25)
# ---- 1. structured logs, every line carries the trace id
def log(level, trace_id, **fields):
return json.dumps({"level": level, "trace_id": trace_id, **fields})
# ---- 2. metrics: a counter for requests and errors, raw latencies
class Metrics:
def __init__(self):
self.requests = 0 # counter: only goes up
self.errors = 0 # counter
self.latencies = [] # for percentiles
def record(self, ms, ok):
self.requests += 1
self.errors += 0 if ok else 1
self.latencies.append(ms)
def pct(self, p):
s = sorted(self.latencies)
return s[max(0, int(len(s) * p / 100) - 1)]
# ---- 3. traces: each piece of work reports a span
def checkout(trace_id, slow_db, spans):
t = 0
for service, ms in [("orders", random.gauss(30, 5)),
("payment", random.gauss(20, 4)),
("payments-db", random.gauss(
1800 if slow_db else 40, 10))]:
spans.append({"trace_id": trace_id, "service": service,
"start": round(t), "ms": round(ms)})
t += ms
ok = random.random() > (0.03 if slow_db else 0.002)
return t, ok
# ---- 4. a simulated night: 10 minutes, 300 checkouts a minute
rule = "p99 > 1000 ms for 3 minutes in a row"
bad_minutes, logs, spans = 0, [], []
for minute in range(10):
m = Metrics()
slow_db = minute >= 4 # the database slows at 2:04
cpu = 95 if minute == 1 else random.randint(35, 50) # a gauge
for _ in range(300):
tid = f"{random.getrandbits(128):032x}" # seeded, repeatable
hit = slow_db and random.random() < 0.02 # 2% hit the slow path
ms, ok = checkout(tid, hit, spans)
m.record(ms, ok)
if not ok or ms > 1000:
logs.append(log("ERROR" if not ok else "WARN", tid,
minute=minute, ms=round(ms)))
p99 = m.pct(99)
bad_minutes = bad_minutes + 1 if p99 > 1000 else 0
alert = " <-- ALERT" if bad_minutes == 3 else ""
print(f"2:{minute:02d} {m.requests}/min"
f" err {m.errors / m.requests:4.1%}"
f" p50 {m.pct(50):4.0f} p99 {m.pct(99):4.0f} ms"
f" cpu {cpu}%{alert}")
print("alert rule:", rule)
# ---- 5. from the alert to the cause: one slow trace, then its logs
slow = max((s for s in spans if s["service"] == "payments-db"),
key=lambda s: s["ms"])
print("\nslowest trace:", slow["trace_id"][:8], "...")
for s in spans:
if s["trace_id"] == slow["trace_id"]:
print(f" {s['service']:<12}{s['start']:>5} ms {s['ms']:>5} ms")
print("its log line:", *[l for l in logs if slow["trace_id"] in l])
print(f"total: {len(logs)} WARN/ERROR lines, {len(spans)} spans")
Output (identical on every run):
2:00 300/min err 0.0% p50 91 p99 114 ms cpu 47%
2:01 300/min err 0.0% p50 90 p99 115 ms cpu 95%
2:02 300/min err 0.0% p50 90 p99 116 ms cpu 44%
2:03 300/min err 0.0% p50 93 p99 119 ms cpu 44%
2:04 300/min err 0.0% p50 90 p99 1851 ms cpu 36%
2:05 300/min err 0.0% p50 90 p99 1846 ms cpu 41%
2:06 300/min err 0.0% p50 90 p99 1848 ms cpu 42% <-- ALERT
2:07 300/min err 0.7% p50 93 p99 1854 ms cpu 40%
2:08 300/min err 0.0% p50 90 p99 1854 ms cpu 39%
2:09 300/min err 0.0% p50 89 p99 122 ms cpu 35%
alert rule: p99 > 1000 ms for 3 minutes in a row
slowest trace: a858fca2 ...
orders 0 ms 36 ms
payment 36 ms 17 ms
payments-db 53 ms 1817 ms
its log line: {"level": "WARN", "trace_id": "a858fca2d2f1dfe6962b44ba45ee3efc", "minute": 5, "ms": 1870}
total: 45 WARN/ERROR lines, 9000 spans
What to notice:
- Each row shows golden signals: traffic (300/min), errors (err), latency (p50, p99), and saturation (cpu, a gauge).
- The CPU spike at 2:01 hurt nobody: p99 stayed at 115 ms. A "CPU > 90%" cause alert would have woken you for nothing.
- The p50 never moved (about 90 ms all night) and the error rate barely moved. Only p99 showed the problem, because only 2% of checkouts were slow.
- The alert fired at 2:06, after p99 was over 1 second for 3 minutes in a row (2:04, 2:05, 2:06), and fired once, not every minute.
- At 2:09, p99 dropped back to 122 ms: by chance, only a few of that minute's 300 checkouts hit the slow path (p99 of 300 is the 297th fastest, so it needs at least 4 slow ones). Percentiles over small windows are noisy, which is one reason alert rules wait several minutes.
- From when to where to why: the metric said when (2:04), the slowest trace said where (payments-db, 1,817 of 1,870 ms), and the trace id led straight to the log line for that one checkout, among 45 WARN/ERROR lines and 9,000 spans.
Extend it: add a Histogram class that keeps counts in buckets (under 100 ms, 100-250, 250-500, 500-1000, 1000-2500, over 2500) instead of every latency; add an error-rate symptom alert ("errors > 1% for 3 minutes"); give each span a span_id and parent_id and build a real traceparent header for each call.
Practice
- Homework: in the JSONLOG demo, add a
trace_idfield and find every line for one checkout. - Homework: in the METRICS demo, make 5% of requests slow instead of 2%. What happens to p95? What about 6%?
- Homework: in the TRACE demo, make fraud-check the slow one (2,100 ms) and payments-db fast (60 ms). What changes in the waterfall?
- Homework: write the four golden signals for an app you use every day.
- 1,000,000 requests: 990,000 take 100 ms and 10,000 take 5,000 ms. What is the average? Roughly what are p50 and p99?
- A service's p99 is 2 s and it handles 50 requests a second. About how many requests a minute take 2 s or longer?
- A traceparent is
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Which part is the trace id, which is the parent id, and how many bytes is each? - Classify each as a symptom or a cause: (a) checkout error rate 4%; (b) payments-db disk 99% full; (c) p99 search latency 3 s; (d) a server's memory at 92%.
Solutions and hints
1. Give log a trace_id and use the same one for every line of one checkout:
import json, uuid
def log(level, **fields): # one JSON object per line
line = json.dumps({"level": level, **fields})
print(line)
return line
tid = uuid.uuid4().hex # one checkout = one trace id
lines = [
log("INFO", trace_id=tid, service="orders", order=5514, ms=2300),
log("INFO", trace_id=tid, service="payment", order=5514, ms=2180),
log("WARN", trace_id=tid, service="payments-db", ms=2100,
msg="slow query"),
log("INFO", trace_id=uuid.uuid4().hex, service="orders",
order=5515, ms=95),
]
events = [json.loads(l) for l in lines]
mine = [e["service"] for e in events if e["trace_id"] == tid]
print("lines for this checkout:", mine)
Output (one run; the ids are random):
{"level": "INFO", "trace_id": "ac2655f8ff92438f9ff1164b086ddf19", "service": "orders", "order": 5514, "ms": 2300}
{"level": "INFO", "trace_id": "ac2655f8ff92438f9ff1164b086ddf19", "service": "payment", "order": 5514, "ms": 2180}
{"level": "WARN", "trace_id": "ac2655f8ff92438f9ff1164b086ddf19", "service": "payments-db", "ms": 2100, "msg": "slow query"}
{"level": "INFO", "trace_id": "69ecf5344f454343aa146de2ef87b399", "service": "orders", "order": 5515, "ms": 95}
lines for this checkout: ['orders', 'payment', 'payments-db']
The payments-db line has no order field, but the trace id still ties it to order 5514. Compare the "which order? None" mistake above.
2. Wrap the demo in a function that takes the slow share:
import random, statistics
random.seed(7)
def report(slow_share):
n_slow = int(100_000 * slow_share)
lat = [random.gauss(110, 20) for _ in range(100_000 - n_slow)]
lat += [random.gauss(2400, 300) for _ in range(n_slow)]
s = sorted(lat)
pct = lambda p: s[int(len(s) * p / 100) - 1]
print(f"{slow_share:>4.0%} slow: average {statistics.mean(lat):>4.0f}"
f" p50 {pct(50):>4.0f} p95 {pct(95):>4.0f}"
f" p99 {pct(99):>4.0f}")
for share in (0.02, 0.05, 0.06):
report(share)
Output:
2% slow: average 156 p50 111 p95 147 p99 2395
5% slow: average 225 p50 111 p95 191 p99 2662
6% slow: average 248 p50 112 p95 2118 p99 2680
The 2% row is the video's demo exactly (same seed, same draws). At 5%, p95 lands exactly on the edge: position 95,000 is the slowest fast request (191 ms), so p95 only creeps up. At 6%, the slow requests start at position 94,001, so p95 jumps to 2,118 ms. A percentile pX shows a problem once more than (100 − X)% of requests have it: p95 needs more than 5%, p99 more than 1%. The average (225, 248 ms) still looks almost fine.
3. Swap the two durations: fraud(...) gets 2100 and db(...) gets 60.
traceparent: 00-b6eb778a7dc5487c8dc08d6392a3c0e0-86c0c003bc0c4ea4-01
gateway 0 ms 2260 ms ######################
orders 10 ms 2250 ms ######################
menu 40 ms 40 ms #
payment 80 ms 2180 ms #####################
fraud-check 100 ms 2100 ms #####################
payments-db 2200 ms 60 ms #
(The ids are random.) The total is still 2,260 ms and the metric would look identical; only the trace shows the slow span moved. payments-db now starts at 2,200 ms: it had to wait for fraud-check to finish, because payment calls them one after another.
4. Example, a food delivery app's "place order" service: latency = p50 and p99 time to place an order; traffic = orders per minute; errors = % of orders that fail with a 5xx or time out; saturation = CPU, database connections in use, or the length of the queue of orders waiting for a restaurant.
5. Average = 0.99 × 100 + 0.01 × 5,000 = 99 + 50 = 149 ms. p50 = 100 ms (the 500,000th request is fast). p99 = the 990,000th: also a fast one, 100 ms, right on the edge (like 5% and p95 in question 2). Just one more slow request (more than 1%) and p99 would become 5,000 ms. That's why teams also look at p99.9 for very busy services.
6. 50 a second × 60 = 3,000 requests a minute; 1% of that is about 30 requests a minute taking 2 s or longer, or about 43,200 a day (50 × 86,400 × 1%).
7. Format: version-trace_id-parent_id-flags. Trace id: 4bf92f3577b34da6a3ce929d0e0e4736 (32 hex characters = 16 bytes). Parent id: 00f067aa0ba902b7 (16 hex characters = 8 bytes). Version 00, flags 01.
8. (a) symptom, users see failures; (b) cause, it may make writes fail, but the symptom is what users see; (c) symptom; (d) cause, and maybe nothing is wrong at all. Alert on (a) and (c); show (b) and (d) on dashboards and use them to explain why (a full disk is still worth a ticket before it causes a symptom).
Quick quiz
# Checkout p99 jumped to 3 s at 2:05 a.m.
# Which tool shows WHICH SERVICE
# was slow for one of those requests?
- A. A metric
- B. A trace
- C. A log level
- D. The average
Answer
B. A trace. It follows one request through every service as spans, so the slow span shows where the time went. The metric (p99) told you when; the logs will tell you why. A log level only says how serious a line is, and the average would hide the problem in the first place.
Interview questions
- What is observability, and how is it different from monitoring? (Being able to explain what's happening inside from the data the system emits; monitoring is watching chosen metrics and alerting on known failure modes.)
- What are logs, metrics and traces, and what is each good for? (Logs: detailed events, the why; metrics: cheap numbers over time, trends and alerts, the when; traces: one request across services, the where.)
- Why structured logs instead of plain text? (Queryable fields, no fragile parsing, consistent across services, filter by level; add trace_id to join with traces.)
- Why use percentiles instead of the average for latency? (The average hides the tail: in our demo, average 156 ms vs p99 2,395 ms; p99 shows the slowness 1 in 100 users feel.)
- What are the four golden signals? (Latency, traffic, errors, saturation, from Google's SRE book; track the latency of errors separately.)
- How does distributed tracing work? (A trace id made at the entry point, propagated in a header (W3C traceparent) on every call; each service reports spans with start, duration and parent span id; the backend assembles the waterfall.)
- What is in a traceparent header? (version-trace_id-parent_id-flags; 16-byte trace id, 8-byte parent id, in hex.)
- What should you alert on? (Symptoms users feel, such as high p99 or error rate, sustained for a few minutes; not causes like CPU; every alert actionable; causes go on dashboards or tickets.)
- Why can't you keep every log line forever? (Volume and cost; keep metrics long term, logs for a shorter time, and keep only some traces if needed.)
- What is OpenTelemetry and why use it? (An open-source framework for traces, metrics and logs, from merging OpenCensus and OpenTracing; OTLP is vendor-neutral, so you can change backends without changing service code.)
- How would you design observability for ChaiNow? (JSON logs with trace_id from every service collected centrally; golden-signal metrics per service with latency histograms; W3C traceparent propagated through the gateway; symptom alerts on checkout p99 and error rate; OpenTelemetry in every service.)
Glossary
| Term | Meaning |
|---|---|
| Observability | Being able to tell what's happening inside a running system from what it reports (logs, metrics, traces) |
| Monitoring | Watching chosen numbers about a system and getting alerted when they go bad |
| Telemetry | Data a system sends about itself: logs, metrics and traces |
| Three pillars | Logs, metrics and traces, the three kinds of observability data |
| Service | One program in a system that does one job, e.g. payment (microservices, episode 17) |
| Log | A record of events, one line per event with its time, e.g. order 5512 placed |
| Free-text log | A log line written as a plain sentence, each developer in their own style |
| Structured log | A log line made of named fields, e.g. {"city": "pune", "ms": 180} |
| JSON | JavaScript Object Notation: a text format for named fields and values, readable by almost every language |
json.dumps / json.loads |
Python functions that turn a dictionary into JSON text, and JSON text back into a dictionary |
| Field | One named value in a structured log, e.g. ms |
| Log level | How serious a log line is: DEBUG, INFO, WARN, ERROR |
| DEBUG / INFO / WARN / ERROR | Developer detail / normal event / unusual but worked / something failed |
| Log tool | Software that collects log lines from every server into one place and lets you search them |
ValueError |
Python's error for a value of the right type but wrong content, e.g. int("s") |
IndexError |
Python's error for a list position that doesn't exist, e.g. s[5] in a 5-item list |
| Metric | A number measured over time, e.g. requests per second |
| Counter | A metric that only goes up, e.g. total requests |
| Gauge | A metric that goes up and down, read as its current value, e.g. CPU % |
| Histogram | Counts of values in ranges (buckets), used to estimate percentiles cheaply |
| Bucket | One range in a histogram, e.g. "100 to 250 ms" |
| Dashboard | A screen of charts of metrics |
| Latency | How long a request takes, e.g. 110 ms |
| ms (millisecond) | One thousandth of a second |
| Average / mean | Sum of the values divided by how many; hides rare slow requests |
| Median | The middle value when sorted; the same as p50 |
| Percentile (pX) | The value that X% of requests are at or below, e.g. p99 = 2,395 ms |
| p50 / p95 / p99 | The time that 50% / 95% / 99% of requests beat |
| Tail / tail latency | The slowest requests, and how slow they are (what p99 shows) |
random.seed |
Fixes Python's random numbers so a program gives the same output every run |
random.gauss(mu, sigma) |
A random number around mu, usually within about sigma of it |
| Golden signals | Latency, traffic, errors, saturation: the four metrics to watch first (Google SRE book) |
| Traffic | How much demand: e.g. requests per second |
| Errors | Requests that fail: HTTP 5xx, connection failures, timeouts |
| HTTP 5xx | Response codes 500-599: the server failed, e.g. 500 Internal Server Error |
| Timeout | The caller stopping waiting after a time limit |
| Saturation | How "full" a service is: CPU, memory, disk, network, queue depth |
| Queue depth | How many jobs are waiting in a queue |
| Error latency | How long failed requests took; tracked separately from successful ones |
| SRE | Site Reliability Engineering: keeping big systems reliable using software engineering; also Google's book of that name |
| Trace | The whole journey of one request across services, made of spans |
| Span | One piece of work in a trace: a service, its start time and duration, its trace id and parent |
| Distributed tracing | Following requests across many services with traces |
| Trace id | A random id made once at the entry point and passed with every call of that request |
| Parent id | The id of the span that made this call; puts spans in a tree |
| Child span | A span created by a call from another span (its parent) |
| Waterfall | A chart of spans as bars by start time and duration |
| Header | Extra named information sent with an HTTP request, e.g. traceparent: ... |
| traceparent | The W3C header carrying version, trace-id, parent-id and trace-flags |
| Trace-flags | The last part of traceparent: a small settings field (e.g. 01) |
| W3C | World Wide Web Consortium: the group that writes web standards |
| W3C Recommendation | A finished W3C standard; Trace Context became one on 23 November 2021 |
| W3C Trace Context | The standard that defines the traceparent header |
| Hex | Base 16 (digits 0-9, a-f); two hex characters = one byte |
| Byte | 8 bits; the trace id is 16 bytes (32 hex characters) |
| UUID | A random 128-bit id; uuid.uuid4().hex gives 32 hex characters |
| Alert | A rule on a metric that notifies a person when it's true |
| Symptom | What's broken, what users feel: slow checkout, failures |
| Cause | Why it's broken: a slow database, a full disk |
| Symptom alert | An alert on what users feel, e.g. p99 > 1 s for 5 minutes |
| False alarm | An alert that fires when nothing needs doing |
| Vendor | A company that sells a product, such as a monitoring tool |
| Vendor lock-in | Being stuck with one vendor because leaving is too expensive |
| Backend (observability) | The tool that stores and shows telemetry |
| Open source | Software whose code is public and free to use |
| OpenTelemetry | An open-source framework to produce and collect traces, metrics and logs; merged OpenCensus and OpenTracing |
| OpenCensus | An older telemetry project from Google, merged into OpenTelemetry |
| OpenTracing | An older tracing project, merged into OpenTelemetry |
| OTLP | OpenTelemetry Protocol: the vendor-neutral way to send telemetry |
| Vendor-neutral | Not tied to any one company's product |
| CNCF | Cloud Native Computing Foundation: hosts open-source cloud projects; OpenTelemetry is its second most active |
Summary
- Observability = seeing inside a running system through logs (why), metrics (when) and traces (where).
- Free-text logs break the first time a line looks different; structured JSON logs with levels turn questions into filters.
- Metrics summarise millions of requests in a few numbers; show latency as percentiles, because the average hides the tail (156 ms average vs 2,395 ms p99).
- Watch the four golden signals first: latency, traffic, errors, saturation (and the latency of errors).
- A trace id made at the front door and passed in the W3C traceparent header ties every span of one request together; the waterfall shows the slow service in one look. Put the trace id in every log line.
- Alert on symptoms users feel (p99, error rate, for a few minutes), not causes; every alert needs an action.
- OpenTelemetry is the open, vendor-neutral standard for producing all three, with OTLP to send them anywhere.
Next episode: every one of these services trusts every request. How does the payment service know a request really came from Priya, and not from someone pretending? Security basics: authentication, OAuth, JWT and encryption.
Sources: Notes on Google's SRE book, ch. 6 "Monitoring Distributed Systems" (dev.to, Hercules Lemke Merscher: the four golden signals, saturation "How 'full' your service is", tracking error latency, "what's broken" = symptom vs "why" = cause); archman.dev, golden signals (traffic, errors, saturation examples; "Percentiles expose tail latency"; "p99 catches slowness affecting 1 in 100 users"); W3C Trace Context, W3C Recommendation 23 November 2021 (traceparent format and example, 16-byte trace-id, 8-byte parent-id); last9.io, "What is OpenTelemetry" (open-source framework, OpenCensus from Google and OpenTracing, traces/metrics/logs, OTLP "a vendor-neutral, tool-agnostic protocol", second most active CNCF project).