SyntaxPilot
Courses/AI: Zero to Agents/Lesson 36
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

AI: Zero to Agents · Lesson 36 of 40

Observing and evaluating agents

queued

These are the notes for AI episode 36. Guardrails (episode 35) limit the damage when an agent is wrong. This episode answers the next questions: how do you find out what an agent did, step by step, and how do you know whether it's getting better or worse? Each idea arrives as the fix for a failure on screen: a final answer that hides a bug, a slow step you can't find, a single passing run that proves nothing, a right answer reached by a wrong path, and a prompt change that keeps the same score while breaking something else. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13; timings rounded to 0.1 s). The agents are toys (if-rules and a seeded random number) standing in for real model calls.

Real-life example: a delivery app and a plane's black box

In real life For an agent
Live order tracking: packed, picked up, on the way trace: every step of one request
"Stuck at the restaurant for 25 minutes" spans with timings: which step is slow or costly
A plane's black box, read after a problem logs you can replay: why did it do that?
Many test drives before a car is sold evals: many runs, many tasks, before shipping

Problem 1: a silent failure

TRAINS = {"08-11": [12127, 11007]}     # the API wants day-month numbers

def search_trains(date):
    return TRAINS.get(date, [])

def agent(question):                   # toy LLM agent
    date = "8 Nov"                     # what the model wrote
    trains = search_trains(date)
    if not trains:
        return "Sorry, no trains to Goa on 8 Nov."
    return f"Trains: {trains}"

print(agent("Any trains to Goa on 8 Nov?"))

Output:

Sorry, no trains to Goa on 8 Nov.

There are two trains on that day (line 1). When only the final answer is visible, every mistake in the middle is invisible.

The fix: record every step (a trace)

TRAINS = {"08-11": [12127, 11007]}
trace = []                             # the agent's flight recorder

def search_trains(date):
    result = TRAINS.get(date, [])
    trace.append({"step": "search_trains", "input": date, "output": result})
    return result

def agent(question):
    trace.append({"step": "llm", "input": question, "output": "date = 8 Nov"})
    trains = search_trains("8 Nov")
    answer = f"Trains: {trains}" if trains else "Sorry, no trains."
    trace.append({"step": "answer", "input": "", "output": answer})
    return answer

agent("Any trains to Goa on 8 Nov?")
for i, span in enumerate(trace, 1):
    print(i, span["step"], "| in:", span["input"], "| out:", span["output"])

Output:

1 llm | in: Any trains to Goa on 8 Nov? | out: date = 8 Nov
2 search_trains | in: 8 Nov | out: []
3 answer | in:  | out: Sorry, no trains.

Step 2 shows the bug: search_trains received "8 Nov", but the API stores dates as "08-11". We only found it because the trace showed the middle step.

Fix it, prove it

TRAINS = {"08-11": [12127, 11007]}
MONTHS = {"Nov": "11", "Dec": "12"}
trace = []

def to_api_date(text):                 # "8 Nov" -> "08-11"
    day, month = text.split()
    return f"{int(day):02d}-{MONTHS[month]}"

def search_trains(date):
    result = TRAINS.get(to_api_date(date), [])
    trace.append({"step": "search_trains", "input": date, "output": result})
    return result

trains = search_trains("8 Nov")
print("answer:", f"Trains: {trains}" if trains else "Sorry, no trains.")
for i, span in enumerate(trace, 1):
    print(i, span["step"], "| in:", span["input"], "| out:", span["output"])

Output:

answer: Trains: [12127, 11007]
1 search_trains | in: 8 Nov | out: [12127, 11007]

The same trace now proves the fix: in 8 Nov, out both trains.

Traces and spans

OpenTelemetry, a widely used open standard for monitoring software, defines:

  • Trace: "the path of a request through your application".
  • Span: one "unit of work or operation", such as one model call or one tool call.

Spans form a tree: the whole agent run is the root span (its parent ID is empty) and each step is a child span pointing to its parent.

agent: trains to Goa            (root span)
├── llm: plan                   (child)
├── tool: search_trains         (child)
└── llm: answer                 (child)
Span field Meaning Example
Name what this step is tool: search_trains
Parent span ID which span it belongs to (empty for the root) agent: trains to Goa
Start and end time when it began and finished 1.5 s long
Attributes extra key-value details input, output, tokens, model name
Events things that happened at a moment inside it "retrying"
Status Unset, Error or Ok Error if the tool crashed

Because it's a shared standard, many monitoring tools can read the same traces.

What goes into a span

Record Never record Mask instead
model name, input, output, tokens, time, cost passwords card ****4242
tool name, arguments, result, status API keys phone ******3210
parent span, retries card numbers email a***@example.com

A trace is read by many people (developers, support, monitoring tools), so it must be useful without leaking secrets.

Finding the slow and the expensive step

import time
trace = []
def span(name, parent=None):           # one timed step of the agent
    s = {"name": name, "parent": parent, "start": time.perf_counter()}
    trace.append(s)
    return s
def end(s, tokens=0):
    s["secs"] = time.perf_counter() - s["start"]
    s["tokens"] = tokens

root = span("agent: trains to Goa")
s = span("llm: plan", root["name"]);           time.sleep(0.2); end(s, 900)
s = span("tool: search_trains", root["name"]); time.sleep(1.5); end(s)
s = span("llm: answer", root["name"]);         time.sleep(0.3); end(s, 1400)
end(root, 2300)
for s in trace:
    pad = "  " if s["parent"] else ""
    print(f"{pad}{s['name']:24} {s['secs']:.1f}s  {s['tokens']} tokens")

Output:

agent: trains to Goa     2.0s  2300 tokens
  llm: plan                0.2s  900 tokens
  tool: search_trains      1.5s  0 tokens
  llm: answer              0.3s  1400 tokens

The search tool took 1.5 s of 2.0 s, and the answer step used the most tokens. Fix those (cache the tool, send less history) instead of guessing. Measure before you optimise.

Problem 2: "it passed once, ship it"

import random
random.seed(42)                        # same "luck" every time we run

def agent_run():                       # toy agent: right about 70% of the time
    return random.random() < 0.7

print("one run:", "PASS" if agent_run() else "FAIL", " -> ship it?")
results = [agent_run() for _ in range(8)]
print("8 runs :", " ".join("P" if r else "F" for r in results))
print("passed :", sum(results), "of 8")

Output:

one run: PASS  -> ship it?
8 runs : P P P F P F P P
passed : 6 of 8

A model is not a normal function: the same input can give a different result. One pass is luck, not proof. (random.seed(42) makes the demo repeat the same "luck" every run.)

τ-bench and pass^k

τ-bench (tau-bench; Yao, Shinn, Razavi, Narasimhan, June 2024) tests agents in conversations: a language model plays the user, and the agent must help using domain API tools and policy rules (for example, a retail shop). It introduced pass^k: the chance that k tries of the same task all succeed, a measure of reliability. The paper reports that even strong function-calling agents such as gpt-4o succeeded on under 50% of tasks, with pass^8 under 25% in the retail domain.

The estimate for one task with c passes out of n runs is C(c, k) / C(n, k), averaged over tasks:

from math import comb
runs = {                               # 8 runs of each task: P = pass
    "book 8 Nov train":  "PPPPPPPP",
    "cancel a ticket":   "PPPFPPPF",
    "change the date":   "PFPPFPPP",
    "refund a ticket":   "PPFPPFPF",
}
def pass_hat_k(record, k):             # chance that k runs in a row ALL pass
    n, c = len(record), record.count("P")
    return comb(c, k) / comb(n, k)

for k in (1, 2, 4, 8):
    score = sum(pass_hat_k(r, k) for r in runs.values()) / len(runs)
    print(f"pass^{k} = {score:.0%}")

Output:

pass^1 = 78%
pass^2 = 61%
pass^4 = 38%
pass^8 = 25%
k pass^k Meaning
1 78% a typical single try works
2 61% two in a row both work
4 38% four in a row
8 25% only 1 in 4 chance of 8 clean runs

Same agent, same runs: reliability is the hard part of agents.

Problem 3: right answer, wrong path

def agent(task):                       # toy agent; returns answer + its trace
    trace = ["search_trains(8 Nov)", "book_ticket(12127)",
             "book_ticket(12127)",     # booked again "to be sure"!
             "send_email(asha)"]
    return "Booked 12127 on 8 Nov", trace

answer, trace = agent("book a Goa train on 8 Nov")
print("answer test:", "PASS" if "8 Nov" in answer else "FAIL")

Output:

answer test: PASS

The answer test passed, but the trace shows book_ticket was called twice: two tickets, paid twice. We graded the destination, not the journey.

The fix: grade the trajectory

def agent(task):                       # the same toy agent
    trace = ["search_trains(8 Nov)", "book_ticket(12127)",
             "book_ticket(12127)",
             "send_email(asha)"]
    return "Booked 12127 on 8 Nov", trace

def check_trajectory(trace, max_steps=5):      # grade the PATH too
    problems = []
    books = [t for t in trace if t.startswith("book_ticket")]
    if len(books) != 1:
        problems.append(f"booked {len(books)} times")
    if len(trace) > max_steps:
        problems.append(f"{len(trace)} steps > {max_steps}")
    return problems

answer, trace = agent("book a Goa train on 8 Nov")
print("answer test:", "PASS" if "8 Nov" in answer else "FAIL")
print("path test  :", check_trajectory(trace) or "PASS")

Output:

answer test: PASS
path test  : ['booked 2 times']

Trajectory (path) checks are cheap and powerful: the right tools, in the right order, no forbidden actions, not too many steps, each irreversible action exactly once. You can only write them because you kept a trace.

Problem 4: same score, new bug

SUITE = [                              # test tasks + what must come out
    ("trains on 8 Nov",   "12127"),
    ("cancel PNR 4521",   "cancelled 4521"),
    ("refund PNR 4521",   "refund Rs 450"),
    ("trains on 08-11",   "12127"),
]
def agent(task, prompt):               # toy agent: v2 adds a date rule
    if "08-11" in task:
        return "12127" if "dates" in prompt else "no trains"
    if "refund" in task and "dates" in prompt:
        return "refund Rs 4500"        # the new rule confused it here!
    return {"trains on 8 Nov": "12127", "cancel PNR 4521": "cancelled 4521",
            "refund PNR 4521": "refund Rs 450"}[task]

for name, prompt in [("v1", "You book trains."),
                     ("v2", "You book trains. Read all dates carefully.")]:
    marks = ["P" if agent(task, prompt) == want else "F" for task, want in SUITE]
    print(name, " ".join(marks), f"{marks.count('P')}/{len(SUITE)}")

Output:

v1 P P P F 3/4
v2 P P F P 3/4

Both prompts score 3/4, but different tasks fail: v2 fixed the 08-11 date task and broke the refund (Rs 4500 instead of Rs 450). A total score hid an expensive regression.

The fix: an eval suite, compared task by task

change (prompt / model / tool)
        │
        ▼
run the suite: every task, k times
        │
        ▼
compare with the last version TASK BY TASK
        │
        ▼
ship only if no task got worse
  • Regression testing: re-running old tests after a change, to catch things that used to work and now don't.
  • Every bug you find becomes a new task in the suite, so it can't come back unnoticed.
  • Check answers and paths in code where you can; for open-ended answers, a judge model (LLM-as-judge, episode 25) can grade.
  • Evals tell you before you ship; traces tell you what happens after (in production).

Glossary

Term Meaning Example from this episode
Observability being able to see what a system did and why traces, spans, logs
Trace the path of one request through the app the 3-step list
Span one unit of work inside a trace tool: search_trains
Root span the top span, with no parent agent: trains to Goa
Attribute a key-value detail on a span tokens: 1400
OpenTelemetry an open standard for traces, metrics and logs
time.perf_counter() Python's precise stopwatch span timings
Latency how long something takes 2.0 s for the agent
Token a piece of text the model reads or writes (episode 12) 1400 tokens
Eval a test of an AI system's behaviour the suite
Non-deterministic the same input can give different outputs 6 of 8 runs
random.seed fixes the random numbers so a run repeats random.seed(42)
τ-bench (tau-bench) an agent benchmark with simulated users, tools and policies June 2024
pass^k chance that k tries in a row all pass 78% → 25%
math.comb(n, k) number of ways to choose k items from n comb(8, 4) = 70
Trajectory the sequence of steps an agent took the trace list
Eval suite a fixed set of test tasks with expected results SUITE
Regression something that used to work and now doesn't the refund in v2
LLM-as-judge a model that grades another model's answers episode 25

Common mistakes

Mistake What you see Why Fix
Printing only the final answer Sorry, no trains to Goa on 8 Nov. middle steps invisible trace every step
Guessing which step is slow optimised the wrong thing no timings spans with time and tokens
Shipping after one pass one run: PASS, then 6 of 8 models vary run to run run k times, use pass^k
Testing only the answer answer test: PASS, booked twice path not graded trajectory checks
Comparing only total scores 3/4 → 3/4, refund broke totals hide regressions compare task by task
Fixing a bug without a test the bug comes back later nothing guards it add it to the suite

Mini project: a tiny agent test bench

Write bench.py with: a trace list and span()/end() helpers; a toy agent with three tools; a suite of 6 tasks with expected answers and path rules (each booking once, at most 5 steps); run every task 8 times with different seeds, print pass^1 and pass^4 per task, and a table comparing two prompt versions task by task.

Practice

1. Add a "Dec" trip: search trains on "25 Dec" with the fixed agent (no trains are stored for that day) and check the trace.

Solution
TRAINS = {"08-11": [12127, 11007]}
MONTHS = {"Nov": "11", "Dec": "12"}
trace = []

def to_api_date(text):
    day, month = text.split()
    return f"{int(day):02d}-{MONTHS[month]}"

def search_trains(date):
    result = TRAINS.get(to_api_date(date), [])
    trace.append({"step": "search_trains", "input": date, "api": to_api_date(date), "output": result})
    return result

search_trains("25 Dec")
for span in trace:
    print(span)

Output:

{'step': 'search_trains', 'input': '25 Dec', 'api': '25-12', 'output': []}

2. Add a status field to each span: "ERROR" if the step raised an exception, else "OK".

Solution
trace = []

def traced(name, func, *args):
    span = {"name": name, "status": "OK"}
    try:
        span["output"] = func(*args)
    except Exception as e:
        span["status"] = "ERROR"
        span["output"] = f"{type(e).__name__}: {e}"
    trace.append(span)
    return span["output"]

traced("tool: fare", lambda no: {12127: 450}[no], 12127)
traced("tool: fare", lambda no: {12127: 450}[no], 99999)
for span in trace:
    print(f"{span['name']:12} {span['status']:6} {span['output']}")

Output:

tool: fare   OK     450
tool: fare   ERROR  KeyError: 99999

3. Compute pass^3 for the quiz agents A (PPPPPPPF) and B (PPPPPFFF).

Solution
from math import comb
for name, record in [("A", "PPPPPPPF"), ("B", "PPPPPFFF")]:
    n, c = len(record), record.count("P")
    print(name, f"pass^3 = {comb(c, 3) / comb(n, 3):.0%}")

Output:

A pass^3 = 62%
B pass^3 = 18%

4. Add the 08-11 bug as a suite task and print which tasks changed between versions.

Solution
v1 = {"trains on 8 Nov": "P", "cancel PNR 4521": "P", "refund PNR 4521": "P", "trains on 08-11": "F"}
v2 = {"trains on 8 Nov": "P", "cancel PNR 4521": "P", "refund PNR 4521": "F", "trains on 08-11": "P"}
for task in v1:
    if v1[task] != v2[task]:
        change = "FIXED" if v2[task] == "P" else "BROKE"
        print(f"{change}: {task}")

Output:

BROKE: refund PNR 4521
FIXED: trains on 08-11

Quick quiz 1

Eight runs of the same task: agent A PPPPPPPF, agent B PPPPPFFF. Which is more reliable?

Answer

A. It passed 7 of 8; pass^4 = C(7,4)/C(8,4) = 35/70 = 50%. Agent B: C(5,4)/C(8,4) = 5/70 ≈ 7%.

Quick quiz 2

A user says: "the agent said my refund failed, but I got the money twice!" What do you look at first?

Answer

B: the trace of that run. It shows both refund calls with their inputs and outputs, and the step that reported a failure (probably a timeout followed by a retry without an idempotency key, episode 35). Then add the case to your eval suite.

Interview questions

  1. What's the difference between a trace and a span? A trace is the whole path of one request; a span is one unit of work inside it, with a parent, timings, attributes and a status.
  2. Why are agents harder to test than normal functions? They're non-deterministic, take many steps, call tools with side effects, and can reach a right answer by a wrong path.
  3. What is pass^k? From τ-bench: the probability that k independent tries of a task all succeed; it measures reliability rather than a lucky single success.
  4. What is trajectory evaluation? Checking the steps an agent took (tools, order, counts, forbidden actions), not just its final answer.
  5. How do you catch regressions after changing a prompt? Run a fixed eval suite several times per task on both versions and compare task by task, not just total scores.
  6. What would you log for each LLM call? Model name, input, output, tokens, latency, cost, status and the parent span, without secrets or unnecessary personal data.
  7. Evals vs monitoring? Evals run before shipping on known tasks; monitoring (traces, alerts) watches real traffic after shipping, and its failures become new eval tasks.

Summary

  • Printing only the answer hides bugs: trace every step (input and output).
  • Spans (OpenTelemetry) form a tree with timings, attributes and status: find the slow and expensive steps.
  • Models vary: run each task k times; pass^k (τ-bench) measures reliability: 78% → 25% here.
  • Grade the path (trajectory), not just the answer.
  • Keep an eval suite; compare task by task on every change; every bug becomes a test.

Next: episode 37, Agent design patterns and frameworks.

Sources: OpenTelemetry documentation, "Traces" (opentelemetry.io/docs/concepts/signals/traces/); Yao, Shinn, Razavi, Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", arXiv 2406.12045 (17 June 2024). Fetched 5 Nov 2026.