SyntaxPilot
Courses/AI: Zero to Agents/Lesson 25

AI: Zero to Agents · Lesson 25 of 40 · 11 min

Evaluating AI apps

queued

These are the notes for AI episode 25. You changed prompts, thresholds and chunk sizes in the last episodes. How do you know a change made your AI app better and didn't quietly break something? You run an eval: many test questions with golden answers, a grader that scores each reply, and a score you can compare between versions. You'll see why exact match marks correct answers wrong, how a key-fact check catches a regression, how to write success criteria, how LLM-as-a-judge works with a rubric, the judge's biases, and how to check a judge against people. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The bots and the judge are toys.

Real-life example: Ramesh changes his recipe

What happened In AI
One customer says "more ginger!", Ramesh changes the recipe you change a prompt or a model
He tastes one cup: perfect, ships it you try one question and ship
Next day the regulars complain users hit the cases you didn't test
Next time 20 regulars taste it first an eval set: many cases, run before release

Try it once

def make_bot(r, fb):                  # toy bot: first matching rule wins
    return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
               "deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
              "We're open 7 am to 10 pm, come anytime!")   # "be helpful"

print("v1:", v1("How much is a samosa?"))
print("v2:", v2("How much is a samosa?"))      # try it once... looks great!

Output:

v1: Sorry, I don't know.
v2: Samosa is 20 rupees!

Version 2 looks better, so it ships. The next morning:

def make_bot(r, fb):                  # toy bot: first matching rule wins
    return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
               "deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
              "We're open 7 am to 10 pm, come anytime!")   # "be helpful"

for q in ["Do you deliver?", "Do you sell pizza?", "Is there parking?"]:
    print(f"{q:20} -> v2: {v2(q)}")

Output:

Do you deliver?      -> v2: We're open 7 am to 10 pm, come anytime!
Do you sell pizza?   -> v2: We're open 7 am to 10 pm, come anytime!
Is there parking?    -> v2: We're open 7 am to 10 pm, come anytime!

Why we missed it: one good example proves almost nothing. AI answers change from question to question, so you test many questions every time something changes.

Real-life example: a school exam

A school exam An AI eval
The question paper covers the syllabus, plus tricky questions eval set (test cases), including edge cases
The answer key golden answers (what a correct reply must contain)
The examiner marks every paper the same way grader (code or a model)
The marks: 72/100 score: compare versions by number

Grader 1: exact match

def make_bot(r, fb):                  # toy bot: first matching rule wins
    return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
               "deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
              "We're open 7 am to 10 pm, come anytime!")   # "be helpful"

TESTS = [("How much is chai?", "Chai costs Rs 15."),       # golden answers
         ("When do you close?", "We close at 10 pm."),
         ("Do you deliver?", "No, we don't deliver.")]
def exact_match(output, gold):            # from Anthropic's eval guide
    return output.strip().lower() == gold.lower()
for q, gold in TESTS:
    out = v1(q)
    print(exact_match(out, gold), "|", out)

Output:

False | Chai is 15 rupees.
True | We close at 10 pm.
False | Sorry, we don't deliver.

exact_match is the grader from Anthropic's evaluation guide (it normalises whitespace and case). "Chai is 15 rupees." and "Chai costs Rs 15." state the same fact but don't match. Exact match is right when there is one correct form (a category label, a number, yes/no), not for free-form sentences.

Grader 2: check the key fact

def make_bot(r, fb):                  # toy bot: first matching rule wins
    return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
               "deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
              "We're open 7 am to 10 pm, come anytime!")   # "be helpful"

TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
         ("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
         ("Do you sell pizza?", "don't know")]      # edge case: not on menu
print(f"{'question':24}{'v1':6}v2")
for q, fact in TESTS:
    marks = ["pass" if fact in b(q).lower() else "FAIL" for b in (v1, v2)]
    print(f"{q:24}{marks[0]:6}{marks[1]}")
for name, bot in [("v1", v1), ("v2", v2)]:
    score = sum(fact in bot(q).lower() for q, fact in TESTS)
    print(f"{name} score: {score}/{len(TESTS)}")

Output:

question                v1    v2
How much is chai?       pass  pass
How much is a samosa?   FAIL  pass
When do you close?      pass  pass
Do you deliver?         pass  FAIL
Do you sell pizza?      pass  FAIL
v1 score: 4/5
v2 score: 3/5
  • Each test stores the key fact a correct answer must contain.
  • "Do you sell pizza?" is an edge case on purpose: the right answer is "I don't know".
  • v2 fixed the samosa question but broke delivery and pizza: v1 4/5, v2 3/5. A change that breaks something that used to work is a regression.

Fix and re-run

def make_bot(r, fb):                  # toy bot: first matching rule wins
    return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
               "deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
              "We're open 7 am to 10 pm, come anytime!")   # "be helpful"
v3 = make_bot({"samosa": "Samosa is 20 rupees.", "chai": "Chai is 15 rupees.",
    "close": "We close at 10 pm.", "deliver": "Sorry, we don't deliver."},
    "Sorry, I don't know.")               # v1's honesty + v2's samosa answer

TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
         ("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
         ("Do you sell pizza?", "don't know")]
for name, bot in [("v1", v1), ("v2", v2), ("v3", v3)]:
    score = sum(fact in bot(q).lower() for q, fact in TESTS)
    print(f"{name}: {score}/{len(TESTS)}", "ship it" if score == 5 else "")

Output:

v1: 4/5 
v2: 3/5 
v3: 5/5 ship it

Re-run the whole eval after every change (prompt, model, chunk size, threshold), like a programmer's test suite.

Success criteria

Weak Strong
"The bot should be good" "Answers 95% of menu questions correctly"
"Good performance" "Accurate sentiment classification" (Anthropic's example)

From Anthropic's guide Define success criteria and build evaluations:

  • Criteria should be Specific, Measurable, Achievable and Relevant.
  • Most apps need several metrics at once, e.g. accuracy, latency (how long users wait) and cost.
  • Be task-specific: mirror your real questions, and "Don't forget to factor in edge cases!"
  • Automate when possible: multiple-choice, string match, code-graded, LLM-graded.
  • Prioritise volume over quality: many automatically graded tests beat a few hand-graded ones.

LLM-as-a-judge

Some qualities have no single key fact: politeness, clarity, whether a summary is faithful. For those, another model grades the answer: LLM-as-a-judge. Judges have biases:

def judge(first, second):   # toy LLM judge: longer wins, a tie goes first
    return "first" if len(first) >= len(second) else "second"

good = "Sorry, we don't deliver. Pick up only."
ramble = ("Thank you so much for asking! We truly value every customer, and "
          "our team works very hard every single day. Do visit us soon!")
print("good vs ramble :", judge(good, ramble))     # ramble never answers!

a = "We close at 10 pm."
b = "We close by 10 pm."
print("a vs b         :", judge(a, b))
print("b vs a         :", judge(b, a))            # swap the order

Output:

good vs ramble : second
a vs b         : first
b vs a         : first
  • Verbosity bias: the judge prefers longer answers, even one that never answers.
  • Position bias: it prefers whichever answer it saw first (a vs b and b vs a both give "first").
  • Fix for position bias: judge both orders and trust the verdict only when they agree.

A rubric

RUBRIC = {                               # specific, checkable criteria
    "answers": lambda q, a: "deliver" in a.lower(),
    "correct": lambda q, a: "don't" in a.lower(),
    "polite": lambda q, a: any(w in a.lower() for w in ["sorry", "please"]),
    "short": lambda q, a: len(a.split()) < 20,
}
def grade(q, answer):
    failed = [name for name, check in RUBRIC.items() if not check(q, answer)]
    return f"{len(RUBRIC) - len(failed)}/{len(RUBRIC)}  failed: {failed}"

q = "Do you deliver?"
print("good  :", grade(q, "Sorry, we don't deliver. Pick up only."))
print("ramble:", grade(q, "Thank you so much for asking! We truly value "
                          "every customer. Do visit us soon!"))

Output:

good  : 4/4  failed: []
ramble: 1/4  failed: ['answers', 'correct', 'polite']

A rubric is a list of specific checks. Instead of "is this good?", the judge answers "does it answer the question? is the fact correct? is it polite? is it short?", and you learn what failed. Anthropic's guide: "Generally best practice to evaluate with a different model from the one used to generate the evaluated output."

Judge the judge

human = [1, 0, 1, 1, 0, 1, 0, 1, 1, 0]    # 1 = good answer, said by people
judge = [1, 0, 1, 0, 0, 1, 1, 1, 1, 0]    # the LLM judge, same answers

same = sum(h == j for h, j in zip(human, judge))
print(f"judge agrees with people: {same}/{len(human)} = {same/len(human):.0%}")
for i, (h, j) in enumerate(zip(human, judge)):
    if h != j:
        print(f"  answer {i}: human said {h}, judge said {j}  <- read these")

Output:

judge agrees with people: 8/10 = 80%
  answer 3: human said 1, judge said 0  <- read these
  answer 6: human said 0, judge said 1  <- read these

Check the judge against labels from people and read the disagreements to improve the rubric.

Zheng and colleagues, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv, 9 June 2023): strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level at which humans agree with each other. The paper also studied position, verbosity and self-enhancement bias (favouring answers like its own) and limited reasoning ability.

Choosing a grader

Code-based LLM judge Human
exact match for labels, key facts, valid JSON tone, clarity, faithfulness, with a rubric the most flexible
thousands of tests in seconds, same result every time fast, but has biases: check it against people slow and expensive
use it whenever you can for what code can't check for small, important samples

Offline and online evaluation

Offline Online
your eval set, before release real users, after release
catches regressions before users see them user feedback (thumbs up/down), A/B tests

An A/B test sends half the users to one version and half to the other, and compares the results. Every failure a real user finds becomes a new test case.

Glossary

Term Meaning Example from this episode
Eval (evaluation) measuring an AI app on many test cases v1 4/5, v2 3/5
Eval set / test cases the questions you test with 5 menu questions
Golden answer the known correct answer for a test "Chai costs Rs 15."
Key fact what a correct answer must contain "15"
Edge case an unusual or tricky input "Do you sell pizza?"
Grader code or a model that scores a reply exact_match, the rubric
Exact match output equals the golden answer (after normalising) False for a correct answer
Normalising making text comparable (spaces, case) .strip().lower()
Score / accuracy fraction of tests passed 4/5 = 80%
Regression a change breaks something that worked v2 broke delivery
Success criteria specific, measurable goals "95% of menu questions"
Latency how long a user waits for an answer seconds per reply
LLM-as-a-judge a model grading another model's answers the toy judge
Rubric a list of specific grading checks answers, correct, polite, short
Verbosity bias a judge preferring longer answers the ramble won
Position bias a judge preferring the first answer shown "first" both times
Self-enhancement bias a judge favouring answers like its own from Zheng et al. 2023
Agreement how often the judge matches people 8/10 = 80%
Offline eval testing before release on your eval set the 5 tests
Online eval measuring with real users after release thumbs up/down
A/B test two versions to two groups, compare results half v1, half v3

Common mistakes

Code / situation Exact error or symptom Why Fix
tested 1 question, shipped v2 broke delivery and pizza answers one example proves almost nothing an eval set with edge cases, every change
exact_match("Chai is 15 rupees.", "Chai costs Rs 15.") False (a correct answer marked wrong) same fact, different words check the key fact, or use a rubric
score / len(TESTS) with TESTS = [] ZeroDivisionError: division by zero an empty eval set never report a score from zero tests
judging in one order only the first answer always wins position bias judge both orders
"Is this answer good?" as the judge prompt vague, biased scores no specific criteria a rubric of specific checks

Mini project: an eval harness

Write evaluate(bot, tests) that returns the score and the list of failed questions, and compare(old, new, tests) that prints every test where the new version got worse ("REGRESSION: Do you deliver?"). Add a check that refuses to release (prints "BLOCK") if the new score is lower than the old one or any regression appears. Run it on v1 → v2 and v2 → v3.

Practice

  1. Add three tests: parking, Sunday hours, card payment. What do v1-v3 score now?
  2. Write v4 that passes them all.
  3. Make the judge fair about position: judge both orders, trust only if they agree.
  4. Write three rubric checks for a polite refusal ("Sorry, we don't deliver.").
Solutions

1 and 2. Add the tests and a v4:

v4 = make_bot({"samosa": "Samosa is 20 rupees.", "chai": "Chai is 15 rupees.",
    "close": "We close at 10 pm.", "deliver": "Sorry, we don't deliver.",
    "park": "Free parking behind the shop.", "sunday": "Yes, open on Sunday too.",
    "card": "Yes, card and UPI are fine."}, "Sorry, I don't know.")

TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
         ("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
         ("Do you sell pizza?", "don't know"), ("Is there parking?", "parking"),
         ("Are you open on Sunday?", "sunday"), ("Can I pay by card?", "card")]
v1: 4/8 
v2: 3/8 
v3: 5/8 
v4: 8/8 ship it

(The parking, Sunday and card answers are made up for Ramesh's toy shop.)

3. Add to the judge example:

def fair_judge(x, y):                 # judge both orders, trust only if they agree
    one, two = judge(x, y), judge(y, x)
    if one == "first" and two == "second": return "x"
    if one == "second" and two == "first": return "y"
    return "tie / unsure"
print("fair a vs b    :", fair_judge(a, b))
print("fair good vs ramble:", fair_judge(good, ramble))
fair a vs b    : tie / unsure
fair good vs ramble: y

Swapping fixes position bias (the two equal answers are now a tie), but not verbosity bias: the ramble still wins. That needs a rubric.

4. For example: says it doesn't deliver ("don't deliver" in a.lower()), apologises ("sorry" in a.lower()), offers an alternative ("pick up" in a.lower()).

Quick quiz 1

Version A scores 9/10 on your eval. Version B scores 7/10, but its demo answer looked amazing. Which ships? A: version B · B: version A · C: neither, evals are useless · D: whichever is newer

Answer

B (version A). A great demo is one example; the eval covers many questions, including edge cases.

Quick quiz 2

judge(x, y) returns "first" and judge(y, x) also returns "first". Which bias? A: verbosity bias · B: position bias · C: a regression · D: exact match

Answer

B. It prefers whatever it sees first. Judge both orders.

Interview questions

  1. How would you evaluate an LLM app before launch? Define specific, measurable success criteria; build an eval set that mirrors real usage with edge cases; grade automatically where possible (exact match, key facts, format checks), with an LLM judge and rubric for subjective qualities; track accuracy, latency and cost; run it on every change.
  2. Why is exact match a poor grader for chat answers? Correct answers can be phrased many ways; use it for labels or fixed formats, and key-fact checks or rubrics for free text.
  3. What is a regression in an AI app and how do you catch it? A change that makes previously correct cases fail; catch it by re-running the full eval and comparing per-test results between versions.
  4. What are the risks of LLM-as-a-judge? Position, verbosity and self-enhancement biases, inconsistent scoring; mitigate with rubrics, order swapping, a different model than the generator, and checking agreement with human labels.
  5. Offline vs online evaluation? Offline: fixed test set before release. Online: real users after release (feedback, A/B tests, monitoring); failures found online become new offline tests.
  6. How do you evaluate a RAG system? Separately measure retrieval (is the right chunk in the top-k?) and generation (is the answer correct and grounded in the retrieved text?), plus "I don't know" behaviour on out-of-scope questions.

Summary

  • One good example proves nothing: test many questions, every change.
  • An eval = an eval set (with edge cases) + golden answers + a grader → a score.
  • Exact match only for one correct form; key-fact checks and rubrics for sentences.
  • Comparing versions per test catches regressions.
  • Write specific, measurable success criteria, and track accuracy, latency and cost.
  • LLM-as-a-judge with a rubric, both orders, a different model, checked against people.
  • Next episode: what is an AI agent?