SyntaxPilot
Courses/AI: Zero to Agents/Lesson 21

AI: Zero to Agents · Lesson 21 of 40 · 11 min

Better prompts

queued

These are the notes for AI episode 21. Episode 20 taught you to ask clearly. Now: questions with several steps, where a clear prompt can still get a wrong answer. You'll learn chain-of-thought prompting ("show your working"), zero-shot chain of thought ("Let's think step by step"), self-consistency (ask several times and vote), prompt chaining, testing prompts on a test set (prompt evaluation), and validate-and-retry for structured replies. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The "models" are small toys that show the idea.

Real-life example: "Show your working!"

In the maths test With an LLM
Bill: 3 chai at ₹15, 2 samosas at ₹20, paid ₹100. Change? a multi-step question
Ravi writes only "30": it looked like last week's sum answering in one jump = one hard guess
Priya writes 45, 40, 85, then 15 writing the steps = chain of thought
Any slip in Priya's steps is easy to spot steps can be checked

One jump vs step by step

A toy model that knows answers it has seen many times. For a new question it copies the answer of the most similar-looking one.

# a toy "model": it knows answers it has seen many times
seen = {"3 * 15": 45, "2 * 20": 40, "45 + 40": 85, "100 - 85": 15,
        "100 - (3 * 10 + 2 * 20)": 30}   # a similar bill it once saw

def model(question):
    if question in seen:
        return seen[question]
    best = max(seen, key=lambda q: sum(a == b for a, b in zip(q, question)))
    return seen[best]                    # never seen: copy a similar one

bill = "100 - (3 * 15 + 2 * 20)"   # 3 chai at 15, 2 samosa at 20, paid 100
print("in one jump :", model(bill))
a, b = model("3 * 15"), model("2 * 20")
total = model(f"{a} + {b}")
change = model(f"100 - {total}")
print("step by step:", a, "+", b, "=", total, "-> change", change)
print("real answer :", eval(bill))

Output:

in one jump : 30
step by step: 45 + 40 = 85 -> change 15
real answer : 15

The whole bill is a rare question, so the model guesses. Each small step is a common pattern it has seen many times. Steps turn one hard guess into several easy ones.

Chain-of-thought prompting (real research)

  • Wei and others (Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou), "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", arXiv 2201.11903, first submitted 28 January 2022, NeurIPS 2022. Few-shot examples that show intermediate reasoning steps. With just eight such examples, a 540-billion-parameter model reached state-of-the-art accuracy on GSM8K (school maths word problems), beating a fine-tuned GPT-3 with a verifier.
  • Kojima and others, "Large Language Models are Zero-Shot Reasoners", arXiv 2205.11916, 24 May 2022. Adding "Let's think step by step" with no examples raised accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7% (InstructGPT, text-davinci-002).

In a prompt

Work it out step by step, then give the final answer
on the last line as: ANSWER: <number>

Put the final answer in a fixed place so your code can find it among the steps.

Finding the answer in a fixed place

replies = [
    ("3 chai at 15 is 45. 2 samosas at 20 is 40.\n"
     "45 + 40 = 85, and 100 - 85 = 15.\nANSWER: 15"),
    "The total is 85, so the change is 15 rupees.",
]
def extract(reply):                  # find the answer in its fixed place
    for line in reply.splitlines():
        if line.startswith("ANSWER:"):
            return int(line.split(":")[1])
    return None                      # no marker: don't guess

for r in replies:
    print("extracted:", extract(r))

Output:

extracted: 15
extracted: None

The second reply also says 15, but code can't safely pick the right number (85? 15? 100?) out of chatty text. Let the model think freely, but put the answer in a fixed place.

Real-life example: asking five people the way

In a new city With an LLM
Ask one person the way: they might be wrong one answer = one reasoning path
Ask five people: 4 point left, 1 points right several samples
Go where most point: wrong guesses rarely agree majority vote

Self-consistency: ask several times, vote

import random
from collections import Counter
random.seed(1)

def solve(answer):          # one reasoning path: right 60% of the time
    if random.random() < 0.6:
        return answer
    return answer + random.choice([-10, -5, 5, 10, 20])   # a slip

one = vote = 0
for _ in range(1000):                       # 1,000 questions
    true = random.randint(10, 99)
    one += solve(true) == true
    answers = Counter(solve(true) for _ in range(5))   # ask 5 times
    vote += answers.most_common(1)[0][0] == true
print("one answer :", one / 10, "% correct")
print("vote of 5  :", vote / 10, "% correct")

Output:

one answer : 61.7 % correct
vote of 5  : 85.7 % correct

Right paths agree on one answer; slips scatter over many wrong answers, so the majority is usually right. Wang and others, "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (arXiv 2203.11171, 21 March 2022): sample several reasoning paths and keep the most consistent answer; +17.9% on GSM8K. The cost: several calls instead of one.

Prompt chaining

[1. Extract items + prices] → [2. Calculate with code] → [3. Write a polite reply]
  • Each prompt does one job, and each result is easy to check.
  • The output of one step is the input of the next.
  • Exact maths? Let code do that step (episode 19). This is the first step towards agents.

Two more prompt tricks

  1. Ask it to check: "Re-check each step. If one is wrong, fix it." A second call can also review the first answer.
  2. Give it an out: "If you can't tell, reply UNKNOWN." Less bluffing (episode 19), and your code can spot UNKNOWN and ask a human.

A model checking itself can still miss its own mistakes, so test whichever trick you use.

Test your prompts

One good-looking answer proves nothing. Score each prompt version on the same test set.

def model(examples, review):     # copies the label of the closest example
    words = set(review.split())
    return max(examples, key=lambda e: len(set(e[0].split()) & words))[1]
P, N = "positive", "negative"
tests = [("loved the masala chai", P), ("samosa was cold and stale", N),
         ("the tea was not bad", P), ("not good at all", N),
         ("tea was too sweet", N), ("great taste will come again", P)]
v1 = [("loved the hot chai", P), ("the tea was cold", N)]
v2 = v1 + [("the food was not bad", P),
           ("the samosa was not good at all", N)]   # tricky cases
def score(name, examples):
    passed = 0
    for review, want in tests:
        got = model(examples, review)
        passed += got == want
        if got != want: print("  FAIL:", review, "->", got)
    print(name, "passed", passed, "/", len(tests))
score("v1", v1); score("v2", v2)

Output:

  FAIL: the tea was not bad -> negative
  FAIL: not good at all -> positive
v1 passed 4 / 6
v2 passed 6 / 6

Warning (episode 6): if you tune only on these cases, you can overfit to them. Keep some test cases you never tune on (a held-out set).

Validate and retry

import json
replies = iter(['Sure! {"sentiment": "positive"}', '{"sentiment": "positive"}',
                '{"sentiment": "positive", "item": "chai"}'])
def ask(prompt): return next(replies)      # stand-in for a real model call

def get_label(prompt, tries=3):
    for t in range(1, tries + 1):
        reply = ask(prompt)
        try:
            data = json.loads(reply)
            if {"sentiment", "item"} <= data.keys():
                return data                          # valid: done
            problem = "missing key 'item'"
        except json.JSONDecodeError as e:
            problem = f"not JSON ({e})"
        print(f"try {t} rejected:", problem)
        prompt += f"\nYour last reply was rejected: {problem}. JSON only."
    raise ValueError("no valid reply")
print("accepted:", get_label("Label the review: the chai was great"))

Output:

try 1 rejected: not JSON (Expecting value: line 1 column 1 (char 0))
try 2 rejected: missing key 'item'
accepted: {'sentiment': 'positive', 'item': 'chai'}

Never trust a reply blindly: validate it, retry with the exact reason, cap the tries, and raise an error rather than use bad data.

Every trick has a price

Trick Cost
Chain of thought more output tokens, slower reply
Voting of 5 five calls, five times the cost
Prompt chaining more calls, but each is small and some steps can use plain code or a cheaper model
Simple task a plain clear prompt is enough

Use the cheapest trick that passes your tests.

Glossary

Term Meaning Example from this episode
Chain of thought the reasoning steps, written out 45, 40, 85, 15
Chain-of-thought prompting asking for (or showing examples of) the steps "Work it out step by step"
Zero-shot chain of thought asking for steps without any examples "Let's think step by step"
GSM8K a benchmark of school maths word problems used by Wei, Kojima, Wang
Benchmark a fixed test used to compare models MultiArith, GSM8K
Reasoning path one chain of steps from question to answer one call to solve()
Self-consistency several reasoning paths, keep the majority answer vote of 5: 85.7%
Majority vote the most common answer wins Counter(...).most_common(1)
Prompt chaining splitting a job into several prompts in sequence extract → calculate → write
Test set inputs with known right answers 6 labelled reviews
Prompt evaluation scoring prompt versions on a test set v1 4/6, v2 6/6
Held-out cases test cases you never tune on to catch overfitting
Validation checking a reply's format before using it JSON with both keys
Retry asking again, with the reason the last reply failed try 1, 2, 3

Common mistakes

Code Exact error Why Fix
next(replies) after all replies were used StopIteration asked again after the last reply limit the tries; stop with a clear error
int(reply.split("ANSWER: ")[1]) when there is no ANSWER line IndexError: list index out of range the final answer wasn't in the fixed place check "ANSWER: " in reply before splitting
Counter([]).most_common(1)[0] IndexError: list index out of range voting over zero answers make sure at least one answer came back

Mini project: a safe answer extractor

Write extract(reply) that finds the line starting with ANSWER: in a chain-of-thought reply and returns the number as an int, or None if it's missing or not a number. Then write vote(replies) that extracts every answer, ignores the Nones and returns the majority answer, or None if nothing valid came back. Test it on five sample replies, including one with no answer line.

Practice

  1. Change solve() to be right 50% of the time. Does voting still help?
  2. Try a vote of 9 instead of 5.
  3. Add ("the samosa was great", "positive") to the tests. Does v2 still pass everything?
  4. Make every reply in replies invalid and read the error.
Solutions

1. Change 0.6 to 0.5:

one answer : 49.0 % correct
vote of 5  : 73.4 % correct

Still a big gain: the right answer is the single most common one even when it's right only half the time, because the slips scatter.

2. Change range(5) to range(9) (with 0.6):

one answer : 60.8 % correct
vote of 9  : 95.0 % correct

(Edit the print label too.) More votes, higher accuracy, more calls.

3. Add the case and run:

  FAIL: the tea was not bad -> negative
  FAIL: not good at all -> positive
  FAIL: the samosa was great -> negative
v1 passed 4 / 7
  FAIL: the samosa was great -> negative
v2 passed 6 / 7

v2 was 6/6 on the cases it was tuned on, but a new case breaks it: "the samosa was great" shares more words with "the samosa was not good at all" than with any positive example. That's exactly why you keep held-out cases. Now try a fix: add ("the samosa was great and hot", "positive") to v2. The new case passes, but samosa was cold and stale now fails (6 / 7 again). Every prompt change can break something else, so re-run the whole test set after each change.

4. With three invalid replies:

try 1 rejected: not JSON (Expecting value: line 1 column 1 (char 0))
try 2 rejected: not JSON (Expecting value: line 1 column 1 (char 0))
try 3 rejected: not JSON (Expecting value: line 1 column 1 (char 0))
Traceback (most recent call last):
  ...
ValueError: no valid reply

Quick quiz 1

Five reasoning paths gave 42, 42, 37, 42, 52. Why does voting help? A: the model gets smarter · B: right paths agree, slips scatter · C: it uses less compute · D: temperature becomes 0

Answer

B. Correct paths land on the same answer; mistakes land on different wrong answers. (C is wrong: voting costs more compute.)

Quick quiz 2

You changed a prompt and one answer looks better. How do you know v2 is better? A: it sounds more confident · B: it's longer · C: score both on the same test set · D: ask the model

Answer

C. Run both on the same test cases and compare scores. Keep some cases you never tune on.

Interview questions

  1. What is chain-of-thought prompting and why does it help? Asking the model to write intermediate steps; each step is an easier, more familiar prediction, and the final answer conditions on the worked steps.
  2. What is zero-shot chain of thought? Triggering step-by-step reasoning with an instruction like "Let's think step by step" instead of worked examples (Kojima et al., 2022).
  3. Explain self-consistency. Sample several reasoning paths (temperature above 0) and take the majority final answer (Wang et al., 2022). It trades extra calls for accuracy.
  4. What is prompt chaining, and when would you use it? Splitting a task into sequential prompts with checkable intermediate outputs; use it for multi-stage jobs and to insert code or tools between steps.
  5. How do you evaluate a prompt change? Run old and new prompts on the same labelled test set, compare scores, inspect failures, and keep a held-out set to avoid overfitting.
  6. How do you make LLM output reliable for code? Specify the format, validate every reply (parse plus schema check), retry with the specific error, cap retries, and fail loudly.

Summary

  • Answering in one jump is one hard guess; steps are easy patterns: chain of thought.
  • "Let's think step by step" works even with no examples.
  • Self-consistency: several paths, majority vote; more accurate, more calls.
  • Prompt chaining: one job per prompt, code for exact steps.
  • Test prompts on a test set; keep held-out cases.
  • Validate and retry every structured reply.
  • Next episode: reasoning models: AI trained to think before it answers.