SyntaxPilot
Courses/AI: Zero to Agents/Lesson 16

AI: Zero to Agents · Lesson 16 of 40 · 11 min

Controlling the output: temperature, top-p, context window

queued

These are the notes for AI episode 16. Last episode your language model got stuck saying "neha drinks chai" forever, because greedy decoding always takes the top token. Here you invent the knobs every AI app has: sampling (with a seed), temperature, the long tail problem, top-k and top-p (nucleus) sampling, the context window and truncation, why long contexts cost so much, and max_tokens. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

Real-life example: ordering at your favourite dhaba

At the dhaba In decoding
Always dal makhani, every day greedy decoding: always the top token, boring and loops
Close your eyes, point at the 120-item menu uniform random: chaos
Choose among dishes you like, favourites more often sampling by probability
Never even look at the weird stuff at the back top-k / top-p: a shortlist first

Choosing the next token from the model's probabilities is called decoding.

Attempt: pure random

import random
random.seed(7)
vocab = ["chai", "coffee", "water", "lassi", "cricket", "homework", "Tuesday", "blue"]

for _ in range(4):                    # pick ANY token, all equally likely
    print("neha drinks", random.choice(vocab))

Output:

neha drinks homework
neha drinks water
neha drinks Tuesday
neha drinks chai

Every token equally likely throws away everything the model learned: Tuesday gets the same chance as chai.

Sampling by probability

import random
from collections import Counter
random.seed(7)
vocab = ["chai", "coffee", "water", "lassi", "cricket", "homework", "Tuesday", "blue"]
probs = [0.45, 0.25, 0.12, 0.10, 0.03, 0.02, 0.02, 0.01]   # from the model

for _ in range(4):                    # pick by probability
    print("neha drinks", random.choices(vocab, weights=probs)[0])
picks = Counter(random.choices(vocab, weights=probs, k=1000))
print("1000 picks:", dict(picks))

Output:

neha drinks chai
neha drinks chai
neha drinks coffee
neha drinks chai
1000 picks: {'coffee': 227, 'chai': 486, 'lassi': 98, 'cricket': 31, 'Tuesday': 23, 'water': 100, 'blue': 9, 'homework': 26}

random.choices picks by the weights, like a lucky draw with more slips for chai: varied and mostly sensible. This is sampling. random.seed(7) makes the draws repeatable (same seed, same picks), which is why some APIs let you set a seed. But the junk tokens (cricket, homework, Tuesday, blue) still came up 89 times in 1,000.

Temperature

probabilities = softmax(logits / T)
import math
vocab = ["chai", "coffee", "water", "lassi", "cricket"]
logits = [3.0, 2.4, 1.6, 1.4, 0.2]   # raw scores from the model

def softmax(scores, temperature):
    e = [math.exp(s / temperature) for s in scores]   # divide, then softmax
    return [round(x / sum(e), 2) for x in e]

for t in [0.2, 1.0, 2.0]:
    print(f"temperature {t}:", dict(zip(vocab, softmax(logits, t))))

Output:

temperature 0.2: {'chai': 0.95, 'coffee': 0.05, 'water': 0.0, 'lassi': 0.0, 'cricket': 0.0}
temperature 1.0: {'chai': 0.49, 'coffee': 0.27, 'water': 0.12, 'lassi': 0.1, 'cricket': 0.03}
temperature 2.0: {'chai': 0.34, 'coffee': 0.25, 'water': 0.17, 'lassi': 0.15, 'cricket': 0.08}
Temperature Effect Use for
below 1 (e.g. 0.2) sharper: the top token wins more often code, maths, facts
1 the model's own probabilities default
above 1 (e.g. 2.0) flatter: rare tokens get more chances brainstorming, stories
0 (in APIs) treated as greedy: take the top token most repeatable output

Temperature doesn't make the model smarter or more creative; it only reshapes the probabilities before the draw.

The long tail

# a real vocabulary is huge: one good token, and a long tail of junk
good = 0.5
junk = [0.5 / 50_000] * 50_000        # 50,000 tokens, each almost never

print("best single junk token:", junk[0])
print("chance of SOME junk token:", round(sum(junk), 2))
print("junk in a 100-token reply:", round(sum(junk) * 100), "tokens")

Output:

best single junk token: 1e-05
chance of SOME junk token: 0.5
junk in a 100-token reply: 50 tokens

Each junk token is almost impossible, but 50,000 of them add up to 50%. This is the long tail. Temperature reshapes the tail but never removes it, so we cut it off.

Top-k sampling

def top_k(probs, k):                  # keep the k most likely tokens
    return dict(sorted(probs.items(), key=lambda t: -t[1])[:k])

sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}

print("the Taj Mahal is in ->", top_k(sure, 3))
print("my favourite drink is ->", top_k(open_, 3))

Output:

the Taj Mahal is in -> {'Agra': 0.94, 'Delhi': 0.02, 'Jaipur': 0.02}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2}

Top-k keeps the k most likely tokens and samples among them. A fixed k is too many when the model is sure (Delhi and Jaipur stay in for the Taj Mahal) and too few when many answers are fine (water and juice are cut).

Top-p (nucleus) sampling

def top_p(probs, p):                  # smallest set of tokens adding up to p
    kept, total = {}, 0.0
    for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
        kept[token] = prob
        total += prob
        if total >= p:
            break
    return kept

sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}

print("the Taj Mahal is in ->", top_p(sure, 0.9))
print("my favourite drink is ->", top_p(open_, 0.9))

Output:

the Taj Mahal is in -> {'Agra': 0.94}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2, 'water': 0.2, 'juice': 0.2}

Top-p sorts tokens from most to least likely and keeps adding until their probabilities reach p, then samples only from that shortlist (the nucleus). It shrinks when the model is sure and grows when many answers are fine. It was proposed by Holtzman, Buys, Du, Forbes and Choi in "The Curious Case of Neural Text Degeneration" (2019, arXiv 1904.09751), who found that decoding by likelihood alone gives text that is "bland and strangely repetitive".

Which settings when?

Task A reasonable start
Code, maths, facts low temperature (or 0)
Everyday chat temperature ≈ 0.7–1, top-p ≈ 0.9
Brainstorming, stories higher temperature
Tuning a common rule of thumb: adjust temperature or top-p, not both at once

The context window: a train window

On a train In an LLM
You only see one stretch of the view the context window: tokens the model can see
The station you left is out of sight truncation: old tokens drop out
Note what matters in a diary summaries and retrieval bring back what matters
chat = "my name is raju . i love chai . i study in pune . what is my name ?"
tokens = chat.split()
WINDOW = 12                           # the model sees only the last 12 tokens

seen = tokens[-WINDOW:]
print("all tokens:  ", len(tokens))
print("model sees:  ", " ".join(seen))
print("'raju' visible?", "raju" in seen)

Output:

all tokens:   19
model sees:   chai . i study in pune . what is my name ?
'raju' visible? False

With a 12-token window, "raju" fell out and the model can't answer. Real windows: GPT-2 small saw 1,024 tokens, GPT-3 2,048. Your prompt and the reply must both fit.

Why not an infinite window?

for window in [1024, 2048, 4096]:     # attention compares every token with every token
    print(f"{window:5} tokens -> {window * window:>10,} attention scores per head")

Output:

 1024 tokens ->  1,048,576 attention scores per head
 2048 tokens ->  4,194,304 attention scores per head
 4096 tokens -> 16,777,216 attention scores per head

Attention compares every token with every token, so doubling the window means four times the attention scores. Newer models use clever tricks to stretch their windows much further, but longer context always costs more time and money.

Living with the window

Technique Idea Where in this course
Drop the oldest messages simple truncation this episode
Summarise the old part keep a short summary episode 30
Retrieve only what's needed put the relevant paragraphs in the window RAG, episode 24
Save facts outside the model agent memory episode 30

max_tokens and stopping

reply = "Chai is made by boiling tea leaves with milk , sugar and ginger ."
tokens = reply.split()

def generate(max_tokens):
    out = []
    for t in tokens:                  # stand-in for the model's tokens
        if len(out) == max_tokens:
            return " ".join(out), "stopped: max_tokens"
        out.append(t)
        if t == ".":
            return " ".join(out), "stopped: end of answer"
    return " ".join(out), "done"

print(generate(6))
print(generate(50))

Output:

('Chai is made by boiling tea', 'stopped: max_tokens')
('Chai is made by boiling tea leaves with milk , sugar and ginger .', 'stopped: end of answer')

Generation stops when the model produces an end token (like GPT-2's <|endoftext|>) or when it reaches max_tokens. An answer cut off mid-sentence usually means max_tokens was too small.

Glossary

Term Meaning Example
Decoding choosing the next token from the probabilities greedy, sampling, top-p
Greedy decoding always take the most likely token loops "neha drinks chai"
Uniform random every token equally likely "neha drinks Tuesday"
Sampling pick at random, weighted by probability random.choices(vocab, weights=probs)
Random seed a starting number that makes random draws repeatable random.seed(7)
Logits raw scores before softmax [3.0, 2.4, 1.6, 1.4, 0.2]
Temperature divide the logits by T before softmax T = 0.2 → chai 0.95
Long tail many unlikely tokens whose chances add up 50,000 × 0.00001 = 0.5
Top-k sampling sample only from the k most likely tokens k = 3
Top-p sampling sample from the smallest set reaching probability p p = 0.9 → just "Agra"
Nucleus the top-p shortlist Holtzman et al., 2019
Context window the most tokens a model can see at once GPT-3: 2,048
Truncation dropping the oldest tokens to fit "raju" falls out
Quadratic cost work grows with the square of the length 2× tokens = 4× scores
max_tokens the most tokens the model may generate 6 → cut mid-sentence
Stop condition when generation ends end token or max_tokens
End token a special token meaning "done" <|endoftext|>

Common mistakes

Mistake Code Error
Temperature 0 in your own softmax softmax(logits, temperature=0) ZeroDivisionError: float division by zero
Tokens and weights of different lengths random.choices(["chai", "coffee"], weights=[0.5, 0.3, 0.2]) ValueError: The number of weights does not match the population
Filtering out every token random.choices(["chai"], weights=[0]) ValueError: Total of weights must be greater than zero
An empty shortlist random.choices([], weights=[]) IndexError: list index out of range
max_tokens too small generate(6) no error; answer cut mid-sentence
A long chat with a small window 19 tokens, window 12 no error; the model forgets your name

Fixes: treat temperature 0 as greedy; keep tokens and probabilities together; always keep at least the top token; raise max_tokens; summarise or retrieve instead of stuffing the window.

Mini project: your own sampler

Combine everything: write generate(follows, start, steps, temperature, p) for the counting model from episode 15 that turns counts into logits (math.log(count)), applies temperature, keeps the top-p shortlist and samples with random.choices. Compare temperature 0.3 and 1.5.

Practice

  1. Run the temperature demo with 0.5 and 5.0.
  2. Run top_p with p = 0.5 on both prompts.
  3. Change WINDOW so "raju" stays visible.
  4. Write sample() = top-p + random.choices.
Solutions

1.

import math
vocab = ["chai", "coffee", "water", "lassi", "cricket"]
logits = [3.0, 2.4, 1.6, 1.4, 0.2]   # raw scores from the model

def softmax(scores, temperature):
    e = [math.exp(s / temperature) for s in scores]   # divide, then softmax
    return [round(x / sum(e), 2) for x in e]

for t in [0.5, 5.0]:
    print(f"temperature {t}:", dict(zip(vocab, softmax(logits, t))))
temperature 0.5: {'chai': 0.71, 'coffee': 0.21, 'water': 0.04, 'lassi': 0.03, 'cricket': 0.0}
temperature 5.0: {'chai': 0.25, 'coffee': 0.23, 'water': 0.19, 'lassi': 0.18, 'cricket': 0.14}

2.

def top_p(probs, p):                  # smallest set of tokens adding up to p
    kept, total = {}, 0.0
    for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
        kept[token] = prob
        total += prob
        if total >= p:
            break
    return kept

sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}

print("the Taj Mahal is in ->", top_p(sure, 0.5))
print("my favourite drink is ->", top_p(open_, 0.5))
the Taj Mahal is in -> {'Agra': 0.94}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2}

With p = 0.5, the drinks shortlist stops at three (0.2 + 0.2 + 0.2 = 0.6 ≥ 0.5).

3.

chat = "my name is raju . i love chai . i study in pune . what is my name ?"
tokens = chat.split()
WINDOW = 16                           # big enough to keep 'raju'

seen = tokens[-WINDOW:]
print("all tokens:  ", len(tokens))
print("model sees:  ", " ".join(seen))
print("'raju' visible?", "raju" in seen)
all tokens:   19
model sees:   raju . i love chai . i study in pune . what is my name ?
'raju' visible? True

"raju" is the 4th of 19 tokens, so the window must be at least 16.

4.

import random
random.seed(1)

def top_p(probs, p):
    kept, total = {}, 0.0
    for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
        kept[token] = prob
        total += prob
        if total >= p:
            break
    return kept

def sample(probs, p=0.9):             # top-p, then pick by probability
    kept = top_p(probs, p)
    return random.choices(list(kept), weights=list(kept.values()))[0]

probs = {"chai": 0.45, "coffee": 0.25, "water": 0.12, "lassi": 0.10,
         "cricket": 0.03, "homework": 0.02, "Tuesday": 0.02, "blue": 0.01}
print("shortlist:", list(top_p(probs, 0.9)))
print([sample(probs) for _ in range(10)])
shortlist: ['chai', 'coffee', 'water', 'lassi']
['chai', 'water', 'water', 'chai', 'coffee', 'chai', 'coffee', 'water', 'chai', 'chai']

The junk tokens can never be picked: they are outside the 0.9 nucleus.

Quick quiz 1

You want Python code that must run. Best starting setting? A: temperature 2.0 · B: temperature 0.2 · C: top-k = 50,000 · D: max_tokens = 5

Answer

B: temperature 0.2. Code needs the most likely tokens.

Quick quiz 2

Probabilities A 0.6, B 0.3, C 0.06, D 0.04. Top-p with p = 0.9 keeps? A: A only · B: A and B · C: A, B and C · D: all four

Answer

B: A and B. 0.6 < 0.9; 0.6 + 0.3 = 0.9 reaches p, so stop.

Interview questions

  1. What does temperature do mathematically? Divides the logits by T before softmax: T < 1 sharpens the distribution, T > 1 flattens it.
  2. Top-k vs top-p? Top-k keeps a fixed number of tokens; top-p keeps the smallest set whose probabilities reach p, so it adapts to the model's confidence.
  3. Why does greedy decoding repeat itself? It deterministically follows the highest-probability path, which often cycles; sampling breaks the cycle.
  4. Why is the long tail dangerous? Thousands of tiny probabilities add up to a large chance of picking a bad token.
  5. What is a context window, and what happens when it overflows? The maximum tokens the model can attend to; older tokens are truncated, so the model "forgets".
  6. Why are long contexts expensive? Self-attention compares every pair of tokens: cost grows with the square of the length.
  7. Why might an answer end mid-sentence? It hit max_tokens before the model produced its end token.

Summary

  • Greedy loops; uniform random is nonsense; sampling by probability is varied and sensible (a seed makes it repeatable).
  • Temperature divides the logits: low = safe, high = wild.
  • The long tail of tiny chances adds up, so cut it: top-k (fixed size) or top-p (adapts).
  • The context window limits what the model sees; old tokens are truncated; cost grows with the square of the length.
  • max_tokens and the end token decide where an answer stops.
  • Next episode: from raw model to assistant: fine-tuning and RLHF.