These are the notes for AI episode 16. Last episode your language model got stuck saying "neha drinks chai" forever, because greedy decoding always takes the top token. Here you invent the knobs every AI app has: sampling (with a seed), temperature, the long tail problem, top-k and top-p (nucleus) sampling, the context window and truncation, why long contexts cost so much, and max_tokens. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).
Real-life example: ordering at your favourite dhaba
| At the dhaba | In decoding |
|---|---|
| Always dal makhani, every day | greedy decoding: always the top token, boring and loops |
| Close your eyes, point at the 120-item menu | uniform random: chaos |
| Choose among dishes you like, favourites more often | sampling by probability |
| Never even look at the weird stuff at the back | top-k / top-p: a shortlist first |
Choosing the next token from the model's probabilities is called decoding.
Attempt: pure random
import random
random.seed(7)
vocab = ["chai", "coffee", "water", "lassi", "cricket", "homework", "Tuesday", "blue"]
for _ in range(4): # pick ANY token, all equally likely
print("neha drinks", random.choice(vocab))
Output:
neha drinks homework
neha drinks water
neha drinks Tuesday
neha drinks chai
Every token equally likely throws away everything the model learned: Tuesday gets the same chance as chai.
Sampling by probability
import random
from collections import Counter
random.seed(7)
vocab = ["chai", "coffee", "water", "lassi", "cricket", "homework", "Tuesday", "blue"]
probs = [0.45, 0.25, 0.12, 0.10, 0.03, 0.02, 0.02, 0.01] # from the model
for _ in range(4): # pick by probability
print("neha drinks", random.choices(vocab, weights=probs)[0])
picks = Counter(random.choices(vocab, weights=probs, k=1000))
print("1000 picks:", dict(picks))
Output:
neha drinks chai
neha drinks chai
neha drinks coffee
neha drinks chai
1000 picks: {'coffee': 227, 'chai': 486, 'lassi': 98, 'cricket': 31, 'Tuesday': 23, 'water': 100, 'blue': 9, 'homework': 26}
random.choices picks by the weights, like a lucky draw with more slips for chai: varied and mostly sensible. This is sampling. random.seed(7) makes the draws repeatable (same seed, same picks), which is why some APIs let you set a seed. But the junk tokens (cricket, homework, Tuesday, blue) still came up 89 times in 1,000.
Temperature
probabilities = softmax(logits / T)
import math
vocab = ["chai", "coffee", "water", "lassi", "cricket"]
logits = [3.0, 2.4, 1.6, 1.4, 0.2] # raw scores from the model
def softmax(scores, temperature):
e = [math.exp(s / temperature) for s in scores] # divide, then softmax
return [round(x / sum(e), 2) for x in e]
for t in [0.2, 1.0, 2.0]:
print(f"temperature {t}:", dict(zip(vocab, softmax(logits, t))))
Output:
temperature 0.2: {'chai': 0.95, 'coffee': 0.05, 'water': 0.0, 'lassi': 0.0, 'cricket': 0.0}
temperature 1.0: {'chai': 0.49, 'coffee': 0.27, 'water': 0.12, 'lassi': 0.1, 'cricket': 0.03}
temperature 2.0: {'chai': 0.34, 'coffee': 0.25, 'water': 0.17, 'lassi': 0.15, 'cricket': 0.08}
| Temperature | Effect | Use for |
|---|---|---|
| below 1 (e.g. 0.2) | sharper: the top token wins more often | code, maths, facts |
| 1 | the model's own probabilities | default |
| above 1 (e.g. 2.0) | flatter: rare tokens get more chances | brainstorming, stories |
| 0 (in APIs) | treated as greedy: take the top token | most repeatable output |
Temperature doesn't make the model smarter or more creative; it only reshapes the probabilities before the draw.
The long tail
# a real vocabulary is huge: one good token, and a long tail of junk
good = 0.5
junk = [0.5 / 50_000] * 50_000 # 50,000 tokens, each almost never
print("best single junk token:", junk[0])
print("chance of SOME junk token:", round(sum(junk), 2))
print("junk in a 100-token reply:", round(sum(junk) * 100), "tokens")
Output:
best single junk token: 1e-05
chance of SOME junk token: 0.5
junk in a 100-token reply: 50 tokens
Each junk token is almost impossible, but 50,000 of them add up to 50%. This is the long tail. Temperature reshapes the tail but never removes it, so we cut it off.
Top-k sampling
def top_k(probs, k): # keep the k most likely tokens
return dict(sorted(probs.items(), key=lambda t: -t[1])[:k])
sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}
print("the Taj Mahal is in ->", top_k(sure, 3))
print("my favourite drink is ->", top_k(open_, 3))
Output:
the Taj Mahal is in -> {'Agra': 0.94, 'Delhi': 0.02, 'Jaipur': 0.02}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2}
Top-k keeps the k most likely tokens and samples among them. A fixed k is too many when the model is sure (Delhi and Jaipur stay in for the Taj Mahal) and too few when many answers are fine (water and juice are cut).
Top-p (nucleus) sampling
def top_p(probs, p): # smallest set of tokens adding up to p
kept, total = {}, 0.0
for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
kept[token] = prob
total += prob
if total >= p:
break
return kept
sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}
print("the Taj Mahal is in ->", top_p(sure, 0.9))
print("my favourite drink is ->", top_p(open_, 0.9))
Output:
the Taj Mahal is in -> {'Agra': 0.94}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2, 'water': 0.2, 'juice': 0.2}
Top-p sorts tokens from most to least likely and keeps adding until their probabilities reach p, then samples only from that shortlist (the nucleus). It shrinks when the model is sure and grows when many answers are fine. It was proposed by Holtzman, Buys, Du, Forbes and Choi in "The Curious Case of Neural Text Degeneration" (2019, arXiv 1904.09751), who found that decoding by likelihood alone gives text that is "bland and strangely repetitive".
Which settings when?
| Task | A reasonable start |
|---|---|
| Code, maths, facts | low temperature (or 0) |
| Everyday chat | temperature ≈ 0.7–1, top-p ≈ 0.9 |
| Brainstorming, stories | higher temperature |
| Tuning | a common rule of thumb: adjust temperature or top-p, not both at once |
The context window: a train window
| On a train | In an LLM |
|---|---|
| You only see one stretch of the view | the context window: tokens the model can see |
| The station you left is out of sight | truncation: old tokens drop out |
| Note what matters in a diary | summaries and retrieval bring back what matters |
chat = "my name is raju . i love chai . i study in pune . what is my name ?"
tokens = chat.split()
WINDOW = 12 # the model sees only the last 12 tokens
seen = tokens[-WINDOW:]
print("all tokens: ", len(tokens))
print("model sees: ", " ".join(seen))
print("'raju' visible?", "raju" in seen)
Output:
all tokens: 19
model sees: chai . i study in pune . what is my name ?
'raju' visible? False
With a 12-token window, "raju" fell out and the model can't answer. Real windows: GPT-2 small saw 1,024 tokens, GPT-3 2,048. Your prompt and the reply must both fit.
Why not an infinite window?
for window in [1024, 2048, 4096]: # attention compares every token with every token
print(f"{window:5} tokens -> {window * window:>10,} attention scores per head")
Output:
1024 tokens -> 1,048,576 attention scores per head
2048 tokens -> 4,194,304 attention scores per head
4096 tokens -> 16,777,216 attention scores per head
Attention compares every token with every token, so doubling the window means four times the attention scores. Newer models use clever tricks to stretch their windows much further, but longer context always costs more time and money.
Living with the window
| Technique | Idea | Where in this course |
|---|---|---|
| Drop the oldest messages | simple truncation | this episode |
| Summarise the old part | keep a short summary | episode 30 |
| Retrieve only what's needed | put the relevant paragraphs in the window | RAG, episode 24 |
| Save facts outside the model | agent memory | episode 30 |
max_tokens and stopping
reply = "Chai is made by boiling tea leaves with milk , sugar and ginger ."
tokens = reply.split()
def generate(max_tokens):
out = []
for t in tokens: # stand-in for the model's tokens
if len(out) == max_tokens:
return " ".join(out), "stopped: max_tokens"
out.append(t)
if t == ".":
return " ".join(out), "stopped: end of answer"
return " ".join(out), "done"
print(generate(6))
print(generate(50))
Output:
('Chai is made by boiling tea', 'stopped: max_tokens')
('Chai is made by boiling tea leaves with milk , sugar and ginger .', 'stopped: end of answer')
Generation stops when the model produces an end token (like GPT-2's <|endoftext|>) or when it reaches max_tokens. An answer cut off mid-sentence usually means max_tokens was too small.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Decoding | choosing the next token from the probabilities | greedy, sampling, top-p |
| Greedy decoding | always take the most likely token | loops "neha drinks chai" |
| Uniform random | every token equally likely | "neha drinks Tuesday" |
| Sampling | pick at random, weighted by probability | random.choices(vocab, weights=probs) |
| Random seed | a starting number that makes random draws repeatable | random.seed(7) |
| Logits | raw scores before softmax | [3.0, 2.4, 1.6, 1.4, 0.2] |
| Temperature | divide the logits by T before softmax | T = 0.2 → chai 0.95 |
| Long tail | many unlikely tokens whose chances add up | 50,000 × 0.00001 = 0.5 |
| Top-k sampling | sample only from the k most likely tokens | k = 3 |
| Top-p sampling | sample from the smallest set reaching probability p | p = 0.9 → just "Agra" |
| Nucleus | the top-p shortlist | Holtzman et al., 2019 |
| Context window | the most tokens a model can see at once | GPT-3: 2,048 |
| Truncation | dropping the oldest tokens to fit | "raju" falls out |
| Quadratic cost | work grows with the square of the length | 2× tokens = 4× scores |
| max_tokens | the most tokens the model may generate | 6 → cut mid-sentence |
| Stop condition | when generation ends | end token or max_tokens |
| End token | a special token meaning "done" | <|endoftext|> |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Temperature 0 in your own softmax | softmax(logits, temperature=0) |
ZeroDivisionError: float division by zero |
| Tokens and weights of different lengths | random.choices(["chai", "coffee"], weights=[0.5, 0.3, 0.2]) |
ValueError: The number of weights does not match the population |
| Filtering out every token | random.choices(["chai"], weights=[0]) |
ValueError: Total of weights must be greater than zero |
| An empty shortlist | random.choices([], weights=[]) |
IndexError: list index out of range |
| max_tokens too small | generate(6) |
no error; answer cut mid-sentence |
| A long chat with a small window | 19 tokens, window 12 | no error; the model forgets your name |
Fixes: treat temperature 0 as greedy; keep tokens and probabilities together; always keep at least the top token; raise max_tokens; summarise or retrieve instead of stuffing the window.
Mini project: your own sampler
Combine everything: write generate(follows, start, steps, temperature, p) for the counting model from episode 15 that turns counts into logits (math.log(count)), applies temperature, keeps the top-p shortlist and samples with random.choices. Compare temperature 0.3 and 1.5.
Practice
- Run the temperature demo with 0.5 and 5.0.
- Run
top_pwith p = 0.5 on both prompts. - Change
WINDOWso "raju" stays visible. - Write
sample()= top-p +random.choices.
Solutions
1.
import math
vocab = ["chai", "coffee", "water", "lassi", "cricket"]
logits = [3.0, 2.4, 1.6, 1.4, 0.2] # raw scores from the model
def softmax(scores, temperature):
e = [math.exp(s / temperature) for s in scores] # divide, then softmax
return [round(x / sum(e), 2) for x in e]
for t in [0.5, 5.0]:
print(f"temperature {t}:", dict(zip(vocab, softmax(logits, t))))
temperature 0.5: {'chai': 0.71, 'coffee': 0.21, 'water': 0.04, 'lassi': 0.03, 'cricket': 0.0}
temperature 5.0: {'chai': 0.25, 'coffee': 0.23, 'water': 0.19, 'lassi': 0.18, 'cricket': 0.14}
2.
def top_p(probs, p): # smallest set of tokens adding up to p
kept, total = {}, 0.0
for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
kept[token] = prob
total += prob
if total >= p:
break
return kept
sure = {"Agra": 0.94, "Delhi": 0.02, "Jaipur": 0.02, "pizza": 0.02}
open_ = {"chai": 0.2, "coffee": 0.2, "lassi": 0.2, "water": 0.2, "juice": 0.2}
print("the Taj Mahal is in ->", top_p(sure, 0.5))
print("my favourite drink is ->", top_p(open_, 0.5))
the Taj Mahal is in -> {'Agra': 0.94}
my favourite drink is -> {'chai': 0.2, 'coffee': 0.2, 'lassi': 0.2}
With p = 0.5, the drinks shortlist stops at three (0.2 + 0.2 + 0.2 = 0.6 ≥ 0.5).
3.
chat = "my name is raju . i love chai . i study in pune . what is my name ?"
tokens = chat.split()
WINDOW = 16 # big enough to keep 'raju'
seen = tokens[-WINDOW:]
print("all tokens: ", len(tokens))
print("model sees: ", " ".join(seen))
print("'raju' visible?", "raju" in seen)
all tokens: 19
model sees: raju . i love chai . i study in pune . what is my name ?
'raju' visible? True
"raju" is the 4th of 19 tokens, so the window must be at least 16.
4.
import random
random.seed(1)
def top_p(probs, p):
kept, total = {}, 0.0
for token, prob in sorted(probs.items(), key=lambda t: -t[1]):
kept[token] = prob
total += prob
if total >= p:
break
return kept
def sample(probs, p=0.9): # top-p, then pick by probability
kept = top_p(probs, p)
return random.choices(list(kept), weights=list(kept.values()))[0]
probs = {"chai": 0.45, "coffee": 0.25, "water": 0.12, "lassi": 0.10,
"cricket": 0.03, "homework": 0.02, "Tuesday": 0.02, "blue": 0.01}
print("shortlist:", list(top_p(probs, 0.9)))
print([sample(probs) for _ in range(10)])
shortlist: ['chai', 'coffee', 'water', 'lassi']
['chai', 'water', 'water', 'chai', 'coffee', 'chai', 'coffee', 'water', 'chai', 'chai']
The junk tokens can never be picked: they are outside the 0.9 nucleus.
Quick quiz 1
You want Python code that must run. Best starting setting? A: temperature 2.0 · B: temperature 0.2 · C: top-k = 50,000 · D: max_tokens = 5
Answer
B: temperature 0.2. Code needs the most likely tokens.
Quick quiz 2
Probabilities A 0.6, B 0.3, C 0.06, D 0.04. Top-p with p = 0.9 keeps? A: A only · B: A and B · C: A, B and C · D: all four
Answer
B: A and B. 0.6 < 0.9; 0.6 + 0.3 = 0.9 reaches p, so stop.
Interview questions
- What does temperature do mathematically? Divides the logits by T before softmax: T < 1 sharpens the distribution, T > 1 flattens it.
- Top-k vs top-p? Top-k keeps a fixed number of tokens; top-p keeps the smallest set whose probabilities reach p, so it adapts to the model's confidence.
- Why does greedy decoding repeat itself? It deterministically follows the highest-probability path, which often cycles; sampling breaks the cycle.
- Why is the long tail dangerous? Thousands of tiny probabilities add up to a large chance of picking a bad token.
- What is a context window, and what happens when it overflows? The maximum tokens the model can attend to; older tokens are truncated, so the model "forgets".
- Why are long contexts expensive? Self-attention compares every pair of tokens: cost grows with the square of the length.
- Why might an answer end mid-sentence? It hit max_tokens before the model produced its end token.
Summary
- Greedy loops; uniform random is nonsense; sampling by probability is varied and sensible (a seed makes it repeatable).
- Temperature divides the logits: low = safe, high = wild.
- The long tail of tiny chances adds up, so cut it: top-k (fixed size) or top-p (adapts).
- The context window limits what the model sees; old tokens are truncated; cost grows with the square of the length.
- max_tokens and the end token decide where an answer stops.
- Next episode: from raw model to assistant: fine-tuning and RLHF.