These are the notes for Generative AI Engineering episode 6. Ask a model the same question twice and you can get two different answers. In this episode you find out exactly why, by building the last step of a language model yourself: the part that picks each next token. You will turn raw scores (logits) into probabilities with softmax, watch the two obvious ways of picking a token fail on screen (greedy decoding loops forever, pure sampling picks junk), and then invent the fixes: temperature, top-k, top-p, seeds, stop sequences, frequency and presence penalties, logprobs and beam search. Finally you will see what real APIs let you change today, and which settings fit which task. Every program below runs as-is in the SyntaxPilot compiler (language: Python) with no API key and no extra packages; every output is a real run on Python 3.13.18.
Important: all the scores (logits) and probabilities in these programs are made up for teaching. A real model has a vocabulary of many thousands of tokens and computes its scores with a neural network. The ideas are exactly the same; only the numbers are invented.
The problem: one model, two moods
Meena runs a saree shop and uses her bot for two jobs:
| Job | Example | What she wants |
|---|---|---|
| Order status | "Your order ships tomorrow." | the same correct answer every time |
| Saree descriptions | 100 products, 100 product lines | variety: a fresh line for each saree |
Same model, two moods. Where does that choice happen? Remember episode 4: a model reads your text as tokens (small pieces of text, like words or parts of words) and, at each step, it gives a score to every token it knows. Then one token is picked, added to the text, and the whole thing repeats. Today we control that pick.
Logits and softmax: from scores to probabilities
A logit is the raw score a model gives a possible next token. Higher means "more likely to come next". Logits can be any number: big, small, even negative. Example: after the text "Delivery takes 3 to", the token 5 might get 4.0 and the token banana might get -3.0.
Raw scores are hard to use directly, so we turn them into probabilities. A probability is a number from 0 to 1 that says how likely something is: 0 means never, 1 means certain, 0.25 means "one time in four". The probabilities of all possible next tokens must add up to exactly 1 (100%).
The tool that does this is softmax. It has two steps:
- Raise e to the power of each score. e is a fixed number, about 2.718 (like π is about 3.14, e is a special constant from maths). "e to the power of 2" (written e², or
math.exp(2)in Python) means 2.718 × 2.718 ≈ 7.39. The small number on top is called the exponent. The useful property: e to any power is always positive, even for a negative score (e⁻³ ≈ 0.05), and a bigger score always gives a bigger result. - Divide each result by the total, so they all add up to 1.
Small worked example. Two tokens with scores 2 and 0:
| Token | Score | e^score | ÷ total (8.39) | Probability |
|---|---|---|---|---|
| A | 2 | 7.39 | 7.39 / 8.39 | 0.88 |
| B | 0 | 1.00 | 1.00 / 8.39 | 0.12 |
0.88 + 0.12 = 1. A gap of 2 in scores became "88% versus 12%".
Now in Python, with five tokens:
import math
# raw scores for the next token after "Delivery takes 3 to"
logits = {"5": 4.0, "7": 2.5, "4": 2.0, "10": 1.0, "banana": -3.0}
def softmax(scores):
exps = {t: math.exp(s) for t, s in scores.items()} # e^score: always > 0
total = sum(exps.values())
return {t: round(e / total, 3) for t, e in exps.items()}
print(softmax(logits))
print("sum:", round(sum(softmax(logits).values()), 3))
print("greedy pick:", max(logits, key=logits.get))
{'5': 0.71, '7': 0.158, '4': 0.096, '10': 0.035, 'banana': 0.001}
sum: 1.0
greedy pick: 5
5 gets 71%, 7 about 16%, banana a tenth of a percent; they add up to 1. max(logits, key=logits.get) finds the token with the highest score.
Now the real question: which token do we actually pick? The rule for that is called decoding. Decoding is the rule that turns probabilities into one chosen token, step after step, until the reply is finished. Example: "always take the most likely token" is one decoding rule; "pick randomly, weighted by probability" is another.
Obvious attempt 1: greedy decoding (always order your favourite dish)
The obvious rule: always pick the token with the highest score. This is greedy decoding. It's like always ordering your favourite dish at a restaurant: predictable, never surprising, and you eat the same meal forever.
Below is a tiny made-up "saree model". For each token, a dictionary gives the scores of the tokens that may come next. <s> is the start token (the beginning of the text) and </s> is the end token (the model's way of saying "I'm done").
LOGITS = {"<s>": {"our": 3.0, "these": 1.0},
"our": {"sarees": 3.0, "silk": 2.0},
"these": {"sarees": 2.0},
"silk": {"sarees": 2.0},
"sarees": {"are": 3.0, "feel": 1.5},
"feel": {"light": 2.0},
"are": {"soft": 2.5, "handwoven": 2.0, "light": 1.5},
"handwoven": {"and": 1.5, "</s>": 1.0},
"light": {"and": 1.0, "</s>": 2.0},
"soft": {"and": 2.0, "</s>": 1.8},
"and": {"our": 2.2, "light": 1.0, "</s>": 0.5}}
def greedy(max_tokens=15):
out = ["<s>"]
while out[-1] != "</s>" and len(out) <= max_tokens:
scores = LOGITS[out[-1]]
out.append(max(scores, key=scores.get)) # always the top token
return " ".join(out[1:])
print(greedy())
print(greedy())
our sarees are soft and our sarees are soft and our sarees are soft and
our sarees are soft and our sarees are soft and our sarees are soft and
Why it failed: after and, the top token is our, which leads back to sarees are soft and, which leads back to our... The text loops until the 15-token limit (max_tokens) cuts it off. And both runs are identical: greedy can never vary. Good for Meena's order status, useless for 100 different saree descriptions.
Obvious attempt 2: pure sampling picks junk
The second obvious rule: pick at random, but use the probabilities as weights, so a 70% token is picked about 70% of the time. This is sampling. It's like drawing a ticket from a box where the likely tokens have many tickets and unlikely ones have few.
To be realistic, we add 40 unlikely "junk" tokens. A real vocabulary has thousands of rare tokens like this. All those individually-unlikely tokens together are called the long tail (draw a bar chart sorted from tallest to shortest, and the many tiny bars on the right look like a long thin tail).
import math, random
random.seed(1)
logits = {"5": 4.0, "7": 2.5, "4": 2.0, "10": 1.0}
for i in range(40):
logits[f"junk{i}"] = -0.5 # 40 unlikely tokens: the long tail
def softmax(scores, T=1.0):
exps = {t: math.exp(s / T) for t, s in scores.items()}
total = sum(exps.values())
return {t: e / total for t, e in exps.items()}
probs = softmax(logits)
picks = random.choices(list(probs), list(probs.values()), k=20)
print(picks)
print("junk picks:", sum(p.startswith("junk") for p in picks), "of 20")
['5', 'junk14', 'junk0', '5', '5', '5', '7', 'junk4', '5', '5', 'junk12', '5', 'junk0', '5', '5', '4', '5', 'junk30', 'junk23', '5']
junk picks: 7 of 20
random.choices(items, weights, k=20)picks 20 items at random, each with a chance proportional to its weight.random.seed(1)makes the random picks the same on every run (seeds are explained below), so you get exactly this output.
Why it failed: each junk token is unlikely on its own (about 0.6%), but 40 of them together hold about a quarter of all the probability. In a sentence that means a nonsense word every few tokens. (Notice 5 fell from 71% to 54%: the junk tokens took a share of the total.)
So: greedy never varies, and pure sampling varies but picks junk. We need a dial between the two.
Fix 1: temperature
Temperature (written T) is that dial. Before softmax, divide every logit by T:
probs = softmax(logits / T)
- T smaller than 1 (for example 0.2): dividing by a small number makes the gaps between scores bigger, so the top token dominates. Sharper, safer.
- T = 1: no change, the original probabilities.
- T bigger than 1 (for example 2): the gaps get smaller, so all tokens become more equal. Flatter, wilder.
Back to our two tokens with scores 2 and 0:
| T | Scores after ÷ T | Probability of A |
|---|---|---|
| 0.5 | 4 and 0 | 54.6 / 55.6 ≈ 0.98 |
| 1 | 2 and 0 | 7.39 / 8.39 ≈ 0.88 |
| 2 | 1 and 0 | 2.72 / 3.72 ≈ 0.73 |
In the video, this loop is added at the end of the sampling program above:
for T in [0.2, 0.5, 1.0, 2.0]:
p = softmax(logits, T)
junk = sum(v for t, v in p.items() if t.startswith("junk"))
print(f"T={T}: P('5')={p['5']:.2f} P('7')={p['7']:.2f} all junk={junk:.2f}")
Full output of the combined program:
['5', 'junk14', 'junk0', '5', '5', '5', '7', 'junk4', '5', '5', 'junk12', '5', 'junk0', '5', '5', '4', '5', 'junk30', 'junk23', '5']
junk picks: 7 of 20
T=0.2: P('5')=1.00 P('7')=0.00 all junk=0.00
T=0.5: P('5')=0.93 P('7')=0.05 all junk=0.00
T=1.0: P('5')=0.54 P('7')=0.12 all junk=0.24
T=2.0: P('5')=0.16 P('7')=0.08 all junk=0.67
- At T = 0.2,
5gets (rounded) 100%: basically greedy. Temperature near 0 behaves like greedy decoding. - At T = 0.5, junk is (rounded) zero.
- At T = 1, the original: junk is 24%.
- At T = 2, junk is 67%: mostly nonsense.
Low temperature for facts, higher temperature for variety. (Note: you can't use T = 0 in this formula, because dividing by zero is an error. That's why programs treat "temperature 0" as a special case meaning "use greedy".)
Fix 2: top-k and top-p (shortlist first, then choose)
Temperature lowers the junk, but never to exactly zero. Better: cut the tail off completely. Think of hiring:
| Hiring | Choosing a token |
|---|---|
| "Interview only the top 3 candidates" (a fixed number) | Top-k: keep only the k most likely tokens |
| "Interview the top candidates who together got 90% of the votes" (as many as needed: sometimes 2 people, sometimes 10) | Top-p (also called nucleus sampling): keep the smallest group of best tokens whose probabilities add up to at least p |
| Then pick one, only from the shortlist | Sample from the shortlist: junk never gets a chance |
After cutting, the kept probabilities no longer add up to 1, so we renormalise: divide each by their new total so they add up to 1 again. Example: kept shares 0.6 and 0.2 (total 0.8) become 0.6/0.8 = 0.75 and 0.2/0.8 = 0.25.
import math
logits = {"5": 4.0, "7": 2.5, "4": 2.0, "10": 1.0}
for i in range(40):
logits[f"junk{i}"] = -0.5
def softmax(scores, T=1.0):
exps = {t: math.exp(s / T) for t, s in scores.items()}
total = sum(exps.values())
return {t: e / total for t, e in exps.items()}
def top_k(probs, k): # keep the k most likely tokens
best = sorted(probs, key=probs.get, reverse=True)[:k]
return {t: probs[t] for t in best}
def top_p(probs, p): # keep the smallest set reaching p
kept, total = {}, 0.0
for t in sorted(probs, key=probs.get, reverse=True):
kept[t] = probs[t]; total += probs[t]
if total >= p: break
return kept
def renorm(kept): # shares must add up to 1 again
s = sum(kept.values())
return {t: round(v / s, 2) for t, v in kept.items()}
probs = softmax(logits)
print("top_k=3: ", renorm(top_k(probs, 3)))
print("top_p=0.6:", renorm(top_p(probs, 0.6)))
print("top_p=0.9:", len(top_p(probs, 0.9)), "tokens kept")
top_k=3: {'5': 0.74, '7': 0.16, '4': 0.1}
top_p=0.6: {'5': 0.82, '7': 0.18}
top_p=0.9: 28 tokens kept
top_k sorts tokens from most to least likely and keeps the first k; top_p walks down the same sorted list, adding probabilities, and stops as soon as the running total reaches p.
- Top-k = 3:
5,7,4and nothing else. - Top-p = 0.6: only two tokens were needed (
5is 0.54, plus7makes 0.66). - Top-p = 0.9 kept 28 tokens: our junk tail is flat, so it needed 24 junk tokens to climb from 76% to 90%.
So top-p adapts to the shape of the probabilities: few tokens when the model is sure, more when it isn't. Top-k is fixed: always exactly k.
Seeds and determinism
Sampling is random, so how do you reproduce a bug? A seed is the starting number for a random number generator (the part of a program that produces random-looking choices). The generator isn't truly random: it follows a fixed recipe from its starting number. Same seed, same sequence of "random" picks.
import random
words = ["soft", "light", "handwoven", "elegant", "festive"]
def sample(n):
return [random.choice(words) for _ in range(n)]
print("no seed: ", sample(4))
print("no seed: ", sample(4))
random.seed(42); print("seed 42: ", sample(4))
random.seed(42); print("seed 42: ", sample(4))
Output of one run:
no seed: ['soft', 'handwoven', 'soft', 'light']
no seed: ['light', 'handwoven', 'handwoven', 'elegant']
seed 42: ['soft', 'soft', 'handwoven', 'light']
seed 42: ['soft', 'soft', 'handwoven', 'light']
The two "no seed" lines are different on every run, so yours will not match these (a second run here gave ['soft', 'light', 'light', 'handwoven'] and ['soft', 'elegant', 'elegant', 'soft']). The two "seed 42" lines are identical to each other and identical on every run.
Determinism means same input, same output, every time (example: the "seed 42" lines). Non-determinism means the same input can give different outputs (example: the "no seed" lines).
On a provider's servers, many things can change the result: different hardware, batching your request together with other people's requests, model updates. OpenAI's reference says its seed parameter is best effort, and determinism is not guaranteed. So never write tests that expect a model's exact words.
Stop sequences
Sometimes a model keeps going and starts writing the customer's next line too, as if it were a play script. A stop sequence is a piece of text that makes generation stop the moment it appears. Example: "\nCustomer:" (a new line followed by Customer:; \n means "new line").
draft = ("Yes, COD is available.\n"
"Customer: and returns?\n"
"Agent: 7 days.")
STOP = ["\nCustomer:"] # stop sequences
def apply_stop(text, stops):
for s in stops:
i = text.find(s)
if i != -1: # cut BEFORE it
return text[:i], "stop_sequence", s
return text, "end_turn", None
print(apply_stop(draft, STOP))
('Yes, COD is available.', 'stop_sequence', '\nCustomer:')
text.find(s)returns the position wheresstarts, or-1if it isn't there;text[:i]keeps everything before that position.- We get only the agent's line, the reason we stopped (
"stop_sequence"), and which stop sequence matched.
With a real API, you pass the stop sequences and the provider does this check for you. In Anthropic's API you pass stop_sequences, and the response says stop_reason "stop_sequence", plus which one matched. OpenAI's reference allows up to 4 stop sequences and says the returned text does not include the stop sequence.
Frequency and presence penalties
Back to the greedy loop. The fix: punish repeats. Before picking, lower the score of each token that was already used.
- Frequency penalty: subtract a bit for every time the token was already used. With 0.6, a token used twice loses 1.2.
- Presence penalty: subtract a fixed amount if the token was used even once (it doesn't grow). With 1.0, a token used once or five times loses 1.0.
LOGITS = {"<s>": {"our": 3.0, "these": 1.0}, "our": {"sarees": 3.0, "silk": 2.0},
"these": {"sarees": 2.0}, "silk": {"sarees": 2.0},
"sarees": {"are": 3.0, "feel": 1.5}, "feel": {"light": 2.0},
"are": {"soft": 2.5, "handwoven": 2.0, "light": 1.5},
"handwoven": {"and": 1.5, "</s>": 1.0}, "light": {"and": 1.0, "</s>": 2.0},
"soft": {"and": 2.0, "</s>": 1.8}, "and": {"our": 2.2, "light": 1.0, "</s>": 0.5}}
def greedy(freq=0.0, pres=0.0, max_tokens=15):
out = ["<s>"]
while out[-1] != "</s>" and len(out) <= max_tokens:
scores = dict(LOGITS[out[-1]])
for t in scores:
n = out.count(t) # times t was already used
scores[t] -= freq * n + pres * (n > 0)
out.append(max(scores, key=scores.get))
return " ".join(out[1:])
print("no penalty: ", greedy())
print("frequency 0.6: ", greedy(freq=0.6))
print("presence 1.0: ", greedy(pres=1.0))
no penalty: our sarees are soft and our sarees are soft and our sarees are soft and
frequency 0.6: our sarees are soft and our sarees are handwoven </s>
presence 1.0: our sarees are soft and our sarees are handwoven </s>
Step by step with frequency 0.6: the second time we reach are, soft has been used once, so its score drops from 2.5 to 1.9, below handwoven (2.0). handwoven wins. After handwoven, and (used once) drops from 1.5 to 0.9, below </s> (1.0), so the sentence ends. ((n > 0) is True or False; Python treats them as 1 or 0 in maths.)
OpenAI's reference allows both penalties from -2.0 to 2.0. Positive values reduce repetition, and a presence penalty pushes the model towards new topics. This code shows the idea, not any provider's exact formula.
Logprobs: a confidence signal
Probabilities also tell you how sure the model is. Some APIs return a logprob for each token: the natural logarithm of its probability.
The natural logarithm (written ln, math.log in Python) is the reverse of "e to the power of": it answers "e to what power gives this number?". For probabilities (0 to 1) it gives 0 or a negative number:
| Probability | logprob (ln) | Meaning |
|---|---|---|
| 1.0 | 0 | 100% sure |
| 0.9 | -0.11 | very sure |
| 0.5 | -0.69 | a coin toss |
| 0.1 | -2.30 | unlikely |
So 0 means 100% sure, and more negative means less sure.
import math
answers = {"Is COD available?": {"Yes": 0.92, "No": 0.05, "Maybe": 0.03},
"Is gift wrap free?": {"Yes": 0.41, "No": 0.38, "Maybe": 0.21}}
for q, probs in answers.items():
best = max(probs, key=probs.get)
lp = math.log(probs[best]) # logprob = log of probability
sure = "confident" if lp > math.log(0.8) else "NOT sure: ask a human"
print(f"{q:20} -> {best:4} logprob={lp:.2f} {sure}")
print("two tokens, 0.9 then 0.5:", round(math.log(0.9) + math.log(0.5), 2),
"=", round(math.log(0.9 * 0.5), 2))
Is COD available? -> Yes logprob=-0.08 confident
Is gift wrap free? -> Yes logprob=-0.89 NOT sure: ask a human
two tokens, 0.9 then 0.5: -0.8 = -0.8
- If the best answer's logprob is below ln(0.8) ≈ -0.22 (less than 80% sure), we hand the question to a human.
- COD: -0.08, confident. Gift wrap: -0.89 (Yes 41% vs No 38%: nearly a tie), so ask a human.
- Why logs are handy: the probability of a whole sentence is the probabilities of its tokens multiplied (0.9 × 0.5 = 0.45). With logs you just add them: ln(0.9) + ln(0.5) = ln(0.45) ≈ -0.8. Adding is simpler, and multiplying many small numbers would get too tiny for a computer to store well.
OpenAI's reference lets you ask for logprobs up to 5.
Beam search: keep several candidates
One more decoding method. Greedy picks the best token at each step, but the best first step doesn't always lead to the best sentence. Beam search keeps several partial sentences at once and extends all of them. The number it keeps is the beam width.
import math
P = {"<s>": {"The": 0.5, "Our": 0.4, "A": 0.1},
"The": {"saree": 0.3, "silk": 0.3, "blue": 0.4},
"Our": {"sarees": 0.9, "silk": 0.1},
"A": {"gift": 1.0}}
def greedy():
seq, lp = ["<s>"], 0.0
for _ in range(2):
nxt = P[seq[-1]]
t = max(nxt, key=nxt.get)
seq.append(t); lp += math.log(nxt[t])
return " ".join(seq[1:]), round(lp, 2)
def beam(width=2):
beams = [(["<s>"], 0.0)]
for _ in range(2): # extend all, keep best
cand = []
for s, lp in beams:
for t, p in P[s[-1]].items():
cand.append((s + [t], lp + math.log(p)))
cand.sort(key=lambda b: b[1], reverse=True)
beams = cand[:width]
return [(" ".join(s[1:]), round(lp, 2))
for s, lp in beams]
print("greedy:", greedy())
print("beam: ", beam())
greedy: ('The blue', -1.61)
beam: [('Our sarees', -1.02), ('The blue', -1.61)]
The maths, by hand:
| Sentence | Probabilities | Multiplied | Total logprob |
|---|---|---|---|
| The blue (greedy) | 0.5 × 0.4 | 0.20 | -1.61 |
| Our sarees (beam) | 0.4 × 0.9 | 0.36 | -1.02 |
The is the most likely first word (0.5), but everything after it is weak. Beam search with width 2 also kept Our, and found the more likely sentence Our sarees. Beam search is common in translation systems, but chat model APIs usually sample instead.
What real APIs allow
From the providers' documentation (fetched 28 Oct 2026). This changed recently, so always re-check.
| Provider | What the docs say |
|---|---|
| Anthropic, API version 2026-10-21 (migration guide) | A request that sends temperature, top_p or top_k returns a 400 error, even at the parameter's default value. |
| Anthropic: Claude 4.7 and later models | Already accept only the defaults, which include a temperature of 1.0. |
Anthropic: stop_sequences |
Still supported; the response has stop_reason "stop_sequence" and the matched stop sequence. |
OpenAI reference: temperature |
Between 0 and 2. |
OpenAI reference: top_p |
Nucleus sampling; 0.1 means only the tokens in the top 10% probability mass are considered. |
OpenAI reference: frequency_penalty, presence_penalty |
From -2.0 to 2.0. |
OpenAI reference: seed |
Best effort; determinism is not guaranteed. |
OpenAI reference: stop |
Up to 4 sequences; the returned text does not include the stop sequence. |
OpenAI reference: logprobs |
Up to 5. |
(Probability mass means a share of the total probability: "the top 10% probability mass" is the most likely tokens that together add up to 0.1.) A 400 error is the HTTP status code for "problem with your request" (episode 3).
The lesson: always check the current docs for the exact model you call, and never copy sampling settings between providers.
Choosing settings per task
Choosing settings per task means matching the decoding knobs to what the job needs: consistency or variety.
| Task | Example (Meena's shop) | Settings |
|---|---|---|
| Facts, extraction, classification | order status, "is COD available?" | low temperature or greedy, where the API allows it |
| Descriptions, ideas, brainstorming | 100 saree descriptions | higher temperature, with top-p to cut the junk tail |
| Every task | everything | max_tokens + stop sequences + check stop_reason |
| Sampling settings not allowed | newer models | get consistency from clear prompts, structured output and tests (coming up in this course) |
| Judging any setting | before going live | never trust one sample: measure on many examples (evaluation, later in the course) |
Common mistakes (and the real symptoms)
| Mistake | Error or symptom | Why | Fix |
|---|---|---|---|
Expecting temperature=0 to give identical answers forever |
no error: outputs still change | provider servers are only best-effort deterministic | test the meaning, not the exact words |
Sending temperature=0.2 on Anthropic API version 2026-10-21 |
400 error (per the migration guide) | the parameter was removed | remove temperature, top_p, top_k |
| High temperature, no top-p | no error: random junk words | the long tail gets a big share | lower T, or cut the tail with top-p |
| Greedy with no limits | no error: ...soft and our sarees are soft... |
greedy can loop | max_tokens, stop sequences, penalties |
Calling our softmax(logits, 0) |
ZeroDivisionError: float division by zero |
the formula divides by T | treat T = 0 as "use greedy" (as the mini project does) |
Mini project: generate() with every knob
One function with everything from today: temperature, top-k, top-p, a frequency penalty, a seed and max_tokens. random.Random(seed) makes its own random number generator from the seed, so it doesn't disturb any other random code.
import math, random
LOGITS = {"<s>": {"our": 3.0, "these": 1.0}, "our": {"sarees": 3.0, "silk": 2.0},
"these": {"sarees": 2.0}, "silk": {"sarees": 2.0},
"sarees": {"are": 3.0, "feel": 1.5}, "feel": {"light": 2.0},
"are": {"soft": 2.5, "handwoven": 2.0, "light": 1.5},
"handwoven": {"and": 1.5, "</s>": 1.0}, "light": {"and": 1.0, "</s>": 2.0},
"soft": {"and": 2.0, "</s>": 1.8}, "and": {"our": 2.2, "light": 1.0, "</s>": 0.5}}
def generate(temperature=1.0, top_k=None, top_p=1.0, freq=0.0, seed=None, max_tokens=15):
rng = random.Random(seed)
out = ["<s>"]
while out[-1] != "</s>" and len(out) <= max_tokens:
scores = {t: s - freq * out.count(t) for t, s in LOGITS[out[-1]].items()}
if temperature == 0: # greedy
out.append(max(scores, key=scores.get)); continue
exps = {t: math.exp(s / temperature) for t, s in scores.items()}
total = sum(exps.values())
probs = sorted(((e / total, t) for t, e in exps.items()), reverse=True)
probs = probs[:top_k] if top_k else probs # top-k
kept, total = [], 0.0
for p, t in probs: # top-p
kept.append((p, t)); total += p
if total >= top_p: break
out.append(rng.choices([t for p, t in kept], [p for p, t in kept])[0])
return " ".join(out[1:])
print("greedy, T=0: ", generate(temperature=0, freq=0.6))
for s in [1, 2, 3]:
print(f"seed {s}, T=1.2:", generate(temperature=1.2, top_p=0.9, freq=0.6, seed=s))
greedy, T=0: our sarees are soft and our sarees are handwoven </s>
seed 1, T=1.2: our silk sarees are handwoven and light </s>
seed 2, T=1.2: these sarees are soft </s>
seed 3, T=1.2: our sarees are handwoven </s>
Order of steps: penalty → (T = 0? greedy) → temperature → softmax → sort → top-k → top-p → sample from the shortlist. Greedy with a penalty gives the same safe sentence every time: good for order messages. The creative runs give three different descriptions, and because of the seeds you'll get exactly these three too. Try temperature=3 and watch odd choices come back.
Practice
- Add
top_k=2to the three creative runs of the mini project. What changes? - In the temperature program, find a temperature where the total junk probability is 0.05 (to 3 decimals).
- Make the beam search also print the result with width 3 and compare.
- Change
apply_stopso that, with several stop sequences, it cuts at whichever appears first in the text. Test withdraft = "Yes, COD is available.\nAgent: Anything else?\nCustomer: and returns?"andSTOP = ["\nCustomer:", "\nAgent:"].
Solutions and hints
- Change the loop line to
generate(temperature=1.2, top_k=2, top_p=0.9, freq=0.6, seed=s). Output:
greedy, T=0: our sarees are soft and our sarees are handwoven </s>
seed 1, T=1.2: our silk sarees are soft and light </s>
seed 2, T=1.2: these sarees are soft </s>
seed 3, T=1.2: our sarees are handwoven </s>
Only seed 1 changes. In this toy model most tokens have at most two possible next tokens, so top-k = 2 only matters after are (three options: soft, handwoven, light), where light is now cut. A smaller shortlist also changes which token each random draw lands on.
- Junk grows as T grows, so search between 0.2 (junk ≈ 0) and 1.0 (junk ≈ 0.24), halving the gap each time (binary search). Add this below
logitsandsoftmaxfrom the temperature program:
def junk(T):
return sum(v for t, v in softmax(logits, T).items() if t.startswith("junk"))
low, high = 0.2, 1.0 # junk(0.2) is ~0, junk(1.0) is ~0.24
for _ in range(30): # halve the gap 30 times
mid = (low + high) / 2
if junk(mid) < 0.05: low = mid
else: high = mid
print(f"T={mid:.3f} gives junk={junk(mid):.3f}")
T=0.696 gives junk=0.050
- Add
print("beam 3:", beam(width=3)). Output:
greedy: ('The blue', -1.61)
beam: [('Our sarees', -1.02), ('The blue', -1.61)]
beam 3: [('Our sarees', -1.02), ('The blue', -1.61), ('The saree', -1.9)]
The best sentence is the same; width 3 just keeps one more candidate (The saree, 0.5 × 0.3 = 0.15). A wider beam explores more, but costs more work.
- The original returns the first stop sequence in the list that appears anywhere, so it gives
('Yes, COD is available.\nAgent: Anything else?', 'stop_sequence', '\nCustomer:'): the extra Agent line leaks through. Fix: find every match and take the smallest position:
draft = "Yes, COD is available.\nAgent: Anything else?\nCustomer: and returns?"
STOP = ["\nCustomer:", "\nAgent:"]
def apply_stop(text, stops):
found = [(text.find(s), s) for s in stops if text.find(s) != -1]
if not found:
return text, "end_turn", None
i, s = min(found) # the EARLIEST match wins
return text[:i], "stop_sequence", s
print(apply_stop(draft, STOP))
print(apply_stop("Yes, COD is available.", STOP))
('Yes, COD is available.', 'stop_sequence', '\nAgent:')
('Yes, COD is available.', 'end_turn', None)
Quick quiz
Logits are [4.0, 2.5, -3.0]. What does temperature 0.2 do?
probs = softmax([4.0, 2.5, -3.0] / T)
# T = 0.2
- A. Makes all three tokens equally likely
- B. Makes the top token almost certain
- C. Removes the top token
- D. Changes the logits inside the model
Answer
B. Dividing by a small T stretches the gaps between scores (4.0, 2.5, -3.0 become 20, 12.5, -15), so softmax gives the top token nearly all the probability. It behaves almost like greedy decoding. (D is wrong: temperature only changes how we pick from the scores, not the model itself.)
Interview questions
- What are logits, and how does softmax turn them into probabilities?
- What is decoding? Compare greedy decoding and sampling: what goes wrong with each on its own?
- What does temperature do mathematically? What happens as T approaches 0, and at a high T?
- What is the difference between top-k and top-p (nucleus sampling)? Why is top-p called adaptive?
- Why can't you rely on a seed for exact reproducibility with a hosted model?
- What is the difference between a frequency penalty and a presence penalty?
- What is a logprob? Why add logprobs instead of multiplying probabilities? How does beam search use them?
- An API rejects your
temperatureparameter. How do you get consistent outputs instead?
Glossary
| Term | Meaning | Example |
|---|---|---|
| Token | a small piece of text a model reads and writes | sarees, 5 |
| Vocabulary | all the tokens a model knows | thousands of tokens in a real model |
| Logit | the raw score a model gives a possible next token; any number | "5": 4.0, "banana": -3.0 |
| Probability | a number from 0 to 1 saying how likely something is | 0.71 = 71% |
| e | a fixed maths constant, about 2.718 | math.exp(1) ≈ 2.72 |
| Exponent / "e to the power of" | multiply e by itself that many times; always positive | e² ≈ 7.39, math.exp(2) |
| Softmax | e^score for each token, divided by the total, so they add up to 1 | 4.0 → 0.71 |
| Decoding | the rule that picks the next token from the probabilities | greedy, sampling, beam search |
| Greedy decoding | always pick the most likely token | the "favourite dish" rule |
| Sampling | pick at random, weighted by probability | random.choices(tokens, probs) |
| Start / end token | markers for the beginning and end of the text | <s>, </s> |
max_tokens |
the most tokens to generate before stopping | max_tokens=15 |
| Long tail | the many individually-unlikely tokens that together hold a big share | 40 junk tokens = 24% |
| Temperature (T) | divide logits by T before softmax: low = sharper, high = flatter | T = 0.2 ≈ greedy |
| Top-k | keep only the k most likely tokens, then sample | top_k=3 |
| Top-p (nucleus sampling) | keep the smallest set of best tokens whose probabilities reach p, then sample | top_p=0.9 |
| Probability mass | a share of the total probability | top 10% mass = 0.1 |
| Renormalise | divide the kept shares by their total so they add up to 1 again | 0.6, 0.2 → 0.75, 0.25 |
| Random number generator | the part of a program that produces random-looking choices | random.Random(seed) |
| Seed | the starting number for a random number generator | random.seed(42) |
| Determinism | same input, same output, every time | seed 42 lines match |
| Non-determinism | the same input can give different outputs | the "no seed" lines |
| Best effort | the provider tries, but doesn't promise | OpenAI's seed |
| Stop sequence | text that makes generation stop when it appears | "\nCustomer:" |
stop_reason |
why the reply ended | "stop_sequence", "end_turn" |
| Frequency penalty | subtract a bit from a token's score for every earlier use | 0.6 × times used |
| Presence penalty | subtract a fixed amount if the token was used at all | 1.0 once used |
| Natural logarithm (ln) | reverse of e-to-the-power: "e to what power gives this?" | ln(0.5) ≈ -0.69 |
| Logprob | the natural logarithm of a token's probability; 0 = sure | math.log(0.92) = -0.08 |
| Beam search | keep the best few partial sentences at each step | finds "Our sarees" |
| Beam width | how many partial sentences beam search keeps | width=2 |
| Binary search | find a value by halving the search range each time | Practice 2 |
| 400 error | HTTP status code for a problem with your request | sending temperature on 2026-10-21 |
| Choosing settings per task | matching decoding knobs to the job's need for consistency or variety | greedy for order status, higher T for descriptions |
Summary
- Logits are raw scores; softmax (e^score ÷ total) turns them into probabilities that add up to 1.
- Decoding picks the token. Greedy never varies and can loop; pure sampling picks junk from the long tail.
- Temperature is the dial: low = safe (near 0 ≈ greedy), high = wild. Top-k keeps k tokens; top-p keeps the best until they reach p and adapts to the shape.
- Seeds make your own sampling repeatable; providers only promise best effort, so test meaning, not exact words.
- Stop sequences end text at a marker; frequency and presence penalties stop repetition; logprobs measure confidence (add them to multiply probabilities); beam search keeps several candidates.
- Real APIs differ: Anthropic API version 2026-10-21 rejects
temperature,top_pandtop_kwith a 400 error; OpenAI's reference still lists them. Always check the current docs. - The toy logits here are made up for teaching, and the penalty code shows the idea, not any provider's exact formula.
Next episode: multi-turn chat and the context window. Models don't remember anything between calls, and long chats get slow and expensive: conversation history, token budgets, truncation and summarising old turns.