SyntaxPilot
Courses/AI: Zero to Agents/Lesson 15

AI: Zero to Agents · Lesson 15 of 40 · 10 min

What an LLM really is

queued

These are the notes for AI episode 15. You have built every part: tokens (episode 12), embeddings (13) and the Transformer (14). Here you put them to work. You build a tiny language model by counting, make it write by predicting, appending and repeating, watch greedy decoding loop forever and a longer context crash on text it never saw, then replace counting with a network that outputs a probability for every token, train it for free with self-supervised learning and the −log(p) loss, and scale it up into a large language model like GPT-3. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

Real-life example: your phone keyboard

On your phone In a language model
"Happy birthday ..." → it suggests "to", then "you" Next-token prediction: guess what comes next
It learned from what people type Training text: where the guesses come from
Keep tapping the middle suggestion: it writes whole sentences Generation: predict, append, repeat

The one job

A language model takes the tokens so far, the context, and outputs a probability for every token in the vocabulary: how likely is each one to come next?

"raju drinks"  →  model  →  chai 0.67, coffee 0.33

A counting model (a bigram model)

from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()

follows = defaultdict(Counter)        # which word comes after each word?
for a, b in zip(words, words[1:]):
    follows[a][b] += 1

for w in ["drinks", "likes"]:
    total = sum(follows[w].values())
    print(f"after '{w}':", {n: round(c / total, 2) for n, c in follows[w].items()})

Output:

after 'drinks': {'chai': 0.67, 'coffee': 0.33}
after 'likes': {'chai': 0.5, 'cricket': 0.5}

Counting which word follows which, and dividing by the total, is already a real language model. One that looks at only the previous word is a bigram model; one that looks at the previous n − 1 words is an n-gram model.

Predict, append, repeat

from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    follows[a][b] += 1

word, story = "raju", ["raju"]
for _ in range(15):                   # predict, append, repeat
    word = follows[word].most_common(1)[0][0]
    story.append(word)
print(" ".join(story))

Output:

raju drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .

Always taking the single most likely token is greedy decoding. With one word of memory, the model has no idea what it already said: after "." it always says "neha", after "neha" always "drinks". It loops forever.

More context, more problems

from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter)        # now: the last TWO words -> next word
for a, b, c in zip(words, words[1:], words[2:]):
    follows[(a, b)][c] += 1

print("seen 2-word contexts:", len(follows))
print(f"possible with GPT-2's vocabulary: {50257 ** 2:,}")
for context in [("neha", "drinks"), ("raju", "plays")]:
    print(context, "->", follows[context].most_common(1)[0][0])

Output:

seen 2-word contexts: 20
possible with GPT-2's vocabulary: 2,525,766,049
('neha', 'drinks') -> coffee
IndexError: list index out of range

"neha drinks" → coffee works, but "raju plays" never appeared, so there is nothing to count and the program crashes. With GPT-2's 50,257-token vocabulary there are 2,525,766,049 possible two-word contexts and about 127 trillion three-word ones (50,257³ = 126,937,424,324,593). No text contains them all: counting cannot generalize to things it never saw. This is the sparsity problem.

Replace counting with a network that generalizes

tokens (ep 12) → embeddings (ep 13) → Transformer blocks (ep 14) → one score per token → softmax → probabilities
  • Similar words have similar embeddings, so what the model learns about "neha plays" carries over to "raju plays".
  • Attention looks back over the whole context, not just one or two words.
  • The last layer gives one score per vocabulary token (50,257 for GPT-2). Raw scores are called logits; softmax turns them into a probability distribution.
import math
vocab = ["chai", "coffee", "cricket", "neha", "."]
scores = [3.1, 2.0, 0.4, -0.5, 1.2]   # pretend output of a Transformer for "neha drinks"

e = [math.exp(s) for s in scores]
probs = [x / sum(e) for x in e]       # softmax, episode 14
for w, p in sorted(zip(vocab, probs), key=lambda t: -t[1]):
    print(f"{w:8} {p:.2f}  " + "#" * round(p * 40))
print("total:", round(sum(probs), 2))

Output:

chai     0.63  #########################
coffee   0.21  ########
.        0.09  ####
cricket  0.04  ##
neha     0.02  #
total: 1.0

Training for free: self-supervised learning

A school workbook Training an LLM
Any page of any book: "Raju drinks hot ____" Training text: web pages, books, code
The answer key is just the next word in the book Free labels: the next token
No teacher needed to mark millions of pages Self-supervised learning: labels come from the data
text = "raju drinks hot chai every morning before college"
tokens = text.split()

pairs = [(tokens[:i], tokens[i]) for i in range(1, len(tokens))]
for context, target in pairs:
    print(f"{' '.join(context):45} -> {target}")
print(len(tokens), "tokens gave", len(pairs), "free training examples")

Output:

raju                                          -> drinks
raju drinks                                   -> hot
raju drinks hot                               -> chai
raju drinks hot chai                          -> every
raju drinks hot chai every                    -> morning
raju drinks hot chai every morning            -> before
raju drinks hot chai every morning before     -> college
8 tokens gave 7 free training examples

A text of N tokens gives N − 1 next-token examples. Thanks to the causal mask (episode 14), a Transformer learns all of them in one pass: each position may only look at the tokens before it.

The loss: −log(p)

import math
# the loss for one prediction: -log(probability given to the right token)
for p in [0.9, 0.5, 0.1, 0.01]:
    print(f"model gave the right token {p:4}:  loss = {-math.log(p):.2f}")

Output:

model gave the right token  0.9:  loss = 0.11
model gave the right token  0.5:  loss = 0.69
model gave the right token  0.1:  loss = 2.30
model gave the right token 0.01:  loss = 4.61

The loss for one prediction is minus the log of the probability the model gave to the right token. Confident and right → near 0; confident and wrong → large. Averaged over many tokens this is the cross-entropy loss. Gradient descent (episode 7) then nudges every weight to lower it, so right tokens get more probability next time.

Pretraining at scale: GPT-3

Fact (Brown et al., OpenAI, 2020, arXiv 2005.14165) Value
Type "an autoregressive language model"
Parameters 175 billion (≈ 350 GB at 2 bytes each)
Training data 300 billion tokens
Transformer blocks 96 layers, d_model 12,288, 96 attention heads
Context window 2,048 tokens

Training a model on a huge pile of general text with next-token prediction is pretraining. For comparison, GPT-2 small has 12 layers, 768 numbers per token, 12 heads and a 1,024-token context.

Why "large"? And how it writes

  • Large = billions of parameters, trained on hundreds of billions of tokens, using huge compute (thousands of GPUs, episode 11).
  • When you chat, it runs our loop: prompt → probabilities → pick a token → append → repeat until done. Generating one token at a time, each fed back as input, is autoregressive generation.

What an LLM is not

Myth Reality
A database of the web patterns squeezed into weights; no stored copy of documents
It looks things up not unless you give it tools (episode 28) or documents (RAG, episode 24)
An assistant by default a pretrained base model only continues text; ask a question and it may write more questions (episode 17)
Always right it predicts likely text, not checked facts (episode 19)

Glossary

Term Meaning Example
Language model predicts the next token from the tokens so far phone keyboard suggestions
Context the tokens the model is given "raju drinks"
Next-token prediction the model's one job "drinks" → chai
Probability a chance from 0 to 1 chai 0.67
Probability distribution a probability for every option, adding to 1 chai 0.63, coffee 0.21, …
Bigram model predicts from the previous word only our counting model
n-gram model predicts from the previous n − 1 words the two-word version
Generation producing text token by token predict, append, repeat
Greedy decoding always pick the most likely token most_common(1)
Sparsity most possible contexts never appear in any text 2,525,766,049 two-word contexts
Generalize make good guesses on unseen inputs "raju plays" like "neha plays"
Logits raw scores before softmax [3.1, 2.0, 0.4, -0.5, 1.2]
Softmax scores → probabilities (episode 14) math.exp(s) / sum
Self-supervised learning labels come from the data itself the next word is the answer
Training example a (context, next token) pair "raju drinks hot" → chai
Loss how wrong a prediction is −log(0.9) = 0.11
Cross-entropy the average −log(p) over many tokens the LLM training loss
Pretraining training on huge general text by next-token prediction GPT-3: 300 billion tokens
Parameters the learned numbers (weights) GPT-3: 175 billion
LLM large language model: billions of parameters GPT-3
Autoregressive generation each new token is fed back as input ChatGPT writing a reply
Base model a model straight out of pretraining continues text, not an assistant
Context window how many tokens it can look back over GPT-3: 2,048
End-of-text token a special token marking the end GPT-2's <|endoftext|>

Common mistakes

Mistake Code Error
A context the counting model never saw follows[("raju", "plays")].most_common(1)[0] IndexError: list index out of range
A probability of exactly 0 in the loss -math.log(0.0) ValueError: math domain error
Asking what follows the last word max(follows["college"], key=...) ValueError: max() iterable argument is empty
Greedy decoding for long text most_common(1) in a loop no error; it loops forever
Thinking more n-gram context fixes it 3-word contexts no error; even sparser, more unseen contexts

Fixes: use a model that generalizes (or back off to a shorter context); never hard-code a 0 chance (softmax never gives exactly 0); add an end-of-text token; sample instead of greedy (episode 16).

Mini project: a "Raju story" generator

Collect 30 sentences about your own day, build the two-word counting model, and generate from every two-word context you can. Then fall back to the one-word model whenever a two-word context is unseen (this trick is called back-off), so it never crashes.

Practice

  1. Add five sentences to the counting model and generate again. Does the loop change?
  2. Generate from "neha" instead of "raju".
  3. How many training examples does a 20-word sentence give? Check it in code.
  4. Compute the loss when the model gives the right token p = 0.25.
Solutions

1.

from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju .
raju drinks lassi after cricket . neha reads a book . raju reads the news .
neha drinks water after lunch . raju plays chess with neha ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    follows[a][b] += 1

word, story = "raju", ["raju"]
for _ in range(15):                   # predict, append, repeat
    word = follows[word].most_common(1)[0][0]
    story.append(word)
print(" ".join(story))
raju drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .

Still a loop: "drinks" → chai and "." → neha are still the most common next words. More text doesn't fix greedy decoding with one word of memory.

2.

from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    follows[a][b] += 1

word, story = "neha", ["neha"]
for _ in range(15):                   # predict, append, repeat
    word = follows[word].most_common(1)[0][0]
    story.append(word)
print(" ".join(story))
neha drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .

3.

text = "on monday raju woke up made masala chai for neha and then took the bus to his college in pune"
tokens = text.split()
pairs = [(tokens[:i], tokens[i]) for i in range(1, len(tokens))]
print(len(tokens), "tokens gave", len(pairs), "free training examples")
20 tokens gave 19 free training examples

4.

import math
print(f"loss when p = 0.25: {-math.log(0.25):.2f}")
loss when p = 0.25: 1.39

Quick quiz 1

One step of an LLM, given "neha drinks". What comes out? A: the whole answer at once · B: one probability per vocabulary token · C: a web search result · D: the single word "chai"

Answer

B. One step gives a probability distribution over the whole vocabulary; one token is then picked and the loop repeats.

Quick quiz 2

A training text of 1,000 tokens gives how many next-token examples? A: 1 · B: 500 · C: 999 · D: 1,000,000

Answer

C: 999. Every token after the first is the answer to the tokens before it.

Interview questions

  1. What is a language model? A model that assigns probabilities to the next token given the previous tokens; generation repeats predict → sample → append.
  2. Why did n-gram models give way to neural language models? Sparsity: most contexts are never seen, and counts can't generalize; embeddings let similar contexts share knowledge, and attention handles long contexts.
  3. What does "self-supervised" mean for LLMs? The labels (next tokens) come from the raw text itself; no human labelling.
  4. What loss do LLMs train with? Cross-entropy: the average of −log(probability of the correct next token).
  5. What is autoregressive generation? Generating one token at a time, feeding each output back as input.
  6. What is a base model? A pretrained model that only continues text; instruction tuning and RLHF turn it into an assistant.
  7. Does an LLM look facts up? No; it predicts likely text from patterns in its weights, unless given tools or retrieved documents.

Summary

  • A language model predicts the next token; generation = predict, append, repeat.
  • A counting (bigram) model works but greedy decoding loops forever.
  • More context makes counting sparse and it can't generalize.
  • A Transformer outputs a probability distribution over the vocabulary (logits → softmax).
  • Self-supervised: the text is its own answer key; loss = −log(p) (cross-entropy).
  • GPT-3: 175 billion parameters, 300 billion tokens, 96 layers, 2,048-token context.
  • Next episode: temperature, top-p and the context window.