Machine Learning Full Course · Episode 75. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output. The data is Tiny Shakespeare (input.txt from the data/tinyshakespeare folder of Andrej Karpathy's char-rnn repository), 1,115,394 characters, saved as shakespeare.txt.
The big idea: self-attention (Episode 74) treats a sentence as a set: "dog" got the same vector in "dog bites man" and "man bites dog". Positional encodings add "I am position p" (learned, sinusoidal) or rotate queries and keys by position (RoPE). Stacking many attention layers makes every word the same vector and kills the gradient; residual connections (x + sublayer(x)) and layer norm fix that. A transformer block = multi-head attention (words talk) + a position-wise feed-forward network (each word thinks), each wrapped in norm + residual. Stack blocks with a causal mask and next-token training and you have a GPT: ours, 818,241 weights trained for 5 minutes on a CPU, writes Shakespeare-like text (validation loss 4.37 → 1.66).
Sources: Vaswani et al., "Attention Is All You Need", 2017 (ar5iv: sinusoidal encodings PE(pos, 2i) = sin(pos/10000^(2i/d_model)), PE(pos, 2i+1) = cos(…), chosen because they may let the model extrapolate to longer sequences; learned positional embeddings gave nearly identical results; each sub-layer is LayerNorm(x + Sublayer(x)); FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ with d_ff = 2048, d_model = 512; N = 6 layers). Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021 (arXiv 2104.09864: encodes the absolute position with a rotation matrix and builds explicit relative position dependency into self-attention). Ruibin Xiong et al., "On Layer Normalization in the Transformer Architecture", 2020 (arXiv 2002.04745: in Post-LN, gradients near the output layer are large at initialisation, which is why warm-up is needed; Pre-LN can be trained without warm-up). Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, BERT, 11 October 2018 (summary of arXiv 1810.04805: Bidirectional Encoder Representations from Transformers; BERT-base has 12 layers, hidden size 768, 12 heads, 110M parameters; masked language model). Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI 2018 (PDF: a 12-layer decoder-only transformer with masked self-attention, 768 dimensions, 12 heads, pre-trained on BooksCorpus, over 7,000 unpublished books). Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (arXiv 1910.10683, ar5iv: the "Text-to-Text Transfer Transformer" (T5), a standard encoder-decoder transformer that treats every text problem as text-to-text).
Words you need first
- Self-attention, causal mask, multi-head attention (Episode 74).
- Embedding /
nn.Embedding(Episode 72): a trainable lookup table, one row per id. - Residual (skip) connection (Episode 66, ResNet): add a layer's input to its output,
x + f(x). - Layer normalisation (Episode 63): rescale each example's vector to mean 0 and standard deviation 1, then apply a learned scale and shift.
- Warm-up (Episode 62): start training with a small learning rate and raise it over the first steps.
- AdamW (Episode 61): Adam with decoupled weight decay.
- Cross-entropy and logits (Episode 23): logits are raw scores; cross-entropy is low when the right class gets high probability.
The story: self-attention ignores word order
import torch, torch.nn as nn
torch.manual_seed(75)
vocab = {"dog": 0, "bites": 1, "man": 2}
emb, attn = nn.Embedding(3, 8), nn.MultiheadAttention(8, 2, batch_first=True)
pos = nn.Embedding(3, 8) # one learned vector per position
def encode(sentence, positions=False):
x = emb(torch.tensor([[vocab[w] for w in sentence.split()]]))
if positions: x = x + pos(torch.arange(3)) # add "I am word 0, 1, 2"
return attn(x, x, x)[0][0] # self-attention
a, b = encode("dog bites man"), encode("man bites dog")
print("'dog' in 'dog bites man':", a[0, :4].detach().numpy().round(3))
print("'dog' in 'man bites dog':", b[2, :4].detach().numpy().round(3))
print(f"no positions : max difference {(a[0] - b[2]).abs().max():.1e}")
print("same vectors, just reordered:", torch.allclose(a, b[[2, 1, 0]], atol=1e-6))
a, b = encode("dog bites man", True), encode("man bites dog", True)
print(f"with positions: max difference {(a[0] - b[2]).abs().max():.1e}")
'dog' in 'dog bites man': [ 0.029 -0.256 0.403 -0.436]
'dog' in 'man bites dog': [ 0.029 -0.256 0.403 -0.436]
no positions : max difference 3.0e-08
same vectors, just reordered: True
with positions: max difference 2.9e-01
Without positions, "dog" gets the same vector in both sentences (the difference 3.0e-08 is float rounding) and the output is the same set of vectors, just reordered. Self-attention is permutation-equivariant: shuffle the input, and the output is shuffled the same way, nothing more. Adding one learned vector per position (pos) makes the two "dog" vectors differ (0.29).
Real-life example: answer sheets with page numbers
| In the exam hall | In a transformer |
|---|---|
| Loose answer pages on the floor: all there, order unknown | Attention alone: a set of words |
| A page number written on every page | Positional encoding: "I am position p" added to each word |
Three kinds of positional encoding
| Kind | How | Used in |
|---|---|---|
| Learned | nn.Embedding(max_len, d): one trainable vector per position, added to the token embedding |
GPT models, BERT, our tiny GPT |
| Sinusoidal | fixed sine and cosine waves of many speeds, added to the token embedding | the original transformer |
| RoPE (rotary) | rotate each pair of numbers of every query and key by position × θ_i |
many modern language models |
Sinusoidal (d = 16, 50 positions):
import torch
def sinusoidal(n_pos, d): # Vaswani et al. 2017
pos = torch.arange(n_pos)[:, None]
i = torch.arange(0, d, 2)
angle = pos / 10000 ** (i / d)
pe = torch.zeros(n_pos, d)
pe[:, 0::2], pe[:, 1::2] = torch.sin(angle), torch.cos(angle)
return pe
pe = sinusoidal(50, 16)
print("position 0:", pe[0, :6].numpy().round(2))
print("position 1:", pe[1, :6].numpy().round(2))
print("position 2:", pe[2, :6].numpy().round(2))
for far in [0, 1, 2, 5, 20]:
sim = (pe[10] @ pe[10 + far]).item()
print(f"position 10 vs position {10 + far:2}: similarity {sim:.2f}")
position 0: [0. 1. 0. 1. 0. 1.]
position 1: [0.84 0.54 0.31 0.95 0.1 1. ]
position 2: [ 0.91 -0.42 0.59 0.81 0.2 0.98]
position 10 vs position 10: similarity 8.00
position 10 vs position 11: similarity 7.49
position 10 vs position 12: similarity 6.37
position 10 vs position 15: similarity 6.14
position 10 vs position 30: similarity 5.78
Each position has its own pattern, like the hands of a clock moving at different speeds, and nearby positions are more similar (8.0 → 7.49 → 6.37 → … → 5.78).
RoPE: rotating pairs of numbers by an angle proportional to the position:
import torch
def rope(x, pos): # rotate each pair of numbers by pos * theta_i
d = x.shape[-1]
theta = 10000 ** (-torch.arange(0, d, 2) / d)
c, s = torch.cos(pos * theta), torch.sin(pos * theta)
x1, x2 = x[0::2], x[1::2]
return torch.stack([x1 * c - x2 * s, x1 * s + x2 * c], -1).flatten()
torch.manual_seed(75)
q, k = torch.randn(8), torch.randn(8) # one query, one key
for qpos, kpos in [(3, 1), (10, 8), (100, 98), (10, 1)]:
score = rope(q, qpos) @ rope(k, kpos)
print(f"query at {qpos:3}, key at {kpos:3} (distance {qpos - kpos}): score {score:.4f}")
print("length kept:", round(q.norm().item(), 4), "->", round(rope(q, 57).norm().item(), 4))
query at 3, key at 1 (distance 2): score -5.0441
query at 10, key at 8 (distance 2): score -5.0441
query at 100, key at 98 (distance 2): score -5.0441
query at 10, key at 1 (distance 9): score -4.9381
length kept: 2.9222 -> 2.9222
The score is −5.0441 whenever the query is 2 positions after the key, wherever they are: after rotation, the dot product depends only on the distance (relative position). Rotations keep vector lengths (2.9222 → 2.9222).
Going deep: stacks collapse
import torch, torch.nn as nn
torch.manual_seed(75)
d, depth = 64, 24
attn = [nn.MultiheadAttention(d, 4, batch_first=True) for _ in range(depth)]
norm = [nn.LayerNorm(d) for _ in range(depth)]
def run(residual):
x0 = torch.randn(1, 10, d, requires_grad=True); x = x0 # 10 words
for a, n in zip(attn, norm):
if residual: h = n(x); x = x + a(h, h, h)[0] # pre-norm + residual
else: x = a(x, x, x)[0] # plain stacking
v = nn.functional.normalize(x[0], dim=-1)
same = (v @ v.T)[~torch.eye(10, dtype=bool)].mean() # word-vs-word cosine
x.pow(2).sum().backward()
print(f"residual={residual!s:5}: words alike {same:.3f}, "
f"input gradient {x0.grad.norm():.1e}")
run(False); run(True)
residual=False: words alike 1.000, input gradient 2.7e-17
residual=True : words alike 0.464, input gradient 3.3e+02
Plain stacking of 24 attention layers: every word ends up the same vector (cosine 1.000), because each layer is a weighted average and averages of averages blur together, and the gradient at the input is 2.7e-17 (vanishing gradient: nothing to learn from). With pre-norm + residual (line 8), the words stay different (0.464) and the gradient reaches the input.
Real-life example: the telephone game
24 friends whisper "meet at 5 near the temple" down a line, and it comes out as nonsense: each copy loses a little. If the original note is passed along and each friend only adds a correction, nothing is lost. That is a residual connection.
Residual connections, layer norm, pre-norm vs post-norm
residual: x = x + sublayer(x)
post-norm: x = LayerNorm(x + sublayer(x)) # original 2017 transformer
pre-norm: x = x + sublayer(LayerNorm(x)) # most models today, and ours
- Residual connection: the input passes straight through; each layer only adds a change; gradients flow back along the
+. - Layer norm: for each word's vector, subtract the mean, divide by the standard deviation, then multiply by a learned scale and add a learned shift. It keeps numbers in a healthy range.
- Post-norm vs pre-norm: Xiong et al. (2020) showed that post-norm has large gradients near the output at the start of training, so it needs learning-rate warm-up; pre-norm keeps the residual path clean and can train without warm-up.
The transformer block
| In a study group | In a transformer block |
|---|---|
| Everyone discusses together | Multi-head attention: words share information |
| Then each one thinks alone and writes notes | Feed-forward network: the same small MLP on every word |
x ─► LayerNorm ─► Multi-head attention ─► (+ x) ─► LayerNorm ─► Feed-forward ─► (+) ─► next block
- Position-wise feed-forward network (FFN):
Linear(d → 4d) → activation → Linear(4d → d), applied to each position separately with the same weights. The original used ReLU with d = 512 and an inner size of 2048; GPT models use GELU (Gaussian Error Linear Unit), a smooth version of ReLU. - A transformer = token embeddings + positional encodings → N blocks → final LayerNorm → a linear head that scores every token in the vocabulary.
A tiny GPT, from scratch
gpt.py, the model:
import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(75)
text = open("shakespeare.txt").read() # Tiny Shakespeare, 1,115,394 characters
chars = sorted(set(text)); V = len(chars) # character-level vocabulary
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
train, val = data[:int(0.9 * len(data))], data[int(0.9 * len(data)):]
T, D, HEADS, LAYERS = 64, 128, 4, 4 # context, width, heads, blocks
class Block(nn.Module): # one pre-norm transformer block
def __init__(self):
super().__init__()
self.ln1, self.ln2 = nn.LayerNorm(D), nn.LayerNorm(D)
self.attn = nn.MultiheadAttention(D, HEADS, batch_first=True)
self.ffn = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
def forward(self, x, mask):
h = self.ln1(x); x = x + self.attn(h, h, h, attn_mask=mask)[0] # talk to others
return x + self.ffn(self.ln2(x)) # think alone
class GPT(nn.Module):
def __init__(self, causal=True):
super().__init__(); self.causal = causal
self.tok, self.pos = nn.Embedding(V, D), nn.Embedding(T, D) # learned positions
self.blocks = nn.ModuleList(Block() for _ in range(LAYERS))
self.ln, self.head = nn.LayerNorm(D), nn.Linear(D, V)
def forward(self, idx):
n = idx.shape[1]; x = self.tok(idx) + self.pos(torch.arange(n))
mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1) if self.causal else None
for b in self.blocks: x = b(x, mask)
return self.head(self.ln(x)) # next-character scores
train.py, the training:
import sys, time
from gpt import *
def batch(split, n=32):
d = train if split == "train" else val
i = torch.randint(len(d) - T - 1, (n,))
x = torch.stack([d[j:j + T] for j in i]); y = torch.stack([d[j + 1:j + T + 1] for j in i])
return x, y # y = x shifted by one character
@torch.no_grad()
def generate(m, n=200):
idx = torch.tensor([[stoi["\n"]]])
for _ in range(n):
p = F.softmax(m(idx[:, -T:])[0, -1], -1) # probabilities of the next char
idx = torch.cat([idx, torch.multinomial(p, 1)[None]], 1)
return "".join(chars[i] for i in idx[0, 1:])
@torch.no_grad()
def val_loss(m):
x, y = batch("val", 256); return F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)).item()
causal = sys.argv[1:] != ["nomask"]; steps = 2000 if causal else 300
m = GPT(causal); opt = torch.optim.AdamW(m.parameters(), 1e-3)
print(f"{len(text):,} characters, {V} different, {sum(p.numel() for p in m.parameters()):,} weights")
print("before training:", repr(generate(m, 60)))
t0 = time.time()
for step in range(steps + 1):
if step % (steps // 4) == 0:
tl = f"{loss.item():.2f}" if step else " - "
print(f"step {step:4}: train loss {tl} val loss {val_loss(m):.2f}")
x, y = batch("train"); loss = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1))
opt.zero_grad(); loss.backward(); opt.step()
print(f"trained in {time.time() - t0:.0f} s; after training:\n" + generate(m, 300))
Output of python3 train.py:
1,115,394 characters, 65 different, 818,241 weights
before training: "bS'WTvpRfMLdaJ'I?Dedu;cUO,ckRXpPYaNgL;ApU td&,vwJfRn\nlJIGbCl"
step 0: train loss - val loss 4.37
step 500: train loss 1.90 val loss 2.01
step 1000: train loss 1.70 val loss 1.82
step 1500: train loss 1.57 val loss 1.70
step 2000: train loss 1.50 val loss 1.66
trained in 296 s; after training:
THARD III:
What love shall the dinheness! O pail Peace
than you, by unished ill doubl. Brareat
The tiditen
The kings, and pape not do comfolled,
I wilsometh the mater Malliamentlershis worthing; and with thou
shout advish do and to be redeeport
To knows;
Whem I'll son that hath bodes.
Were cunk of M
- Language model: predicts the next token from the tokens before it. Character-level: the tokens are single characters (65 different ones here).
- Targets: the input shifted by one (
y = x[1:]), so the text is its own answer key: no labels needed (self-supervised learning). - Thanks to the causal mask, all 64 positions are trained at once.
- Sampling:
torch.multinomialpicks the next character at random according to the predicted probabilities, so each run writes different text. - Validation loss (on the last 10% of the text, never trained on): 4.37 → 1.66. Guessing uniformly among 65 characters scores ln 65 ≈ 4.17. Training and validation losses stay close (1.50 vs 1.66), so it is not just memorising.
- After 5 minutes it already writes a speaker name in capitals with a colon, verse-like line breaks and real words (love, peace, kings, thou) mixed with invented ones.
Mistake: no causal mask, so the model cheats
python3 train.py nomask builds GPT(causal=False) and trains 300 steps:
1,115,394 characters, 65 different, 818,241 weights
before training: "bS'WTvpRfMLdaJ'I?Dedu;cUO,ckRXpPYaNgL;ApU tdh,vwJfRn\nlJIGbCl"
step 0: train loss - val loss 4.38
step 75: train loss 2.48 val loss 2.45
step 150: train loss 0.15 val loss 0.14
step 225: train loss 0.06 val loss 0.06
step 300: train loss 0.05 val loss 0.05
trained in 181 s; after training:
(The output goes on with blank lines: the generated text is 300 newline characters and nothing else.)
The loss drops to 0.05, far "better" than the real model's 1.66, and generation produces 300 newline characters and nothing else. Without the mask every position can see the next character, the very answer it must predict, so it learns to copy it. At generation time the next character doesn't exist yet, so there is nothing to copy. A loss that looks too good to be true usually is.
Three families of transformers
| Family | Mask | Trained to | Example | Good for |
|---|---|---|---|---|
| Encoder-only | none: every token sees both sides | fill in masked-out tokens (masked language modelling) | BERT (Devlin et al., 2018): BERT-base 12 layers, 768, 12 heads, 110M | understanding: classification, search, tagging |
| Decoder-only | causal | predict the next token | GPT (Radford et al., OpenAI 2018): 12 layers, 768, 12 heads; our tiny GPT | writing: generation, chat |
| Encoder-decoder | encoder none, decoder causal + cross-attention to the encoder | text in → text out | T5 (Raffel et al.), Text-to-Text Transfer Transformer | translation, summarising |
Cross-attention is Episode 73's attention: decoder queries, encoder keys and values.
Common mistakes
import torch, torch.nn as nn
T = 64 # the model was built for 64 positions
pos = nn.Embedding(T, 128) # learned positional embeddings: rows 0..63
idx = torch.zeros(1, 65, dtype=torch.long) # generation kept growing: 65 characters
try:
pos(torch.arange(idx.shape[1]))
except IndexError as e:
print("IndexError:", e)
print("fix: crop the context ->", tuple(pos(torch.arange(idx[:, -T:].shape[1])).shape))
IndexError: index out of range in self
fix: crop the context -> (64, 128)
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Context longer than the position table | IndexError: index out of range in self |
nn.Embedding(64, d) has rows 0-63 |
crop: idx[:, -T:] |
| No causal mask in a GPT | val loss 0.05, then 300 newlines | each position copies the next character | torch.triu(..., diagonal=1) mask |
| No positional encoding | dog bites man = man bites dog | attention is order-blind | learned, sinusoidal or RoPE |
| Deep stack without residuals | cosine 1.000, gradient 2.7e-17 | averages of averages, vanishing gradient | x = x + sublayer(norm(x)) |
| Post-norm, large learning rate, no warm-up | unstable early training | large gradients at the start | pre-norm, or warm-up |
d_model not divisible by heads |
AssertionError: embed_dim must be divisible by num_heads |
heads must split evenly | e.g. 128 and 4 |
Mini project: where are the weights?
from gpt import *
m = GPT()
parts = {"token embeddings": m.tok, "position embeddings": m.pos, "one block: attention": m.blocks[0].attn,
"one block: feed-forward": m.blocks[0].ffn, "one block: 2 layer norms": nn.ModuleList([m.blocks[0].ln1, m.blocks[0].ln2]),
"final norm + head": nn.ModuleList([m.ln, m.head])}
for name, p in parts.items():
print(f"{name:26} {sum(x.numel() for x in p.parameters()):8,}")
print(f"{'whole model (4 blocks)':26} {sum(x.numel() for x in m.parameters()):8,}")
x = torch.randint(V, (2, 10)); print("input", tuple(x.shape), "-> output", tuple(m(x).shape), "(batch, characters, scores per character)")
token embeddings 8,320
position embeddings 8,192
one block: attention 66,048
one block: feed-forward 131,712
one block: 2 layer norms 512
final norm + head 8,641
whole model (4 blocks) 818,241
input (2, 10) -> output (2, 10, 65) (batch, characters, scores per character)
The feed-forward network holds two thirds of each block's weights (131,712 of 198,272): 128×512 + 512 + 512×128 + 128. Attention has 4 × (128×128 + 128) = 66,048.
Practice
1. Swap learned positions for sinusoidal ones, and remove the residual connections. Compare validation loss after 500 steps.
Solution
import sys; sys.argv = ["train.py"]
import gpt
from gpt import *
def sinusoidal(n_pos, d):
pos = torch.arange(n_pos)[:, None]; angle = pos / 10000 ** (torch.arange(0, d, 2) / d)
pe = torch.zeros(n_pos, d); pe[:, 0::2], pe[:, 1::2] = torch.sin(angle), torch.cos(angle); return pe
class SinGPT(GPT): # 1. fixed sinusoidal positions
def __init__(self):
super().__init__(); del self.pos; self.register_buffer("pe", sinusoidal(T, D))
def forward(self, idx):
n = idx.shape[1]; x = self.tok(idx) + self.pe[:n]
mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
for b in self.blocks: x = b(x, mask)
return self.head(self.ln(x))
class PlainBlock(Block): # 2. no residual connections
def forward(self, x, mask):
x = self.attn(self.ln1(x), self.ln1(x), self.ln1(x), attn_mask=mask)[0]
return self.ffn(self.ln2(x))
def batch(d, n=32):
i = torch.randint(len(d) - T - 1, (n,))
return torch.stack([d[j:j + T] for j in i]), torch.stack([d[j + 1:j + T + 1] for j in i])
def run(name, m, steps=500):
torch.manual_seed(75); opt = torch.optim.AdamW(m.parameters(), 1e-3)
for _ in range(steps):
x, y = batch(train); loss = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1))
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
x, y = batch(val, 256); vl = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)).item()
print(f"{name:28} val loss after {steps} steps: {vl:.2f}")
torch.manual_seed(75); run("learned positions (ours)", GPT())
torch.manual_seed(75); run("sinusoidal positions", SinGPT())
torch.manual_seed(75); m = GPT(); m.blocks = nn.ModuleList(PlainBlock() for _ in range(LAYERS)); run("no residual connections", m)
learned positions (ours) val loss after 500 steps: 2.04
sinusoidal positions val loss after 500 steps: 1.99
no residual connections val loss after 500 steps: 3.39
Sinusoidal positions do as well as learned ones (1.99 vs 2.04), as the original paper found. Without residual connections the same 4-block model is far behind after 500 steps (3.39).
2. Train with 8 blocks instead of 4. Hint
Set LAYERS = 8: about 1.6M weights and twice the time per step. Compare the validation loss at the same number of steps.
3. Train it on your own text. Hint
Replace shakespeare.txt with any plain-text file of at least a few hundred thousand characters; the vocabulary is built from whatever characters it contains.
4. Why does RoPE give relative positions for free? Answer
Rotating q by angle a and k by angle b makes their dot product depend only on a − b, i.e. on the distance between positions.
Quick quiz
Task: read a customer review and say positive or negative; every word should see the words on both sides. Which family?
Answer
Encoder-only (BERT): no causal mask, so each token sees both sides. A decoder-only model can be used too, but each token only looks left.
Interview questions
- Why does a transformer need positional encoding? Self-attention is permutation-equivariant: it sees a set, not a sequence.
- Learned vs sinusoidal vs RoPE? Learned: trainable but limited to the trained length; sinusoidal: fixed, may extrapolate; RoPE: rotates q and k so scores depend on relative distance.
- What do residual connections and layer norm do in a transformer? Keep information and gradients flowing through deep stacks and keep activations in a stable range.
- Pre-norm vs post-norm? Pre-norm normalises each sublayer's input and keeps a clean residual path; it is more stable and needs less warm-up.
- What does the feed-forward network add? Per-token non-linear processing; attention only mixes tokens (a weighted average).
- Encoder-only vs decoder-only vs encoder-decoder? BERT-style understanding, GPT-style generation, T5-style text-to-text.
- Why is a too-low training loss suspicious in a language model? It often means information leakage, e.g. a missing causal mask.
Glossary
| Term | Meaning, with an example |
|---|---|
| Transformer | A network of stacked attention + feed-forward blocks, with no recurrence |
| Transformer block | LayerNorm → attention → +residual → LayerNorm → FFN → +residual (pre-norm) |
| Positional encoding | A way to give each token its position; added vector or rotation |
| Learned positional embedding | nn.Embedding(max_len, d), trained like any weight |
| Sinusoidal positional encoding | Fixed sin/cos waves: PE(p, 2i) = sin(p/10000^(2i/d)) |
| RoPE (rotary position embedding) | Rotate pairs of query/key numbers by position × θ; scores depend on distance |
| Relative position | The distance between two tokens (2), not where they sit (3 and 1) |
| Permutation-equivariant | Shuffling inputs only shuffles outputs; attention without positions |
| Residual (skip) connection | x + sublayer(x): keep the input, add a change |
| Layer normalisation | Per-token rescale to mean 0, std 1, then learned scale and shift |
| Post-norm | LayerNorm(x + sublayer(x)) (original 2017 design) |
| Pre-norm | x + sublayer(LayerNorm(x)) (most current models) |
| Warm-up | A small learning rate that grows over the first steps |
| Vanishing gradient | Gradients that shrink to ~0 before reaching early layers; 2.7e-17 |
| Position-wise feed-forward network | The same 2-layer MLP (d → 4d → d) applied to each token |
| GELU | Gaussian Error Linear Unit, a smooth ReLU used in GPT models |
| Language model | Predicts the next token from the previous ones |
| Character-level model | Tokens are single characters; 65 in Tiny Shakespeare |
| Validation loss | Loss on text the model never trained on |
Sampling (torch.multinomial) |
Drawing the next token at random from the predicted probabilities |
| Encoder-only (BERT) | No causal mask; trained by masked language modelling |
| Masked language modelling | Hide some tokens and train the model to fill them in |
| Decoder-only (GPT) | Causal mask; trained by next-token prediction |
| Encoder-decoder (T5) | Encoder + decoder with cross-attention; text in, text out |
| Cross-attention | Decoder queries attend to encoder keys and values |
Summary
- Self-attention is order-blind; positions fix it: learned, sinusoidal, or RoPE (relative distance for free).
- Deep plain stacks collapse (cosine 1.000, gradient 2.7e-17); residuals + layer norm (pre-norm) keep them alive.
- Block = attention (talk) + feed-forward (think), each with norm and residual.
- BERT (encoder-only), GPT (decoder-only), T5 (encoder-decoder) are the same blocks with different masks and training.
- Our 818,241-weight GPT trained 5 minutes on Tiny Shakespeare: validation loss 4.37 → 1.66; without the causal mask it "scores" 0.05 and writes only newlines.
Next: Episode 76, pretrained transformers with Hugging Face: tokenizers, pipelines, fine-tuning BERT, and the Vision Transformer.