SyntaxPilot
Courses/Machine Learning/Lesson 75
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 75 of 90

Transformers from scratch

transformer blockpositional encoding (sinusoidal, learned, RoPE)residual connections with layer normpre-norm vs post-normposition-wise feed-forward networkencoder-only (BERT)decoder-only (GPT)encoder-decoder (T5)building a tiny character-level GPT

Machine Learning Full Course · Episode 75. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output. The data is Tiny Shakespeare (input.txt from the data/tinyshakespeare folder of Andrej Karpathy's char-rnn repository), 1,115,394 characters, saved as shakespeare.txt.

The big idea: self-attention (Episode 74) treats a sentence as a set: "dog" got the same vector in "dog bites man" and "man bites dog". Positional encodings add "I am position p" (learned, sinusoidal) or rotate queries and keys by position (RoPE). Stacking many attention layers makes every word the same vector and kills the gradient; residual connections (x + sublayer(x)) and layer norm fix that. A transformer block = multi-head attention (words talk) + a position-wise feed-forward network (each word thinks), each wrapped in norm + residual. Stack blocks with a causal mask and next-token training and you have a GPT: ours, 818,241 weights trained for 5 minutes on a CPU, writes Shakespeare-like text (validation loss 4.37 → 1.66).

Sources: Vaswani et al., "Attention Is All You Need", 2017 (ar5iv: sinusoidal encodings PE(pos, 2i) = sin(pos/10000^(2i/d_model)), PE(pos, 2i+1) = cos(…), chosen because they may let the model extrapolate to longer sequences; learned positional embeddings gave nearly identical results; each sub-layer is LayerNorm(x + Sublayer(x)); FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ with d_ff = 2048, d_model = 512; N = 6 layers). Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021 (arXiv 2104.09864: encodes the absolute position with a rotation matrix and builds explicit relative position dependency into self-attention). Ruibin Xiong et al., "On Layer Normalization in the Transformer Architecture", 2020 (arXiv 2002.04745: in Post-LN, gradients near the output layer are large at initialisation, which is why warm-up is needed; Pre-LN can be trained without warm-up). Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, BERT, 11 October 2018 (summary of arXiv 1810.04805: Bidirectional Encoder Representations from Transformers; BERT-base has 12 layers, hidden size 768, 12 heads, 110M parameters; masked language model). Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI 2018 (PDF: a 12-layer decoder-only transformer with masked self-attention, 768 dimensions, 12 heads, pre-trained on BooksCorpus, over 7,000 unpublished books). Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (arXiv 1910.10683, ar5iv: the "Text-to-Text Transfer Transformer" (T5), a standard encoder-decoder transformer that treats every text problem as text-to-text).

Words you need first

  • Self-attention, causal mask, multi-head attention (Episode 74).
  • Embedding / nn.Embedding (Episode 72): a trainable lookup table, one row per id.
  • Residual (skip) connection (Episode 66, ResNet): add a layer's input to its output, x + f(x).
  • Layer normalisation (Episode 63): rescale each example's vector to mean 0 and standard deviation 1, then apply a learned scale and shift.
  • Warm-up (Episode 62): start training with a small learning rate and raise it over the first steps.
  • AdamW (Episode 61): Adam with decoupled weight decay.
  • Cross-entropy and logits (Episode 23): logits are raw scores; cross-entropy is low when the right class gets high probability.

The story: self-attention ignores word order

import torch, torch.nn as nn
torch.manual_seed(75)
vocab = {"dog": 0, "bites": 1, "man": 2}
emb, attn = nn.Embedding(3, 8), nn.MultiheadAttention(8, 2, batch_first=True)
pos = nn.Embedding(3, 8)                       # one learned vector per position
def encode(sentence, positions=False):
    x = emb(torch.tensor([[vocab[w] for w in sentence.split()]]))
    if positions: x = x + pos(torch.arange(3))  # add "I am word 0, 1, 2"
    return attn(x, x, x)[0][0]                 # self-attention
a, b = encode("dog bites man"), encode("man bites dog")
print("'dog' in 'dog bites man':", a[0, :4].detach().numpy().round(3))
print("'dog' in 'man bites dog':", b[2, :4].detach().numpy().round(3))
print(f"no positions  : max difference {(a[0] - b[2]).abs().max():.1e}")
print("same vectors, just reordered:", torch.allclose(a, b[[2, 1, 0]], atol=1e-6))
a, b = encode("dog bites man", True), encode("man bites dog", True)
print(f"with positions: max difference {(a[0] - b[2]).abs().max():.1e}")
'dog' in 'dog bites man': [ 0.029 -0.256  0.403 -0.436]
'dog' in 'man bites dog': [ 0.029 -0.256  0.403 -0.436]
no positions  : max difference 3.0e-08
same vectors, just reordered: True
with positions: max difference 2.9e-01

Without positions, "dog" gets the same vector in both sentences (the difference 3.0e-08 is float rounding) and the output is the same set of vectors, just reordered. Self-attention is permutation-equivariant: shuffle the input, and the output is shuffled the same way, nothing more. Adding one learned vector per position (pos) makes the two "dog" vectors differ (0.29).

Real-life example: answer sheets with page numbers

In the exam hall In a transformer
Loose answer pages on the floor: all there, order unknown Attention alone: a set of words
A page number written on every page Positional encoding: "I am position p" added to each word

Three kinds of positional encoding

Kind How Used in
Learned nn.Embedding(max_len, d): one trainable vector per position, added to the token embedding GPT models, BERT, our tiny GPT
Sinusoidal fixed sine and cosine waves of many speeds, added to the token embedding the original transformer
RoPE (rotary) rotate each pair of numbers of every query and key by position × θ_i many modern language models

Sinusoidal (d = 16, 50 positions):

import torch
def sinusoidal(n_pos, d):                       # Vaswani et al. 2017
    pos = torch.arange(n_pos)[:, None]
    i = torch.arange(0, d, 2)
    angle = pos / 10000 ** (i / d)
    pe = torch.zeros(n_pos, d)
    pe[:, 0::2], pe[:, 1::2] = torch.sin(angle), torch.cos(angle)
    return pe
pe = sinusoidal(50, 16)
print("position 0:", pe[0, :6].numpy().round(2))
print("position 1:", pe[1, :6].numpy().round(2))
print("position 2:", pe[2, :6].numpy().round(2))
for far in [0, 1, 2, 5, 20]:
    sim = (pe[10] @ pe[10 + far]).item()
    print(f"position 10 vs position {10 + far:2}: similarity {sim:.2f}")
position 0: [0. 1. 0. 1. 0. 1.]
position 1: [0.84 0.54 0.31 0.95 0.1  1.  ]
position 2: [ 0.91 -0.42  0.59  0.81  0.2   0.98]
position 10 vs position 10: similarity 8.00
position 10 vs position 11: similarity 7.49
position 10 vs position 12: similarity 6.37
position 10 vs position 15: similarity 6.14
position 10 vs position 30: similarity 5.78

Each position has its own pattern, like the hands of a clock moving at different speeds, and nearby positions are more similar (8.0 → 7.49 → 6.37 → … → 5.78).

RoPE: rotating pairs of numbers by an angle proportional to the position:

import torch
def rope(x, pos):                     # rotate each pair of numbers by pos * theta_i
    d = x.shape[-1]
    theta = 10000 ** (-torch.arange(0, d, 2) / d)
    c, s = torch.cos(pos * theta), torch.sin(pos * theta)
    x1, x2 = x[0::2], x[1::2]
    return torch.stack([x1 * c - x2 * s, x1 * s + x2 * c], -1).flatten()
torch.manual_seed(75)
q, k = torch.randn(8), torch.randn(8)            # one query, one key
for qpos, kpos in [(3, 1), (10, 8), (100, 98), (10, 1)]:
    score = rope(q, qpos) @ rope(k, kpos)
    print(f"query at {qpos:3}, key at {kpos:3} (distance {qpos - kpos}): score {score:.4f}")
print("length kept:", round(q.norm().item(), 4), "->", round(rope(q, 57).norm().item(), 4))
query at   3, key at   1 (distance 2): score -5.0441
query at  10, key at   8 (distance 2): score -5.0441
query at 100, key at  98 (distance 2): score -5.0441
query at  10, key at   1 (distance 9): score -4.9381
length kept: 2.9222 -> 2.9222

The score is −5.0441 whenever the query is 2 positions after the key, wherever they are: after rotation, the dot product depends only on the distance (relative position). Rotations keep vector lengths (2.9222 → 2.9222).

Going deep: stacks collapse

import torch, torch.nn as nn
torch.manual_seed(75)
d, depth = 64, 24
attn = [nn.MultiheadAttention(d, 4, batch_first=True) for _ in range(depth)]
norm = [nn.LayerNorm(d) for _ in range(depth)]
def run(residual):
    x0 = torch.randn(1, 10, d, requires_grad=True); x = x0       # 10 words
    for a, n in zip(attn, norm):
        if residual: h = n(x); x = x + a(h, h, h)[0]             # pre-norm + residual
        else: x = a(x, x, x)[0]                                  # plain stacking
    v = nn.functional.normalize(x[0], dim=-1)
    same = (v @ v.T)[~torch.eye(10, dtype=bool)].mean()         # word-vs-word cosine
    x.pow(2).sum().backward()
    print(f"residual={residual!s:5}: words alike {same:.3f}, "
          f"input gradient {x0.grad.norm():.1e}")
run(False); run(True)
residual=False: words alike 1.000, input gradient 2.7e-17
residual=True : words alike 0.464, input gradient 3.3e+02

Plain stacking of 24 attention layers: every word ends up the same vector (cosine 1.000), because each layer is a weighted average and averages of averages blur together, and the gradient at the input is 2.7e-17 (vanishing gradient: nothing to learn from). With pre-norm + residual (line 8), the words stay different (0.464) and the gradient reaches the input.

Real-life example: the telephone game

24 friends whisper "meet at 5 near the temple" down a line, and it comes out as nonsense: each copy loses a little. If the original note is passed along and each friend only adds a correction, nothing is lost. That is a residual connection.

Residual connections, layer norm, pre-norm vs post-norm

residual:   x = x + sublayer(x)
post-norm:  x = LayerNorm(x + sublayer(x))        # original 2017 transformer
pre-norm:   x = x + sublayer(LayerNorm(x))        # most models today, and ours
  • Residual connection: the input passes straight through; each layer only adds a change; gradients flow back along the +.
  • Layer norm: for each word's vector, subtract the mean, divide by the standard deviation, then multiply by a learned scale and add a learned shift. It keeps numbers in a healthy range.
  • Post-norm vs pre-norm: Xiong et al. (2020) showed that post-norm has large gradients near the output at the start of training, so it needs learning-rate warm-up; pre-norm keeps the residual path clean and can train without warm-up.

The transformer block

In a study group In a transformer block
Everyone discusses together Multi-head attention: words share information
Then each one thinks alone and writes notes Feed-forward network: the same small MLP on every word
x ─► LayerNorm ─► Multi-head attention ─► (+ x) ─► LayerNorm ─► Feed-forward ─► (+) ─► next block
  • Position-wise feed-forward network (FFN): Linear(d → 4d) → activation → Linear(4d → d), applied to each position separately with the same weights. The original used ReLU with d = 512 and an inner size of 2048; GPT models use GELU (Gaussian Error Linear Unit), a smooth version of ReLU.
  • A transformer = token embeddings + positional encodings → N blocks → final LayerNorm → a linear head that scores every token in the vocabulary.

A tiny GPT, from scratch

gpt.py, the model:

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(75)
text = open("shakespeare.txt").read()                # Tiny Shakespeare, 1,115,394 characters
chars = sorted(set(text)); V = len(chars)              # character-level vocabulary
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
train, val = data[:int(0.9 * len(data))], data[int(0.9 * len(data)):]
T, D, HEADS, LAYERS = 64, 128, 4, 4                    # context, width, heads, blocks
class Block(nn.Module):                                # one pre-norm transformer block
    def __init__(self):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(D), nn.LayerNorm(D)
        self.attn = nn.MultiheadAttention(D, HEADS, batch_first=True)
        self.ffn = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
    def forward(self, x, mask):
        h = self.ln1(x); x = x + self.attn(h, h, h, attn_mask=mask)[0]   # talk to others
        return x + self.ffn(self.ln2(x))                                # think alone
class GPT(nn.Module):
    def __init__(self, causal=True):
        super().__init__(); self.causal = causal
        self.tok, self.pos = nn.Embedding(V, D), nn.Embedding(T, D)    # learned positions
        self.blocks = nn.ModuleList(Block() for _ in range(LAYERS))
        self.ln, self.head = nn.LayerNorm(D), nn.Linear(D, V)
    def forward(self, idx):
        n = idx.shape[1]; x = self.tok(idx) + self.pos(torch.arange(n))
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1) if self.causal else None
        for b in self.blocks: x = b(x, mask)
        return self.head(self.ln(x))                    # next-character scores

train.py, the training:

import sys, time
from gpt import *
def batch(split, n=32):
    d = train if split == "train" else val
    i = torch.randint(len(d) - T - 1, (n,))
    x = torch.stack([d[j:j + T] for j in i]); y = torch.stack([d[j + 1:j + T + 1] for j in i])
    return x, y                                         # y = x shifted by one character
@torch.no_grad()
def generate(m, n=200):
    idx = torch.tensor([[stoi["\n"]]])
    for _ in range(n):
        p = F.softmax(m(idx[:, -T:])[0, -1], -1)        # probabilities of the next char
        idx = torch.cat([idx, torch.multinomial(p, 1)[None]], 1)
    return "".join(chars[i] for i in idx[0, 1:])
@torch.no_grad()
def val_loss(m):
    x, y = batch("val", 256); return F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)).item()
causal = sys.argv[1:] != ["nomask"]; steps = 2000 if causal else 300
m = GPT(causal); opt = torch.optim.AdamW(m.parameters(), 1e-3)
print(f"{len(text):,} characters, {V} different, {sum(p.numel() for p in m.parameters()):,} weights")
print("before training:", repr(generate(m, 60)))
t0 = time.time()
for step in range(steps + 1):
    if step % (steps // 4) == 0:
        tl = f"{loss.item():.2f}" if step else " -  "
        print(f"step {step:4}: train loss {tl}  val loss {val_loss(m):.2f}")
    x, y = batch("train"); loss = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
print(f"trained in {time.time() - t0:.0f} s; after training:\n" + generate(m, 300))

Output of python3 train.py:

1,115,394 characters, 65 different, 818,241 weights
before training: "bS'WTvpRfMLdaJ'I?Dedu;cUO,ckRXpPYaNgL;ApU td&,vwJfRn\nlJIGbCl"
step    0: train loss  -    val loss 4.37
step  500: train loss 1.90  val loss 2.01
step 1000: train loss 1.70  val loss 1.82
step 1500: train loss 1.57  val loss 1.70
step 2000: train loss 1.50  val loss 1.66
trained in 296 s; after training:
THARD III:
What love shall the dinheness! O pail Peace
than you, by unished ill doubl. Brareat
The tiditen
The kings, and pape not do comfolled,
I wilsometh the mater Malliamentlershis worthing; and with thou
shout advish do and to be redeeport
To knows;
Whem I'll son that hath bodes.
Were cunk of M
  • Language model: predicts the next token from the tokens before it. Character-level: the tokens are single characters (65 different ones here).
  • Targets: the input shifted by one (y = x[1:]), so the text is its own answer key: no labels needed (self-supervised learning).
  • Thanks to the causal mask, all 64 positions are trained at once.
  • Sampling: torch.multinomial picks the next character at random according to the predicted probabilities, so each run writes different text.
  • Validation loss (on the last 10% of the text, never trained on): 4.37 → 1.66. Guessing uniformly among 65 characters scores ln 65 ≈ 4.17. Training and validation losses stay close (1.50 vs 1.66), so it is not just memorising.
  • After 5 minutes it already writes a speaker name in capitals with a colon, verse-like line breaks and real words (love, peace, kings, thou) mixed with invented ones.

Mistake: no causal mask, so the model cheats

python3 train.py nomask builds GPT(causal=False) and trains 300 steps:

1,115,394 characters, 65 different, 818,241 weights
before training: "bS'WTvpRfMLdaJ'I?Dedu;cUO,ckRXpPYaNgL;ApU tdh,vwJfRn\nlJIGbCl"
step    0: train loss  -    val loss 4.38
step   75: train loss 2.48  val loss 2.45
step  150: train loss 0.15  val loss 0.14
step  225: train loss 0.06  val loss 0.06
step  300: train loss 0.05  val loss 0.05
trained in 181 s; after training:

(The output goes on with blank lines: the generated text is 300 newline characters and nothing else.)

The loss drops to 0.05, far "better" than the real model's 1.66, and generation produces 300 newline characters and nothing else. Without the mask every position can see the next character, the very answer it must predict, so it learns to copy it. At generation time the next character doesn't exist yet, so there is nothing to copy. A loss that looks too good to be true usually is.

Three families of transformers

Family Mask Trained to Example Good for
Encoder-only none: every token sees both sides fill in masked-out tokens (masked language modelling) BERT (Devlin et al., 2018): BERT-base 12 layers, 768, 12 heads, 110M understanding: classification, search, tagging
Decoder-only causal predict the next token GPT (Radford et al., OpenAI 2018): 12 layers, 768, 12 heads; our tiny GPT writing: generation, chat
Encoder-decoder encoder none, decoder causal + cross-attention to the encoder text in → text out T5 (Raffel et al.), Text-to-Text Transfer Transformer translation, summarising

Cross-attention is Episode 73's attention: decoder queries, encoder keys and values.

Common mistakes

import torch, torch.nn as nn
T = 64                                       # the model was built for 64 positions
pos = nn.Embedding(T, 128)                   # learned positional embeddings: rows 0..63
idx = torch.zeros(1, 65, dtype=torch.long)   # generation kept growing: 65 characters
try:
    pos(torch.arange(idx.shape[1]))
except IndexError as e:
    print("IndexError:", e)
print("fix: crop the context ->", tuple(pos(torch.arange(idx[:, -T:].shape[1])).shape))
IndexError: index out of range in self
fix: crop the context -> (64, 128)
Mistake Exact message / symptom Why Fix
Context longer than the position table IndexError: index out of range in self nn.Embedding(64, d) has rows 0-63 crop: idx[:, -T:]
No causal mask in a GPT val loss 0.05, then 300 newlines each position copies the next character torch.triu(..., diagonal=1) mask
No positional encoding dog bites man = man bites dog attention is order-blind learned, sinusoidal or RoPE
Deep stack without residuals cosine 1.000, gradient 2.7e-17 averages of averages, vanishing gradient x = x + sublayer(norm(x))
Post-norm, large learning rate, no warm-up unstable early training large gradients at the start pre-norm, or warm-up
d_model not divisible by heads AssertionError: embed_dim must be divisible by num_heads heads must split evenly e.g. 128 and 4

Mini project: where are the weights?

from gpt import *
m = GPT()
parts = {"token embeddings": m.tok, "position embeddings": m.pos, "one block: attention": m.blocks[0].attn,
         "one block: feed-forward": m.blocks[0].ffn, "one block: 2 layer norms": nn.ModuleList([m.blocks[0].ln1, m.blocks[0].ln2]),
         "final norm + head": nn.ModuleList([m.ln, m.head])}
for name, p in parts.items():
    print(f"{name:26} {sum(x.numel() for x in p.parameters()):8,}")
print(f"{'whole model (4 blocks)':26} {sum(x.numel() for x in m.parameters()):8,}")
x = torch.randint(V, (2, 10)); print("input", tuple(x.shape), "-> output", tuple(m(x).shape), "(batch, characters, scores per character)")
token embeddings              8,320
position embeddings           8,192
one block: attention         66,048
one block: feed-forward     131,712
one block: 2 layer norms        512
final norm + head             8,641
whole model (4 blocks)      818,241
input (2, 10) -> output (2, 10, 65) (batch, characters, scores per character)

The feed-forward network holds two thirds of each block's weights (131,712 of 198,272): 128×512 + 512 + 512×128 + 128. Attention has 4 × (128×128 + 128) = 66,048.

Practice

1. Swap learned positions for sinusoidal ones, and remove the residual connections. Compare validation loss after 500 steps.

Solution
import sys; sys.argv = ["train.py"]
import gpt
from gpt import *
def sinusoidal(n_pos, d):
    pos = torch.arange(n_pos)[:, None]; angle = pos / 10000 ** (torch.arange(0, d, 2) / d)
    pe = torch.zeros(n_pos, d); pe[:, 0::2], pe[:, 1::2] = torch.sin(angle), torch.cos(angle); return pe
class SinGPT(GPT):                                   # 1. fixed sinusoidal positions
    def __init__(self):
        super().__init__(); del self.pos; self.register_buffer("pe", sinusoidal(T, D))
    def forward(self, idx):
        n = idx.shape[1]; x = self.tok(idx) + self.pe[:n]
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), 1)
        for b in self.blocks: x = b(x, mask)
        return self.head(self.ln(x))
class PlainBlock(Block):                             # 2. no residual connections
    def forward(self, x, mask):
        x = self.attn(self.ln1(x), self.ln1(x), self.ln1(x), attn_mask=mask)[0]
        return self.ffn(self.ln2(x))
def batch(d, n=32):
    i = torch.randint(len(d) - T - 1, (n,))
    return torch.stack([d[j:j + T] for j in i]), torch.stack([d[j + 1:j + T + 1] for j in i])
def run(name, m, steps=500):
    torch.manual_seed(75); opt = torch.optim.AdamW(m.parameters(), 1e-3)
    for _ in range(steps):
        x, y = batch(train); loss = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        x, y = batch(val, 256); vl = F.cross_entropy(m(x).reshape(-1, V), y.reshape(-1)).item()
    print(f"{name:28} val loss after {steps} steps: {vl:.2f}")
torch.manual_seed(75); run("learned positions (ours)", GPT())
torch.manual_seed(75); run("sinusoidal positions", SinGPT())
torch.manual_seed(75); m = GPT(); m.blocks = nn.ModuleList(PlainBlock() for _ in range(LAYERS)); run("no residual connections", m)
learned positions (ours)     val loss after 500 steps: 2.04
sinusoidal positions         val loss after 500 steps: 1.99
no residual connections      val loss after 500 steps: 3.39

Sinusoidal positions do as well as learned ones (1.99 vs 2.04), as the original paper found. Without residual connections the same 4-block model is far behind after 500 steps (3.39).

2. Train with 8 blocks instead of 4.

HintSet LAYERS = 8: about 1.6M weights and twice the time per step. Compare the validation loss at the same number of steps.

3. Train it on your own text.

HintReplace shakespeare.txt with any plain-text file of at least a few hundred thousand characters; the vocabulary is built from whatever characters it contains.

4. Why does RoPE give relative positions for free?

AnswerRotating q by angle a and k by angle b makes their dot product depend only on a − b, i.e. on the distance between positions.

Quick quiz

Task: read a customer review and say positive or negative; every word should see the words on both sides. Which family?

Answer

Encoder-only (BERT): no causal mask, so each token sees both sides. A decoder-only model can be used too, but each token only looks left.

Interview questions

  1. Why does a transformer need positional encoding? Self-attention is permutation-equivariant: it sees a set, not a sequence.
  2. Learned vs sinusoidal vs RoPE? Learned: trainable but limited to the trained length; sinusoidal: fixed, may extrapolate; RoPE: rotates q and k so scores depend on relative distance.
  3. What do residual connections and layer norm do in a transformer? Keep information and gradients flowing through deep stacks and keep activations in a stable range.
  4. Pre-norm vs post-norm? Pre-norm normalises each sublayer's input and keeps a clean residual path; it is more stable and needs less warm-up.
  5. What does the feed-forward network add? Per-token non-linear processing; attention only mixes tokens (a weighted average).
  6. Encoder-only vs decoder-only vs encoder-decoder? BERT-style understanding, GPT-style generation, T5-style text-to-text.
  7. Why is a too-low training loss suspicious in a language model? It often means information leakage, e.g. a missing causal mask.

Glossary

Term Meaning, with an example
Transformer A network of stacked attention + feed-forward blocks, with no recurrence
Transformer block LayerNorm → attention → +residual → LayerNorm → FFN → +residual (pre-norm)
Positional encoding A way to give each token its position; added vector or rotation
Learned positional embedding nn.Embedding(max_len, d), trained like any weight
Sinusoidal positional encoding Fixed sin/cos waves: PE(p, 2i) = sin(p/10000^(2i/d))
RoPE (rotary position embedding) Rotate pairs of query/key numbers by position × θ; scores depend on distance
Relative position The distance between two tokens (2), not where they sit (3 and 1)
Permutation-equivariant Shuffling inputs only shuffles outputs; attention without positions
Residual (skip) connection x + sublayer(x): keep the input, add a change
Layer normalisation Per-token rescale to mean 0, std 1, then learned scale and shift
Post-norm LayerNorm(x + sublayer(x)) (original 2017 design)
Pre-norm x + sublayer(LayerNorm(x)) (most current models)
Warm-up A small learning rate that grows over the first steps
Vanishing gradient Gradients that shrink to ~0 before reaching early layers; 2.7e-17
Position-wise feed-forward network The same 2-layer MLP (d → 4d → d) applied to each token
GELU Gaussian Error Linear Unit, a smooth ReLU used in GPT models
Language model Predicts the next token from the previous ones
Character-level model Tokens are single characters; 65 in Tiny Shakespeare
Validation loss Loss on text the model never trained on
Sampling (torch.multinomial) Drawing the next token at random from the predicted probabilities
Encoder-only (BERT) No causal mask; trained by masked language modelling
Masked language modelling Hide some tokens and train the model to fill them in
Decoder-only (GPT) Causal mask; trained by next-token prediction
Encoder-decoder (T5) Encoder + decoder with cross-attention; text in, text out
Cross-attention Decoder queries attend to encoder keys and values

Summary

  • Self-attention is order-blind; positions fix it: learned, sinusoidal, or RoPE (relative distance for free).
  • Deep plain stacks collapse (cosine 1.000, gradient 2.7e-17); residuals + layer norm (pre-norm) keep them alive.
  • Block = attention (talk) + feed-forward (think), each with norm and residual.
  • BERT (encoder-only), GPT (decoder-only), T5 (encoder-decoder) are the same blocks with different masks and training.
  • Our 818,241-weight GPT trained 5 minutes on Tiny Shakespeare: validation loss 4.37 → 1.66; without the causal mask it "scores" 0.05 and writes only newlines.

Next: Episode 76, pretrained transformers with Hugging Face: tokenizers, pipelines, fine-tuning BERT, and the Vision Transformer.