Machine Learning Full Course · Episode 74. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output.
The big idea: a word's meaning depends on its neighbours ("river bank" vs "bank account"), but an embedding gives each word one fixed vector, and averaging a sentence's vectors loses order ("dog bites man" = "man bites dog"). Self-attention lets every word score every word in the same sentence and take a weighted mix, all words at once, with no RNN. Each word plays three roles, so it gets three learned vectors: a query (what it is looking for), a key (what it offers to be matched) and a value (what it hands over). Scaled dot-product attention is softmax(Q·Kᵀ / √d_k)·V. Masks hide the future (causal) or padding; multi-head attention runs several attentions side by side. Our class matched nn.MultiheadAttention exactly, and on a contact-lookup task self-attention reached 98.4% where averaging got 34.0%.
Sources: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, "Attention Is All You Need", submitted 12 June 2017 (arXiv 1706.03762: the Transformer relies entirely on attention, "dispensing with recurrence and convolutions entirely"; ar5iv: Attention(Q, K, V) = softmax(QKᵀ/√d_k)V; for large d_k the dot products grow large and push the softmax into regions with very small gradients; h = 8 heads with d_k = d_v = 64 and d_model = 512; the decoder's self-attention is masked by setting illegal connections to −∞ before the softmax).
Words you need first
- Embedding (Episode 72): a learned vector of numbers per token.
- Attention (Episode 73): scores → softmax → weighted sum. There, a decoder attended to an encoder.
- Dot product and cosine similarity (Episode 3): the dot product is bigger when vectors point the same way; cosine = dot ÷ (length × length), 1 means the same direction.
- Matrix multiplication and transpose (Episode 4):
X @ X.Tgives every row's dot product with every row;.Tswaps rows and columns. - Softmax (Episode 23): positive numbers that add up to 1.
- Tensor shape:
(2, 5, 8)= batch of 2 sentences, 5 words each, 8 numbers per word.
The story: averaging word vectors fails
import numpy as np
E = {"dog": [0, 0, 1, 0], "man": [0, 0, 0.8, 0.6], "bites": [0.2, 0.9, 0, 0], # made-up
"the": [0, 0, 0, 0.1], "river": [1, 0, 0, 0], "bank": [0.5, 0.5, 0, 0], # 4-number
"account": [0, 1, 0, 0]} # embeddings
avg = lambda s: np.mean([E[w] for w in s.split()], axis=0)
cos = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print("dog bites man :", avg("dog bites man").round(3))
print("man bites dog :", avg("man bites dog").round(3))
print("cosine :", round(cos(avg("dog bites man"), avg("man bites dog")), 3))
print("'bank' in 'the river bank' :", E["bank"])
print("'bank' in 'the bank account':", E["bank"])
dog bites man : [0.067 0.3 0.6 0.2 ]
man bites dog : [0.067 0.3 0.6 0.2 ]
cosine : 1.0
'bank' in 'the river bank' : [0.5, 0.5, 0, 0]
'bank' in 'the bank account': [0.5, 0.5, 0, 0]
Averaging gives the same vector for "dog bites man" and "man bites dog" (cosine 1.0), and "bank" keeps one vector whatever its neighbours say. An average throws away who relates to whom.
Self-attention: every word looks at every word
Self-attention: each word scores every word in its own sentence (itself included), the scores become weights with softmax, and the word's new vector is the weighted mix of all the words. Because nothing waits for a previous word, all positions are computed at once (in parallel), unlike an RNN.
import numpy as np
E = {"the": [0, 0, 0, 0.1], "river": [1, 0, 0, 0],
"bank": [0.5, 0.5, 0, 0], "account": [0, 1, 0, 0]}
def softmax(s): e = np.exp(s - s.max(-1, keepdims=True)); return e / e.sum(-1, keepdims=True)
def self_attend(sentence):
words = sentence.split(); X = np.array([E[w] for w in words]) # one row per word
A = softmax(X @ X.T) # every word scores EVERY word (itself too)
return words, A, A @ X # new vector = weighted mix of all words
for s in ["the river bank", "the bank account"]:
words, A, out = self_attend(s)
i = words.index("bank")
print(f"{s!r}: bank attends to"); print(" ", dict(zip(words, A[i].round(2).tolist())))
print(" new 'bank' vector:", out[i].round(2))
'the river bank': bank attends to
{'the': 0.23, 'river': 0.38, 'bank': 0.38}
new 'bank' vector: [0.58 0.19 0. 0.02]
'the bank account': bank attends to
{'the': 0.23, 'bank': 0.38, 'account': 0.38}
new 'bank' vector: [0.19 0.58 0. 0.02]
"bank" next to "river" becomes water-like (0.58 in the first, "water" number); next to "account" it becomes money-like (0.58 in the second). A vector that depends on the surrounding words is a contextual embedding.
The flaw: the same vector X was used to ask, to be matched and to be mixed. A vector's dot product with itself tends to be the largest, so words mostly look at themselves, and there is nothing to learn: "the" took 23% of bank's attention with no way to learn to ignore it.
Real-life example: searching your phone's contacts
| On your phone | In self-attention |
|---|---|
| What you type: "Priya" | Query: what this word is looking for |
| Each saved contact's name | Key: what each word offers to be matched against |
| The phone number you get back | Value: what each word hands over when chosen |
Queries, keys, values and scaled dot-product attention
Q = X·W_q K = X·W_k V = X·W_v (W_q, W_k, W_v are learned matrices)
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
- d_k: the number of numbers in each query and key.
- Attention matrix:
softmax(Q·Kᵀ/√d_k), shape (words × words); row i = how much word i (asking) takes from each word j (answering); each row adds up to 1. - Scaled dot-product attention: the formula above, from Vaswani et al. (2017).
import torch
torch.manual_seed(74)
X = torch.randn(3, 4) # 3 words, 4 numbers each (embeddings)
Wq, Wk, Wv = torch.randn(3, 4, 2) # learned matrices: 4 numbers -> 2
Q, K, V = X @ Wq, X @ Wk, X @ Wv # query, key, value for every word
A = torch.softmax(Q @ K.T / 2 ** 0.5, dim=-1) # scaled by sqrt(d_k), d_k = 2
print("Q, K, V shapes:", tuple(Q.shape), tuple(K.shape), tuple(V.shape))
print("attention matrix (row = word asking, column = word answering):\n", A.round(decimals=2))
print("rows add up to:", A.sum(-1).round(decimals=2))
print("output:", tuple((A @ V).shape))
for d in [4, 64, 512]: # why divide by sqrt(d_k)?
q, k = torch.randn(10, d), torch.randn(10, d)
raw = q[0] @ k.T
print(f"d_k={d:3}: spread {raw.std():4.1f} | max weight unscaled "
f"{torch.softmax(raw, -1).max():.2f}, scaled {torch.softmax(raw / d ** 0.5, -1).max():.2f}")
Q, K, V shapes: (3, 2) (3, 2) (3, 2)
attention matrix (row = word asking, column = word answering):
tensor([[0.0900, 0.0000, 0.9100],
[0.2400, 0.0400, 0.7300],
[0.0100, 0.9900, 0.0000]])
rows add up to: tensor([1., 1., 1.])
output: (3, 2)
d_k= 4: spread 1.9 | max weight unscaled 0.61, scaled 0.33
d_k= 64: spread 7.1 | max weight unscaled 0.95, scaled 0.26
d_k=512: spread 22.4 | max weight unscaled 1.00, scaled 0.49
With separate queries and keys, word 0 gives 91% to word 2 rather than to itself. Why divide by √d_k? For random vectors the spread of dot products grows like √d_k (1.9, 7.1, 22.4 above). Large scores make softmax put almost all weight on one word (1.00 at d_k = 512), where its gradients are almost zero and learning stalls. Dividing by √d_k keeps the spread near 1, so the softmax stays soft.
Masks
| In real life | In attention |
|---|---|
| A live cricket commentator can only talk about balls already bowled | Causal mask: a position may not look at later positions |
| Empty slots in a half-full egg tray are not counted | Padding mask: ignore the <pad> tokens |
- Causal mask:
Trueabove the diagonal (the future). Scores there become −∞, and softmax turns −∞ into exactly 0. Needed in any model that generates text left to right, so it cannot peek at the word it must predict. - Padding: sentences in a batch must have the same length, so short ones are filled with a
<pad>token. Padding mask:Trueat pad positions, so no word attends to them.
import torch
torch.manual_seed(74)
scores = torch.randn(4, 4) # 4 words scoring 4 words
causal = torch.triu(torch.ones(4, 4, dtype=torch.bool), diagonal=1) # True = the future
print("causal mask (True = not allowed):\n", causal.int())
A = torch.softmax(scores.masked_fill(causal, float("-inf")), dim=-1)
print("causal attention:\n", A.round(decimals=2))
# padding: sentence 2 has only 2 real words, then 2 <pad> tokens
lengths = torch.tensor([4, 2])
pad = torch.arange(4)[None, :] >= lengths[:, None] # True where <pad>
print("padding mask:\n", pad.int())
s = torch.randn(2, 4, 4).masked_fill(pad[:, None, :], float("-inf"))
print("sentence 2, word 0 attends:", [round(w, 2) for w in torch.softmax(s, -1)[1, 0].tolist()])
causal mask (True = not allowed):
tensor([[0, 1, 1, 1],
[0, 0, 1, 1],
[0, 0, 0, 1],
[0, 0, 0, 0]], dtype=torch.int32)
causal attention:
tensor([[1.0000, 0.0000, 0.0000, 0.0000],
[0.4800, 0.5200, 0.0000, 0.0000],
[0.0500, 0.5300, 0.4200, 0.0000],
[0.0700, 0.1400, 0.6500, 0.1400]])
padding mask:
tensor([[0, 0, 0, 0],
[0, 0, 1, 1]], dtype=torch.int32)
sentence 2, word 0 attends: [0.47, 0.53, 0.0, 0.0]
Multi-head attention
| Reviewing an essay | In attention |
|---|---|
| One reviewer checks grammar | Head 1: its own W_q, W_k, W_v |
| Another checks facts | Head 2: a different pattern |
| The editor combines the notes | Concatenate the heads, multiply by W_o |
- Split
d_modelnumbers intohheads ofd_k = d_model / hnumbers each. - Each head runs scaled dot-product attention on its slice, so different heads can learn different relationships.
- Concatenate (join end to end) the heads and multiply by the output matrix
W_o. - The original transformer: d_model = 512, h = 8, d_k = 64.
import torch, torch.nn as nn
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__(); self.h, self.dk = n_heads, d_model // n_heads
self.qkv = nn.Linear(d_model, 3 * d_model) # W_q, W_k, W_v in one matrix
self.out = nn.Linear(d_model, d_model) # W_o: mixes the heads
def forward(self, x, mask=None):
B, T, D = x.shape; q, k, v = self.qkv(x).split(D, dim=-1)
q, k, v = [t.view(B, T, self.h, self.dk).transpose(1, 2) for t in (q, k, v)]
scores = q @ k.transpose(-2, -1) / self.dk ** 0.5 # (B, heads, T, T)
if mask is not None: scores = scores.masked_fill(mask, float("-inf"))
y = torch.softmax(scores, dim=-1) @ v # each head on its own
return self.out(y.transpose(1, 2).reshape(B, T, D)) # glue heads together
torch.manual_seed(74); ours, ref = MultiHeadAttention(8, 2), nn.MultiheadAttention(8, 2, batch_first=True)
ref.in_proj_weight.data, ref.in_proj_bias.data = ours.qkv.weight.data, ours.qkv.bias.data
ref.out_proj.weight.data, ref.out_proj.bias.data = ours.out.weight.data, ours.out.bias.data
x = torch.randn(2, 5, 8); causal = torch.triu(torch.ones(5, 5, dtype=torch.bool), 1)
a = ours(x, causal); b, _ = ref(x, x, x, attn_mask=causal); n = sum(p.numel() for p in ours.parameters())
print(f"shape {tuple(a.shape)} | max diff vs PyTorch {(a - b).abs().max():.1e} | weights {n}")
shape (2, 5, 8) | max diff vs PyTorch 0.0e+00 | weights 288
Copying our weights into PyTorch's nn.MultiheadAttention gives an identical result, causal mask included. Weights: qkv = 8×24 + 24 = 216, out = 8×8 + 8 = 72, total 288.
Does it learn? A contact-lookup task
Each sequence is 6 "contacts" (a name 0-7 and a digit 0-9 glued into one token, name*10 + digit) followed by a query name; the answer is that contact's digit. Model A averages all tokens; model B runs our self-attention and reads the query token's new vector. mha.py contains the MultiHeadAttention class above.
import torch, torch.nn as nn, torch.nn.functional as F
from mha import MultiHeadAttention # our class from before
torch.manual_seed(74)
def batch(n): # 6 contacts "name+number" as one token (name*10 + digit), then a query name
names = torch.rand(n, 8).argsort(1)[:, :6] # 6 different names out of 8
digits = torch.randint(10, (n, 6))
pick = torch.randint(6, (n,))
x = torch.cat([names * 10 + digits, 80 + names[torch.arange(n), pick, None]], 1)
return x, digits[torch.arange(n), pick] # answer: that contact's digit
class Model(nn.Module):
def __init__(self, attention):
super().__init__(); self.attention = attention
self.emb = nn.Embedding(88, 32); self.mha = MultiHeadAttention(32, 2)
self.head = nn.Linear(32, 10)
def forward(self, x):
h = self.emb(x)
if self.attention: return self.head(self.mha(h)[:, -1]) # the query word's new vector
return self.head(h.mean(1)) # no attention: average all
for attention in [False, True]:
m = Model(attention); opt = torch.optim.Adam(m.parameters(), 3e-3)
for step in range(2000):
x, y = batch(128); loss = F.cross_entropy(m(x), y)
opt.zero_grad(); loss.backward(); opt.step()
x, y = batch(2000)
print(f"attention={attention!s:5} test accuracy {(m(x).argmax(1) == y).float().mean():.1%}")
Output (2,000 training steps each):
attention=False test accuracy 34.0%
attention=True test accuracy 98.4%
Averaging sees all six digits but cannot tell which belongs to the name asked for (34.0%). With self-attention the query can find the matching key and take its value (98.4%).
Common mistakes
import torch, torch.nn as nn
try:
mha = nn.MultiheadAttention(embed_dim=10, num_heads=3)
except AssertionError as e:
print("AssertionError:", e)
scores = torch.tensor([[1.0, 2.0, 3.0]])
everything_masked = torch.tensor([[True, True, True]]) # e.g. only <pad>
print(torch.softmax(scores.masked_fill(everything_masked, float("-inf")), dim=-1))
AssertionError: embed_dim must be divisible by num_heads
tensor([[nan, nan, nan]])
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
nn.MultiheadAttention(embed_dim=10, num_heads=3) |
AssertionError: embed_dim must be divisible by num_heads |
10 numbers can't form 3 equal heads | make embed_dim a multiple of num_heads |
| Every position of a row masked | tensor([[nan, nan, nan]]) |
softmax of all −∞ is 0/0 | keep at least one real position per row |
Forgetting / √d_k |
max weight 1.00, slow or stuck training | softmax saturates, gradients vanish | divide scores by d_k ** 0.5 |
| No causal mask in a text generator | a suspiciously low training loss | it sees the word it must predict | torch.triu(..., diagonal=1) mask |
| Softmax over the wrong dimension | rows don't add up to 1 | normalised over queries | softmax(scores, dim=-1) |
| Averaging word vectors | dog bites man = man bites dog | order and relations lost | self-attention (plus positions, Episode 75) |
Mini project: check against PyTorch's fused attention
PyTorch has a built-in F.scaled_dot_product_attention. Write your own and compare, with and without a causal mask.
import torch, torch.nn.functional as F
def attention(q, k, v, causal=False): # our scaled dot-product attention
scores = q @ k.transpose(-2, -1) / q.shape[-1] ** 0.5
if causal:
T = q.shape[-2]; scores = scores.masked_fill(torch.triu(torch.ones(T, T, dtype=torch.bool), 1), float("-inf"))
return torch.softmax(scores, -1) @ v
torch.manual_seed(74)
q, k, v = torch.randn(3, 2, 4, 6, 16) # batch 2, 4 heads, 6 tokens, 16 numbers
for causal in [False, True]:
ours = attention(q, k, v, causal)
ref = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
print(f"causal={causal!s:5} shape {tuple(ours.shape)} max difference {(ours - ref).abs().max():.1e}")
causal=False shape (2, 4, 6, 16) max difference 2.4e-07
causal=True shape (2, 4, 6, 16) max difference 3.6e-07
Differences around 10⁻⁷ are floating-point rounding: the two compute the same thing in a slightly different order.
Practice
1. Print where the lookup model's query looks, then re-train it without dividing by √d_k.
Solution
import torch, torch.nn.functional as F
exec(open("r6.py").read().split("for attention in")[0]) # reuse batch() and Model
import mha
def train(model, steps=2000):
opt = torch.optim.Adam(model.parameters(), 3e-3)
for _ in range(steps):
x, y = batch(128); loss = F.cross_entropy(model(x), y)
opt.zero_grad(); loss.backward(); opt.step()
x, y = batch(2000); return (model(x).argmax(1) == y).float().mean().item()
m = Model(True); print(f"with / sqrt(d_k): accuracy {train(m):.1%}")
x, y = batch(1) # one example: where does the query look?
h = m.emb(x); B, T, D = h.shape
q, k, v = m.mha.qkv(h).split(D, -1)
q, k = [t.view(B, T, 2, 16).transpose(1, 2) for t in (q, k)]
w = torch.softmax(q @ k.transpose(-2, -1) / 4, -1)[0, :, -1] # both heads, query row
print("contacts:", [f"{t // 10}:{t % 10}" for t in x[0, :6].tolist()], "| asking for name", x[0, 6].item() - 80, "| answer", y.item())
for head in range(2): print(f"head {head} weights:", [round(x, 2) for x in w[head, :6].tolist()])
src = open("mha.py").read().replace(" / self.dk ** 0.5", "") # remove the scaling
exec(src, mha.__dict__)
m2 = Model(True); m2.mha = mha.MultiHeadAttention(32, 2)
print(f"without / sqrt(d_k): accuracy {train(m2):.1%}")
with / sqrt(d_k): accuracy 100.0%
contacts: ['1:7', '0:1', '3:5', '4:4', '5:5', '6:0'] | asking for name 5 | answer 5
head 0 weights: [0.13, 0.54, 0.0, 0.0, 0.05, 0.02]
head 1 weights: [0.0, 0.0, 0.0, 0.0, 0.97, 0.0]
without / sqrt(d_k): accuracy 96.5%
r6.py is the lookup program above. This run reaches 100.0% (98.4% in the episode: the random batches differ because the averaging model is not trained first). Head 1 puts 0.97 of the query's attention on contact 5:5, the one asked for; head 0 spreads out. Without the scaling the model still learns here (96.5%), because d_k is only 16 per head; the effect grows with d_k (see the 512 case above).
2. Change the lookup task to 12 contacts out of 16 names. Hint
Use torch.rand(n, 16).argsort(1)[:, :12], nn.Embedding(176, 32) (16×10 pair tokens + 16 query tokens) and 160 + names[...] for the query. Train both models and compare their accuracy with the 6-contact version.
3. Add a key-padding mask argument to MultiHeadAttention. Solution
Pass pad of shape (B, T) with True at pads and use scores = scores.masked_fill(pad[:, None, None, :], float("-inf")) so every head and every query ignores those keys.
4. What is the shape of Q·Kᵀ for one head with 50 tokens? Answer
(50, 50): one score for every (asking word, answering word) pair.
Quick quiz
x = torch.randn(16, 50, 512) # batch 16, 50 words, 512 numbers
mha = MultiHeadAttention(512, 8)
# inside: scores = q @ k.transpose(-2, -1)
# scores.shape = ?
Answer
(16, 8, 50, 50): each of the 8 heads has its own 50 × 50 attention matrix. (16, 8, 50, 64) is the shape of q.
Interview questions
- What are Q, K and V? Three learned projections of each token: the query is what it searches for, the key what it is matched on, the value what it contributes.
- Why scale by √d_k? Dot products grow with dimension; large scores saturate the softmax and give tiny gradients.
- Self-attention vs RNN? Attention connects any two positions in one step and runs in parallel; its cost grows with the square of the sequence length (a T × T matrix).
- Why multiple heads? Each head can attend to a different relationship (syntax, coreference, nearby words) at about the same total cost as one big head.
- Causal vs padding mask? Causal hides future positions in generation; padding hides filler tokens in batched sequences.
- Is self-attention order-aware? No: without positional information it treats the input as a set (Episode 75 fixes this).
- What does −∞ do in a mask?
exp(−∞) = 0, so masked positions get exactly zero weight after softmax.
Glossary
| Term | Meaning, with an example |
|---|---|
| Self-attention | Attention where a sequence attends to itself; every word scores every word |
| Contextual embedding | A word vector that depends on the neighbours; "bank" near "river" vs "account" |
| Query (Q) | Learned vector for what a token is looking for; what you type into contacts search |
| Key (K) | Learned vector a token offers for matching; a saved contact's name |
| Value (V) | Learned vector a token hands over when attended to; the phone number |
| W_q, W_k, W_v, W_o | Learned matrices making queries, keys, values, and mixing the heads |
| d_k | Number of numbers in each query/key; 64 in the original transformer |
| Scaled dot-product attention | softmax(Q·Kᵀ/√d_k)·V |
| Attention matrix | The T × T softmax weights; row = asking word, column = answering word |
| Scaling by √d_k | Dividing scores so their spread stays near 1 and softmax doesn't saturate |
| Saturation | Softmax giving ~1.0 to one item and ~0 to the rest, with near-zero gradients |
| Mask | A True/False grid of positions that may not be attended to |
| Causal mask | Hides later positions; upper triangle torch.triu(..., diagonal=1) |
Padding / <pad> |
Filler tokens that make sentences in a batch the same length |
| Padding mask | Hides pad positions from attention |
| Multi-head attention | Several attention heads in parallel, concatenated and mixed by W_o |
| Head | One attention computation on a d_model/h slice |
| Concatenate | Join tensors end to end; 2 heads of 4 → 8 numbers |
nn.MultiheadAttention |
PyTorch's built-in multi-head attention layer |
F.scaled_dot_product_attention |
PyTorch's built-in (fused) scaled dot-product attention |
| NaN | "Not a number", e.g. 0/0; spreads through every later calculation |
Summary
- Averaging word vectors loses order and context (cosine 1.0 for dog/man).
- Self-attention: every word scores every word, softmax, weighted mix; "bank" changes with its neighbours.
- Queries, keys and values give each role its own learned vector; softmax(QKᵀ/√d_k)V, and the √d_k keeps softmax from saturating.
- Causal masks hide the future, padding masks hide filler; −∞ → weight 0.
- Multi-head attention = several heads + W_o; our class matched PyTorch exactly, and solved contact lookup at 98.4% vs 34.0%.
Next: Episode 75, transformers from scratch: positional encoding, residuals, layer norm, feed-forward layers and a tiny GPT.