SyntaxPilot
Courses/Machine Learning/Lesson 74
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 74 of 90

Self-attention from scratch

self-attentionquerieskeysvaluesscaled dot-product attentionattention matrixcausal maskpadding maskmulti-head attentionimplementing attention in PyTorch

Machine Learning Full Course · Episode 74. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output.

The big idea: a word's meaning depends on its neighbours ("river bank" vs "bank account"), but an embedding gives each word one fixed vector, and averaging a sentence's vectors loses order ("dog bites man" = "man bites dog"). Self-attention lets every word score every word in the same sentence and take a weighted mix, all words at once, with no RNN. Each word plays three roles, so it gets three learned vectors: a query (what it is looking for), a key (what it offers to be matched) and a value (what it hands over). Scaled dot-product attention is softmax(Q·Kᵀ / √d_k)·V. Masks hide the future (causal) or padding; multi-head attention runs several attentions side by side. Our class matched nn.MultiheadAttention exactly, and on a contact-lookup task self-attention reached 98.4% where averaging got 34.0%.

Sources: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, "Attention Is All You Need", submitted 12 June 2017 (arXiv 1706.03762: the Transformer relies entirely on attention, "dispensing with recurrence and convolutions entirely"; ar5iv: Attention(Q, K, V) = softmax(QKᵀ/√d_k)V; for large d_k the dot products grow large and push the softmax into regions with very small gradients; h = 8 heads with d_k = d_v = 64 and d_model = 512; the decoder's self-attention is masked by setting illegal connections to −∞ before the softmax).

Words you need first

  • Embedding (Episode 72): a learned vector of numbers per token.
  • Attention (Episode 73): scores → softmax → weighted sum. There, a decoder attended to an encoder.
  • Dot product and cosine similarity (Episode 3): the dot product is bigger when vectors point the same way; cosine = dot ÷ (length × length), 1 means the same direction.
  • Matrix multiplication and transpose (Episode 4): X @ X.T gives every row's dot product with every row; .T swaps rows and columns.
  • Softmax (Episode 23): positive numbers that add up to 1.
  • Tensor shape: (2, 5, 8) = batch of 2 sentences, 5 words each, 8 numbers per word.

The story: averaging word vectors fails

import numpy as np
E = {"dog": [0, 0, 1, 0], "man": [0, 0, 0.8, 0.6], "bites": [0.2, 0.9, 0, 0],   # made-up
     "the": [0, 0, 0, 0.1], "river": [1, 0, 0, 0], "bank": [0.5, 0.5, 0, 0],      # 4-number
     "account": [0, 1, 0, 0]}                                                     # embeddings
avg = lambda s: np.mean([E[w] for w in s.split()], axis=0)
cos = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print("dog bites man :", avg("dog bites man").round(3))
print("man bites dog :", avg("man bites dog").round(3))
print("cosine        :", round(cos(avg("dog bites man"), avg("man bites dog")), 3))
print("'bank' in 'the river bank'  :", E["bank"])
print("'bank' in 'the bank account':", E["bank"])
dog bites man : [0.067 0.3   0.6   0.2  ]
man bites dog : [0.067 0.3   0.6   0.2  ]
cosine        : 1.0
'bank' in 'the river bank'  : [0.5, 0.5, 0, 0]
'bank' in 'the bank account': [0.5, 0.5, 0, 0]

Averaging gives the same vector for "dog bites man" and "man bites dog" (cosine 1.0), and "bank" keeps one vector whatever its neighbours say. An average throws away who relates to whom.

Self-attention: every word looks at every word

Self-attention: each word scores every word in its own sentence (itself included), the scores become weights with softmax, and the word's new vector is the weighted mix of all the words. Because nothing waits for a previous word, all positions are computed at once (in parallel), unlike an RNN.

import numpy as np
E = {"the": [0, 0, 0, 0.1], "river": [1, 0, 0, 0],
     "bank": [0.5, 0.5, 0, 0], "account": [0, 1, 0, 0]}
def softmax(s): e = np.exp(s - s.max(-1, keepdims=True)); return e / e.sum(-1, keepdims=True)
def self_attend(sentence):
    words = sentence.split(); X = np.array([E[w] for w in words])   # one row per word
    A = softmax(X @ X.T)                  # every word scores EVERY word (itself too)
    return words, A, A @ X                # new vector = weighted mix of all words
for s in ["the river bank", "the bank account"]:
    words, A, out = self_attend(s)
    i = words.index("bank")
    print(f"{s!r}: bank attends to"); print("  ", dict(zip(words, A[i].round(2).tolist())))
    print("   new 'bank' vector:", out[i].round(2))
'the river bank': bank attends to
   {'the': 0.23, 'river': 0.38, 'bank': 0.38}
   new 'bank' vector: [0.58 0.19 0.   0.02]
'the bank account': bank attends to
   {'the': 0.23, 'bank': 0.38, 'account': 0.38}
   new 'bank' vector: [0.19 0.58 0.   0.02]

"bank" next to "river" becomes water-like (0.58 in the first, "water" number); next to "account" it becomes money-like (0.58 in the second). A vector that depends on the surrounding words is a contextual embedding.

The flaw: the same vector X was used to ask, to be matched and to be mixed. A vector's dot product with itself tends to be the largest, so words mostly look at themselves, and there is nothing to learn: "the" took 23% of bank's attention with no way to learn to ignore it.

Real-life example: searching your phone's contacts

On your phone In self-attention
What you type: "Priya" Query: what this word is looking for
Each saved contact's name Key: what each word offers to be matched against
The phone number you get back Value: what each word hands over when chosen

Queries, keys, values and scaled dot-product attention

Q = X·W_q    K = X·W_k    V = X·W_v            (W_q, W_k, W_v are learned matrices)
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
  • d_k: the number of numbers in each query and key.
  • Attention matrix: softmax(Q·Kᵀ/√d_k), shape (words × words); row i = how much word i (asking) takes from each word j (answering); each row adds up to 1.
  • Scaled dot-product attention: the formula above, from Vaswani et al. (2017).
import torch
torch.manual_seed(74)
X = torch.randn(3, 4)                       # 3 words, 4 numbers each (embeddings)
Wq, Wk, Wv = torch.randn(3, 4, 2)           # learned matrices: 4 numbers -> 2
Q, K, V = X @ Wq, X @ Wk, X @ Wv            # query, key, value for every word
A = torch.softmax(Q @ K.T / 2 ** 0.5, dim=-1)   # scaled by sqrt(d_k), d_k = 2
print("Q, K, V shapes:", tuple(Q.shape), tuple(K.shape), tuple(V.shape))
print("attention matrix (row = word asking, column = word answering):\n", A.round(decimals=2))
print("rows add up to:", A.sum(-1).round(decimals=2))
print("output:", tuple((A @ V).shape))
for d in [4, 64, 512]:                      # why divide by sqrt(d_k)?
    q, k = torch.randn(10, d), torch.randn(10, d)
    raw = q[0] @ k.T
    print(f"d_k={d:3}: spread {raw.std():4.1f} | max weight unscaled "
          f"{torch.softmax(raw, -1).max():.2f}, scaled {torch.softmax(raw / d ** 0.5, -1).max():.2f}")
Q, K, V shapes: (3, 2) (3, 2) (3, 2)
attention matrix (row = word asking, column = word answering):
 tensor([[0.0900, 0.0000, 0.9100],
        [0.2400, 0.0400, 0.7300],
        [0.0100, 0.9900, 0.0000]])
rows add up to: tensor([1., 1., 1.])
output: (3, 2)
d_k=  4: spread  1.9 | max weight unscaled 0.61, scaled 0.33
d_k= 64: spread  7.1 | max weight unscaled 0.95, scaled 0.26
d_k=512: spread 22.4 | max weight unscaled 1.00, scaled 0.49

With separate queries and keys, word 0 gives 91% to word 2 rather than to itself. Why divide by √d_k? For random vectors the spread of dot products grows like √d_k (1.9, 7.1, 22.4 above). Large scores make softmax put almost all weight on one word (1.00 at d_k = 512), where its gradients are almost zero and learning stalls. Dividing by √d_k keeps the spread near 1, so the softmax stays soft.

Masks

In real life In attention
A live cricket commentator can only talk about balls already bowled Causal mask: a position may not look at later positions
Empty slots in a half-full egg tray are not counted Padding mask: ignore the <pad> tokens
  • Causal mask: True above the diagonal (the future). Scores there become −∞, and softmax turns −∞ into exactly 0. Needed in any model that generates text left to right, so it cannot peek at the word it must predict.
  • Padding: sentences in a batch must have the same length, so short ones are filled with a <pad> token. Padding mask: True at pad positions, so no word attends to them.
import torch
torch.manual_seed(74)
scores = torch.randn(4, 4)                                 # 4 words scoring 4 words
causal = torch.triu(torch.ones(4, 4, dtype=torch.bool), diagonal=1)   # True = the future
print("causal mask (True = not allowed):\n", causal.int())
A = torch.softmax(scores.masked_fill(causal, float("-inf")), dim=-1)
print("causal attention:\n", A.round(decimals=2))
# padding: sentence 2 has only 2 real words, then 2 <pad> tokens
lengths = torch.tensor([4, 2])
pad = torch.arange(4)[None, :] >= lengths[:, None]         # True where <pad>
print("padding mask:\n", pad.int())
s = torch.randn(2, 4, 4).masked_fill(pad[:, None, :], float("-inf"))
print("sentence 2, word 0 attends:", [round(w, 2) for w in torch.softmax(s, -1)[1, 0].tolist()])
causal mask (True = not allowed):
 tensor([[0, 1, 1, 1],
        [0, 0, 1, 1],
        [0, 0, 0, 1],
        [0, 0, 0, 0]], dtype=torch.int32)
causal attention:
 tensor([[1.0000, 0.0000, 0.0000, 0.0000],
        [0.4800, 0.5200, 0.0000, 0.0000],
        [0.0500, 0.5300, 0.4200, 0.0000],
        [0.0700, 0.1400, 0.6500, 0.1400]])
padding mask:
 tensor([[0, 0, 0, 0],
        [0, 0, 1, 1]], dtype=torch.int32)
sentence 2, word 0 attends: [0.47, 0.53, 0.0, 0.0]

Multi-head attention

Reviewing an essay In attention
One reviewer checks grammar Head 1: its own W_q, W_k, W_v
Another checks facts Head 2: a different pattern
The editor combines the notes Concatenate the heads, multiply by W_o
  • Split d_model numbers into h heads of d_k = d_model / h numbers each.
  • Each head runs scaled dot-product attention on its slice, so different heads can learn different relationships.
  • Concatenate (join end to end) the heads and multiply by the output matrix W_o.
  • The original transformer: d_model = 512, h = 8, d_k = 64.
import torch, torch.nn as nn
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__(); self.h, self.dk = n_heads, d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)    # W_q, W_k, W_v in one matrix
        self.out = nn.Linear(d_model, d_model)        # W_o: mixes the heads
    def forward(self, x, mask=None):
        B, T, D = x.shape; q, k, v = self.qkv(x).split(D, dim=-1)
        q, k, v = [t.view(B, T, self.h, self.dk).transpose(1, 2) for t in (q, k, v)]
        scores = q @ k.transpose(-2, -1) / self.dk ** 0.5      # (B, heads, T, T)
        if mask is not None: scores = scores.masked_fill(mask, float("-inf"))
        y = torch.softmax(scores, dim=-1) @ v                  # each head on its own
        return self.out(y.transpose(1, 2).reshape(B, T, D))    # glue heads together
torch.manual_seed(74); ours, ref = MultiHeadAttention(8, 2), nn.MultiheadAttention(8, 2, batch_first=True)
ref.in_proj_weight.data, ref.in_proj_bias.data = ours.qkv.weight.data, ours.qkv.bias.data
ref.out_proj.weight.data, ref.out_proj.bias.data = ours.out.weight.data, ours.out.bias.data
x = torch.randn(2, 5, 8); causal = torch.triu(torch.ones(5, 5, dtype=torch.bool), 1)
a = ours(x, causal); b, _ = ref(x, x, x, attn_mask=causal); n = sum(p.numel() for p in ours.parameters())
print(f"shape {tuple(a.shape)} | max diff vs PyTorch {(a - b).abs().max():.1e} | weights {n}")
shape (2, 5, 8) | max diff vs PyTorch 0.0e+00 | weights 288

Copying our weights into PyTorch's nn.MultiheadAttention gives an identical result, causal mask included. Weights: qkv = 8×24 + 24 = 216, out = 8×8 + 8 = 72, total 288.

Does it learn? A contact-lookup task

Each sequence is 6 "contacts" (a name 0-7 and a digit 0-9 glued into one token, name*10 + digit) followed by a query name; the answer is that contact's digit. Model A averages all tokens; model B runs our self-attention and reads the query token's new vector. mha.py contains the MultiHeadAttention class above.

import torch, torch.nn as nn, torch.nn.functional as F
from mha import MultiHeadAttention                       # our class from before
torch.manual_seed(74)
def batch(n):     # 6 contacts "name+number" as one token (name*10 + digit), then a query name
    names = torch.rand(n, 8).argsort(1)[:, :6]           # 6 different names out of 8
    digits = torch.randint(10, (n, 6))
    pick = torch.randint(6, (n,))
    x = torch.cat([names * 10 + digits, 80 + names[torch.arange(n), pick, None]], 1)
    return x, digits[torch.arange(n), pick]               # answer: that contact's digit
class Model(nn.Module):
    def __init__(self, attention):
        super().__init__(); self.attention = attention
        self.emb = nn.Embedding(88, 32); self.mha = MultiHeadAttention(32, 2)
        self.head = nn.Linear(32, 10)
    def forward(self, x):
        h = self.emb(x)
        if self.attention: return self.head(self.mha(h)[:, -1])   # the query word's new vector
        return self.head(h.mean(1))                               # no attention: average all
for attention in [False, True]:
    m = Model(attention); opt = torch.optim.Adam(m.parameters(), 3e-3)
    for step in range(2000):
        x, y = batch(128); loss = F.cross_entropy(m(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    x, y = batch(2000)
    print(f"attention={attention!s:5}  test accuracy {(m(x).argmax(1) == y).float().mean():.1%}")

Output (2,000 training steps each):

attention=False  test accuracy 34.0%
attention=True   test accuracy 98.4%

Averaging sees all six digits but cannot tell which belongs to the name asked for (34.0%). With self-attention the query can find the matching key and take its value (98.4%).

Common mistakes

import torch, torch.nn as nn
try:
    mha = nn.MultiheadAttention(embed_dim=10, num_heads=3)
except AssertionError as e:
    print("AssertionError:", e)
scores = torch.tensor([[1.0, 2.0, 3.0]])
everything_masked = torch.tensor([[True, True, True]])   # e.g. only <pad>
print(torch.softmax(scores.masked_fill(everything_masked, float("-inf")), dim=-1))
AssertionError: embed_dim must be divisible by num_heads
tensor([[nan, nan, nan]])
Mistake Exact message / symptom Why Fix
nn.MultiheadAttention(embed_dim=10, num_heads=3) AssertionError: embed_dim must be divisible by num_heads 10 numbers can't form 3 equal heads make embed_dim a multiple of num_heads
Every position of a row masked tensor([[nan, nan, nan]]) softmax of all −∞ is 0/0 keep at least one real position per row
Forgetting / √d_k max weight 1.00, slow or stuck training softmax saturates, gradients vanish divide scores by d_k ** 0.5
No causal mask in a text generator a suspiciously low training loss it sees the word it must predict torch.triu(..., diagonal=1) mask
Softmax over the wrong dimension rows don't add up to 1 normalised over queries softmax(scores, dim=-1)
Averaging word vectors dog bites man = man bites dog order and relations lost self-attention (plus positions, Episode 75)

Mini project: check against PyTorch's fused attention

PyTorch has a built-in F.scaled_dot_product_attention. Write your own and compare, with and without a causal mask.

import torch, torch.nn.functional as F
def attention(q, k, v, causal=False):                  # our scaled dot-product attention
    scores = q @ k.transpose(-2, -1) / q.shape[-1] ** 0.5
    if causal:
        T = q.shape[-2]; scores = scores.masked_fill(torch.triu(torch.ones(T, T, dtype=torch.bool), 1), float("-inf"))
    return torch.softmax(scores, -1) @ v
torch.manual_seed(74)
q, k, v = torch.randn(3, 2, 4, 6, 16)                  # batch 2, 4 heads, 6 tokens, 16 numbers
for causal in [False, True]:
    ours = attention(q, k, v, causal)
    ref = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
    print(f"causal={causal!s:5} shape {tuple(ours.shape)} max difference {(ours - ref).abs().max():.1e}")
causal=False shape (2, 4, 6, 16) max difference 2.4e-07
causal=True  shape (2, 4, 6, 16) max difference 3.6e-07

Differences around 10⁻⁷ are floating-point rounding: the two compute the same thing in a slightly different order.

Practice

1. Print where the lookup model's query looks, then re-train it without dividing by √d_k.

Solution
import torch, torch.nn.functional as F
exec(open("r6.py").read().split("for attention in")[0])         # reuse batch() and Model
import mha
def train(model, steps=2000):
    opt = torch.optim.Adam(model.parameters(), 3e-3)
    for _ in range(steps):
        x, y = batch(128); loss = F.cross_entropy(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    x, y = batch(2000); return (model(x).argmax(1) == y).float().mean().item()
m = Model(True); print(f"with / sqrt(d_k):    accuracy {train(m):.1%}")
x, y = batch(1)                                                  # one example: where does the query look?
h = m.emb(x); B, T, D = h.shape
q, k, v = m.mha.qkv(h).split(D, -1)
q, k = [t.view(B, T, 2, 16).transpose(1, 2) for t in (q, k)]
w = torch.softmax(q @ k.transpose(-2, -1) / 4, -1)[0, :, -1]       # both heads, query row
print("contacts:", [f"{t // 10}:{t % 10}" for t in x[0, :6].tolist()], "| asking for name", x[0, 6].item() - 80, "| answer", y.item())
for head in range(2): print(f"head {head} weights:", [round(x, 2) for x in w[head, :6].tolist()])
src = open("mha.py").read().replace(" / self.dk ** 0.5", "")       # remove the scaling
exec(src, mha.__dict__)
m2 = Model(True); m2.mha = mha.MultiHeadAttention(32, 2)
print(f"without / sqrt(d_k): accuracy {train(m2):.1%}")
with / sqrt(d_k):    accuracy 100.0%
contacts: ['1:7', '0:1', '3:5', '4:4', '5:5', '6:0'] | asking for name 5 | answer 5
head 0 weights: [0.13, 0.54, 0.0, 0.0, 0.05, 0.02]
head 1 weights: [0.0, 0.0, 0.0, 0.0, 0.97, 0.0]
without / sqrt(d_k): accuracy 96.5%

r6.py is the lookup program above. This run reaches 100.0% (98.4% in the episode: the random batches differ because the averaging model is not trained first). Head 1 puts 0.97 of the query's attention on contact 5:5, the one asked for; head 0 spreads out. Without the scaling the model still learns here (96.5%), because d_k is only 16 per head; the effect grows with d_k (see the 512 case above).

2. Change the lookup task to 12 contacts out of 16 names.

HintUse torch.rand(n, 16).argsort(1)[:, :12], nn.Embedding(176, 32) (16×10 pair tokens + 16 query tokens) and 160 + names[...] for the query. Train both models and compare their accuracy with the 6-contact version.

3. Add a key-padding mask argument to MultiHeadAttention.

SolutionPass pad of shape (B, T) with True at pads and use scores = scores.masked_fill(pad[:, None, None, :], float("-inf")) so every head and every query ignores those keys.

4. What is the shape of Q·Kᵀ for one head with 50 tokens?

Answer(50, 50): one score for every (asking word, answering word) pair.

Quick quiz

x = torch.randn(16, 50, 512)   # batch 16, 50 words, 512 numbers
mha = MultiHeadAttention(512, 8)
# inside: scores = q @ k.transpose(-2, -1)
# scores.shape = ?
Answer

(16, 8, 50, 50): each of the 8 heads has its own 50 × 50 attention matrix. (16, 8, 50, 64) is the shape of q.

Interview questions

  1. What are Q, K and V? Three learned projections of each token: the query is what it searches for, the key what it is matched on, the value what it contributes.
  2. Why scale by √d_k? Dot products grow with dimension; large scores saturate the softmax and give tiny gradients.
  3. Self-attention vs RNN? Attention connects any two positions in one step and runs in parallel; its cost grows with the square of the sequence length (a T × T matrix).
  4. Why multiple heads? Each head can attend to a different relationship (syntax, coreference, nearby words) at about the same total cost as one big head.
  5. Causal vs padding mask? Causal hides future positions in generation; padding hides filler tokens in batched sequences.
  6. Is self-attention order-aware? No: without positional information it treats the input as a set (Episode 75 fixes this).
  7. What does −∞ do in a mask? exp(−∞) = 0, so masked positions get exactly zero weight after softmax.

Glossary

Term Meaning, with an example
Self-attention Attention where a sequence attends to itself; every word scores every word
Contextual embedding A word vector that depends on the neighbours; "bank" near "river" vs "account"
Query (Q) Learned vector for what a token is looking for; what you type into contacts search
Key (K) Learned vector a token offers for matching; a saved contact's name
Value (V) Learned vector a token hands over when attended to; the phone number
W_q, W_k, W_v, W_o Learned matrices making queries, keys, values, and mixing the heads
d_k Number of numbers in each query/key; 64 in the original transformer
Scaled dot-product attention softmax(Q·Kᵀ/√d_k)·V
Attention matrix The T × T softmax weights; row = asking word, column = answering word
Scaling by √d_k Dividing scores so their spread stays near 1 and softmax doesn't saturate
Saturation Softmax giving ~1.0 to one item and ~0 to the rest, with near-zero gradients
Mask A True/False grid of positions that may not be attended to
Causal mask Hides later positions; upper triangle torch.triu(..., diagonal=1)
Padding / <pad> Filler tokens that make sentences in a batch the same length
Padding mask Hides pad positions from attention
Multi-head attention Several attention heads in parallel, concatenated and mixed by W_o
Head One attention computation on a d_model/h slice
Concatenate Join tensors end to end; 2 heads of 4 → 8 numbers
nn.MultiheadAttention PyTorch's built-in multi-head attention layer
F.scaled_dot_product_attention PyTorch's built-in (fused) scaled dot-product attention
NaN "Not a number", e.g. 0/0; spreads through every later calculation

Summary

  • Averaging word vectors loses order and context (cosine 1.0 for dog/man).
  • Self-attention: every word scores every word, softmax, weighted mix; "bank" changes with its neighbours.
  • Queries, keys and values give each role its own learned vector; softmax(QKᵀ/√d_k)V, and the √d_k keeps softmax from saturating.
  • Causal masks hide the future, padding masks hide filler; −∞ → weight 0.
  • Multi-head attention = several heads + W_o; our class matched PyTorch exactly, and solved contact lookup at 98.4% vs 34.0%.

Next: Episode 75, transformers from scratch: positional encoding, residuals, layer norm, feed-forward layers and a tiny GPT.