SyntaxPilot
Courses/AI: Zero to Agents/Lesson 14

AI: Zero to Agents · Lesson 14 of 40 · 13 min

Attention and the Transformer

queued

These are the notes for AI episode 14. Last episode you invented embeddings, but a lookup table gives the word "bank" the same vector by a river and at a loan desk. Here you fix that: you watch a plain sentence average turn every word to mush, then invent attention (a relevance-weighted average with softmax), queries, keys and values, scaling by √d_k, multi-head attention, positional encoding and the causal mask, and stack them into the Transformer from the 2017 paper Attention Is All You Need. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

Meaning needs context: the Hindi word "kal"

Sentence What "kal" means Which neighbour decided
"kal" on its own yesterday or tomorrow none
"kal main gaya tha" (I went) yesterday the past-tense verb
"kal main jaunga" (I will go) tomorrow the future-tense verb

You looked at the neighbours, and not all of them equally: "main" (I) tells you nothing, the verb decides. That is exactly what attention does.

Attempt 1: average the whole sentence

Two dimensions, money and water. Every word's new vector is the plain average of its sentence:

# word vectors from episode 13:  money  water
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
       "bank": [0.7, 0.7], "loan": [1.0, 0.0]}

def average(vectors):                 # every word counts equally
    return [round(sum(col) / len(vectors), 2) for col in zip(*vectors)]

for s in ["the river bank", "the bank loan"]:
    vs = [emb[w] for w in s.split()]
    print(f"{s:15}", {w: average(vs) for w in s.split()})

Output:

the river bank  {'the': [0.27, 0.6], 'river': [0.27, 0.6], 'bank': [0.27, 0.6]}
the bank loan   {'the': [0.6, 0.27], 'bank': [0.6, 0.27], 'loan': [0.6, 0.27]}

"bank" does move (more water by the river, more money by the loan), but "the", "river" and "bank" all become the same vector. Every neighbour counted equally, even "the", so every word blended into one soup.

Real-life example: your family WhatsApp group

In a group chat In attention
200 new messages the other tokens in the sentence
"Is this about me?" for each message a relevance score per token
Read some closely, skim the rest; attention adds up to 100% attention weights that add up to 1

Scores to weights: the dot product and softmax

  • Relevance score = the dot product of two vectors (episode 13): multiply matching numbers and add. Vectors pointing the same way give a big score.
  • Softmax turns any list of scores into weights: raise e (≈ 2.718) to each score, then divide each by the total. The weights are always positive and add up to 1.

For "bank" in "the river bank": scores 0.14, 0.70, 0.98 → softmax → weights 0.20, 0.35, 0.46.

Attention in Python

import math
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
       "bank": [0.7, 0.7], "loan": [1.0, 0.0]}

def softmax(scores):                  # scores -> weights that add up to 1
    e = [math.exp(s) for s in scores]
    return [x / sum(e) for x in e]

def attend(word, sentence):
    me = emb[word]
    scores = [sum(a * b for a, b in zip(me, emb[w])) for w in sentence]  # dot product
    weights = softmax(scores)
    new = [sum(wt * emb[w][d] for wt, w in zip(weights, sentence)) for d in range(2)]
    return [round(x, 2) for x in weights], [round(x, 2) for x in new]

for s in ["the river bank", "the bank loan"]:
    weights, new = attend("bank", s.split())
    print(f"{s:15} weights {weights}  bank -> {new}")

Output:

the river bank  weights [0.2, 0.35, 0.46]  bank -> [0.34, 0.69]
the bank loan   weights [0.2, 0.46, 0.35]  bank -> [0.69, 0.34]

By the river, bank's new vector is mostly water [0.34, 0.69]; at the loan desk it is mostly money [0.69, 0.34]. Attention = replace each token's vector with a weighted average of the tokens around it, weighted by relevance. The result is a contextual embedding: it changes with the sentence.

The next problem: what you look for is not what you are

When every word attends to the words of its own sentence, it is called self-attention. Test it on "raju made chai, drank it": which word is "it"?

import math
#        thing drink person pronoun
emb = {"raju":  [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
       "drank": [0, 0, 0, 0], "it":   [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]

scores = [dot(emb["it"], emb[w]) for w in sentence]
for w, wt in zip(sentence, softmax(scores)):
    print(f"it -> {w:6} {wt:.2f}")

Output:

it -> raju   0.15
it -> made   0.15
it -> chai   0.15
it -> drank  0.15
it -> it     0.40

"it" gives 40% to itself and only 15% to chai. A plain dot product rewards words that are similar to you, but a pronoun is not similar to the thing it refers to.

Query, key and value: a shopping app

On a shopping app In attention
You type "red cotton kurta" Query: what this token is looking for
Each product's tags: kurta, cotton, red, size M Key: what each token offers to be matched on
The product you get: photos, price, the real thing Value: what a token hands over when chosen
import math
#        thing drink person pronoun
emb = {"raju":  [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
       "drank": [0, 0, 0, 0], "it":   [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]
def query(v): return [3 * v[3], 0]          # "I'm a pronoun: I look for a thing"
def key(v):   return [v[0], v[2]]           # "I'm a thing" / "I'm a person"
def value(v): return v                      # what I hand over if chosen
q = query(emb["it"])
weights = softmax([dot(q, key(emb[w])) for w in sentence])
for w, wt in zip(sentence, weights):
    print(f"it -> {w:6} {wt:.2f}")
new = [round(sum(wt * value(emb[w])[d] for wt, w in zip(weights, sentence)), 2) for d in range(4)]
print("new it:", new)

Output:

it -> raju   0.04
it -> made   0.04
it -> chai   0.83
it -> drank  0.04
it -> it     0.04
new it: [0.83, 0.83, 0.04, 0.04]

Now "it" gives 83% to chai and its new vector is mostly thing and drink ([thing, drink, person, pronoun]). In a real model, the query, key and value are learned weight matrices (Q = x·W_Q, K = x·W_K, V = x·W_V), trained by backpropagation like episode 9.

Scaling: divide by √d_k

import math
def softmax(s):
    e = [math.exp(x) for x in s]
    return [round(x / sum(e), 2) for x in e]

scores = [1.0, 2.0, 3.0]
print("small vectors:  ", softmax(scores))
big = [x * 64 for x in scores]          # 4,096 numbers per vector: dot products grow huge
print("big vectors:    ", softmax(big))
print("scaled by sqrt: ", softmax([x / math.sqrt(4096) for x in big]))

Output:

small vectors:   [0.09, 0.24, 0.67]
big vectors:     [0.0, 0.0, 1.0]
scaled by sqrt:  [0.09, 0.24, 0.67]

Long vectors give huge dot products, and softmax turns all-or-nothing [0.0, 0.0, 1.0]: one token grabs everything and learning nearly stops. Dividing by the square root of the key size (√d_k) brings the shares back. The full formula from the 2017 paper is scaled dot-product attention:

Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V

The paper's reason: large dot products push softmax "into regions where it has extremely small gradients".

Multi-head attention

One set of queries and keys asks one kind of question. Multi-head attention runs several attentions side by side, each a head with its own learned Q, K and V (like cricket selectors watching batting, bowling and fielding). One head might link pronouns to things, another match a verb's tense. The original Transformer used h = 8 heads, each 64 numbers wide (d_k = 64), joined back into d_model = 512.

Attention is order-blind

import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}

def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]

def self_attention(words):            # every word attends to every word
    out = {}
    for w in words:
        wts = softmax([sum(a * b for a, b in zip(emb[w], emb[o])) for o in words])
        out[w] = [round(sum(t * emb[o][d] for t, o in zip(wts, words)), 3) for d in range(2)]
    return out

a = self_attention(["raju", "pays", "neha"])
b = self_attention(["neha", "pays", "raju"])
print("Raju pays Neha:", a)
print("Neha pays Raju:", b)
print("Same result?   ", a == b)

Output:

Raju pays Neha: {'raju': [0.576, 0.212], 'pays': [0.333, 0.333], 'neha': [0.212, 0.576]}
Neha pays Raju: {'neha': [0.212, 0.576], 'pays': [0.333, 0.333], 'raju': [0.576, 0.212]}
Same result?    True

Attention compares every word with every word, like a bag of words (episode 10), so "Raju pays Neha" and "Neha pays Raju" give the same vectors.

Positional encoding: seat numbers on a train

On a train In a Transformer
The same passengers the same token embeddings
A seat number on each ticket a position vector for each position
Ticket = passenger + seat input = embedding + position
import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}
def position(p):                      # a unique wave pattern per position
    return [round(math.sin(p), 2), round(math.cos(p), 2)]
def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]
def self_attention(words):
    vecs = [emb[w] + position(p) for p, w in enumerate(words)]   # word + where it is
    out = {}
    for w, v in zip(words, vecs):
        wts = softmax([sum(a * b for a, b in zip(v, o)) for o in vecs])
        out[w] = [round(sum(t * o[d] for t, o in zip(wts, vecs)), 2) for d in range(4)]
    return out
a = self_attention(["raju", "pays", "neha"])
b = self_attention(["neha", "pays", "raju"])
print("raju, as payer:   ", a["raju"])
print("raju, as receiver:", b["raju"])
print("Same result?      ", a == b)

Output:

raju, as payer:    [0.76, 0.07, 0.21, 0.82]
raju, as receiver: [0.76, 0.07, 0.84, -0.16]
Same result?       False

Adding a position vector to each embedding is positional encoding. The 2017 paper used "sine and cosine functions of different frequencies"; many newer models learn their position signals instead.

The causal mask: no peeking ahead

import math
words = ["raju", "made", "chai", "for", "neha"]

def masked_weights(scores, i):        # word i may only look at words 0..i
    s = [x if j <= i else -math.inf for j, x in enumerate(scores)]
    e = [math.exp(x) for x in s]
    return [round(x / sum(e), 2) for x in e]

scores = [1.0, 1.0, 1.0, 1.0, 1.0]
for i, w in enumerate(words):
    print(f"{w:5} looks at {masked_weights(scores, i)}")

Output:

raju  looks at [1.0, 0.0, 0.0, 0.0, 0.0]
made  looks at [0.5, 0.5, 0.0, 0.0, 0.0]
chai  looks at [0.33, 0.33, 0.33, 0.0, 0.0]
for   looks at [0.25, 0.25, 0.25, 0.25, 0.0]
neha  looks at [0.2, 0.2, 0.2, 0.2, 0.2]

Models like GPT write one token at a time, so while training to predict the next token a word must not see the words after it. Scores for later words are set to −∞ before softmax (e^−∞ = 0). This is the causal mask (the paper: "prevent leftward information flow in the decoder to preserve the auto-regressive property").

The Transformer block

tokens → embedding + position → [ multi-head attention → add & norm → feed-forward → add & norm ] × N → better vectors
Part What it does
Multi-head attention tokens share information with each other
Feed-forward network a small neural network (episode 9) works on each token alone
Residual connection adds each step's change to the original vector instead of replacing it
Layer normalization rescales each vector so the numbers stay in a steady range
Stacking a block outputs vectors of the same size, so blocks stack: 6 in the original Transformer, 96 in GPT-3

History: "Attention Is All You Need" (2017)

Fact Value
Paper Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, submitted 12 June 2017 (arXiv 1706.03762)
Idea a model "based solely on attention mechanisms": no RNN, no convolutions
Base model N = 6 layers, d_model = 512, 8 heads, d_ff = 2048
Training base model 12 hours, big model 3.5 days, on 8 NVIDIA P100 GPUs
Results WMT 2014 English-German 28.4 BLEU, English-French 41.8 BLEU

Why it won: an RNN (episode 10) reads one word at a time, each step waiting for the last; attention processes every word at once, which suits GPUs (episode 11).

Glossary

Term Meaning Example
Context the words around a word "gaya tha" around "kal"
Attention a relevance-weighted average of the other tokens' vectors bank → [0.34, 0.69] by the river
Relevance score how much one token matters to another dot product of two vectors
Dot product multiply matching numbers, add them up sum(a * b for a, b in zip(x, y))
Softmax turns scores into positive weights that add up to 1 [1, 2, 3] → [0.09, 0.24, 0.67]
Attention weights softmax of the scores 0.20, 0.35, 0.46
Weighted average each item counts by its weight sum(w * v)
Self-attention every token attends to the tokens of the same text "it" looks at "chai"
Contextual embedding a vector that changes with the surrounding words two vectors for "bank"
Query what a token is looking for "I look for a thing"
Key what a token offers to be matched on "I'm a thing"
Value what a token hands over when chosen chai's vector
Weight matrix learned numbers that turn a vector into Q, K or V W_Q, W_K, W_V
d_k the size of each query/key vector 64 in the original Transformer
Scaled dot-product attention softmax(Q·Kᵀ/√d_k)·V the 2017 formula
Head one attention with its own Q, K, V 8 heads in 2017
Multi-head attention several heads side by side, results joined 8 × 64 = 512
Order-blind gives the same result for any word order "Raju pays Neha" = "Neha pays Raju"
Positional encoding a vector per position added to each embedding sin(p), cos(p)
Causal mask hides later tokens by setting their scores to −∞ "raju" sees only itself
Autoregressive generating one token at a time, each fed back in GPT (episode 15)
Feed-forward network a small dense network applied to each token episode 9
Residual connection output = input + change pencil corrections on notes
Layer normalization rescales a vector to a steady range after each sub-layer
Transformer block attention + feed-forward + residuals + norms stacked N times
Transformer a model made of stacked Transformer blocks GPT, BERT
BLEU a translation quality score 28.4 English-German

Common mistakes

Mistake Code Error
A huge score in softmax math.exp(1000) OverflowError: math range error
Query and key of different sizes zip([1, 2, 3], [1, 2], strict=True) ValueError: zip() argument 2 is shorter than argument 1
More tokens than positions pos_table[3] with only 3 rows IndexError: list index out of range
Forgetting the √d_k scaling big dot products no error; softmax goes all-or-nothing [0.0, 0.0, 1.0]
Forgetting positions attention on "dog bites man" no error; same as "man bites dog"
Note on plain zip zip([1, 2, 3], [1, 2]) in a dot product no error; silently drops the extra number

Fixes: subtract the largest score before math.exp (same weights, no overflow); give every query and key in a head the same size d_k; stay within the context window (episode 16); always scale and always add positions.

Mini project: a pronoun resolver

Extend the Q, K, V demo with a fifth feature female and the sentence "neha met raju and gave him chai". Write query() for "him" so that it looks for a male person and check it gives the most weight to raju.

Practice

  1. In the attention demo, attend "bank" in ["river", "bank", "loan"]. Where does bank end up?
  2. In the Q, K, V demo, change query() so "it" looks for a person. Which word does it find?
  3. In the position demo, run self-attention on "neha pays raju pays" and print every word's vector. Do the two "pays" get the same vector?
  4. Write a softmax that subtracts the largest score first, and test it on [1000.0, 1001.0, 1002.0].
Solutions

1.

import math
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
       "bank": [0.7, 0.7], "loan": [1.0, 0.0]}

def softmax(scores):
    e = [math.exp(s) for s in scores]
    return [x / sum(e) for x in e]

def attend(word, sentence):
    me = emb[word]
    scores = [sum(a * b for a, b in zip(me, emb[w])) for w in sentence]
    weights = softmax(scores)
    new = [sum(wt * emb[w][d] for wt, w in zip(weights, sentence)) for d in range(2)]
    return [round(x, 2) for x in weights], [round(x, 2) for x in new]

print(attend("bank", ["river", "bank", "loan"]))
([0.3, 0.4, 0.3], [0.58, 0.58])

Bank listens to river and loan equally (0.3 each) and stays in the middle [0.58, 0.58]: the sentence really is ambiguous.

2.

import math
#        thing drink person pronoun
emb = {"raju":  [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
       "drank": [0, 0, 0, 0], "it":   [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]
def query(v): return [0, 3 * v[3]]          # now: "I look for a PERSON"
def key(v):   return [v[0], v[2]]           # "I'm a thing" / "I'm a person"
def value(v): return v                      # what I hand over if chosen
q = query(emb["it"])
weights = softmax([dot(q, key(emb[w])) for w in sentence])
for w, wt in zip(sentence, weights):
    print(f"it -> {w:6} {wt:.2f}")
new = [round(sum(wt * value(emb[w])[d] for wt, w in zip(weights, sentence)), 2) for d in range(4)]
print("new it:", new)
it -> raju   0.83
it -> made   0.04
it -> chai   0.04
it -> drank  0.04
it -> it     0.04
new it: [0.04, 0.04, 0.83, 0.04]

3.

import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}

def position(p):                      # a unique wave pattern per position
    return [round(math.sin(p), 2), round(math.cos(p), 2)]

def softmax(s):
    e = [math.exp(x) for x in s]
    return [x / sum(e) for x in e]

def self_attention(words):
    vecs = [emb[w] + position(p) for p, w in enumerate(words)]   # word + where it is
    out = []
    for w, v in zip(words, vecs):
        wts = softmax([sum(a * b for a, b in zip(v, o)) for o in vecs])
        out.append((w, [round(sum(t * o[d] for t, o in zip(wts, vecs)), 2) for d in range(4)]))
    return out

for w, v in self_attention(["neha", "pays", "raju", "pays"]):
    print(f"{w:5}", v)
neha  [0.06, 0.73, 0.21, 0.76]
pays  [0.25, 0.25, 0.58, 0.27]
raju  [0.64, 0.06, 0.73, -0.28]
pays  [0.31, 0.07, 0.46, -0.49]

The two "pays" get different vectors, because they sit in different positions.

4.

import math
def softmax(scores):
    top = max(scores)                       # subtract the largest score first
    e = [math.exp(s - top) for s in scores]
    return [round(x / sum(e), 2) for x in e]

print(softmax([1.0, 2.0, 3.0]))
print(softmax([1000.0, 1001.0, 1002.0]))   # no OverflowError
[0.09, 0.24, 0.67]
[0.09, 0.24, 0.67]

Subtracting the same number from every score doesn't change softmax's weights, but keeps math.exp small.

Quick quiz 1

In "The cat sat on the mat because it was tired", "it" gives cat 0.71, mat 0.12, sat 0.05, others 0.12. What do the weights add up to? A: 0.71 · B: 1.0 · C: any number · D: 100.0

Answer

B: 1.0. Softmax weights are always positive and add up to 1.

Quick quiz 2

Without positional encoding, what does self-attention see in "dog bites man" vs "man bites dog"? A: two different sentences · B: the same words, no order · C: an error · D: only the first word

Answer

B. Attention is order-blind; positions must be added.

Interview questions

  1. What problem does attention solve? Static embeddings give a word one vector everywhere; attention builds a contextual vector from the relevant surrounding tokens.
  2. Explain query, key and value. Each token projects its vector into a query (what it looks for), a key (what it offers) and a value (what it passes on); weights = softmax(Q·Kᵀ/√d_k), output = weights · V.
  3. Why divide by √d_k? Dot products grow with vector size; large scores push softmax into a near one-hot region with tiny gradients.
  4. Why multiple heads? Each head can learn a different relationship (reference, tense, phrase); their outputs are concatenated and projected.
  5. Why do Transformers need positional encoding? Self-attention is permutation-invariant: without positions, word order is lost.
  6. What is the causal mask for? In decoder (GPT-style) models it stops a token from seeing future tokens, so next-token training works and all positions train in parallel.
  7. Why did Transformers replace RNNs? They process all tokens in parallel (fast on GPUs) and connect distant tokens directly instead of through a fading hidden state.

Summary

  • Averaging a sentence turns every word to mush.
  • Attention = a relevance-weighted average; softmax turns scores into weights that add to 1.
  • Query, key, value: what I look for, what I offer, what I hand over ("it" → chai 0.83).
  • Scale by √d_k so softmax isn't all-or-nothing; use many heads for many questions.
  • Attention is order-blind: add positional encoding. GPT-style models add a causal mask.
  • Transformer block = attention + feed-forward + residuals + layer norm, stacked (6 in 2017, 96 in GPT-3).
  • Next episode: what an LLM really is.