SyntaxPilot
Courses/Machine Learning/Lesson 73
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 73 of 90

Seq2seq and attention

encoder-decoder architecturesequence-to-sequence (seq2seq)teacher forcinginformation bottleneckattention mechanismalignment scoresattention weightscontext vectorBahdanau and Luong attention

Machine Learning Full Course · Episode 73. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output.

The big idea: many problems take one sequence in and give a different sequence out (translation, summarising, answering). Word-by-word lookup fails because word order and length change. An encoder-decoder (seq2seq) model reads the whole input with one RNN (the encoder) and writes the output one token at a time with another (the decoder), trained with teacher forcing. But squeezing the whole input into one vector is an information bottleneck: our model reversed 5 letters with 89.8% of words exactly right and 25 letters with 0.0%. Attention lets the decoder look at every encoder state at every step: score each state, turn the scores into weights with softmax, and mix the states into a context vector. With attention the same model got 93.2% of 25-letter words exactly right.

Sources: Ilya Sutskever, Oriol Vinyals and Quoc V. Le (Google), "Sequence to Sequence Learning with Neural Networks", 2014 (arXiv 1409.3215: a multilayer LSTM maps the input to "a vector of a fixed dimensionality" and a second LSTM decodes it; four-layer LSTMs; WMT'14 English to French, BLEU 34.8 against 33.3 for a phrase-based SMT baseline; reversing the source sentence made optimisation easier). Dzmitry Bahdanau, KyungHyun Cho and Yoshua Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv 1409.0473: "the use of a fixed-length vector is a bottleneck"; the model learns to "(soft-)search for parts of a source sentence"; the alignment model is a feedforward network; softmax weights; the context vector is a weighted sum of the annotations from a bidirectional RNN). Minh-Thang Luong, Hieu Pham and Christopher D. Manning, "Effective Approaches to Attention-based Neural Machine Translation", 2015 (arXiv 1508.04025, ar5iv: global attention over all source words vs local attention over a subset; score functions dot, general and concat).

Words you need first

  • Embedding (Episode 72): a learned list of numbers for each token; nn.Embedding is a lookup table.
  • RNN, hidden state, GRU (Episodes 69-70): a recurrent network reads a sequence one item at a time and carries a hidden state, a vector that summarises what it has read so far. A GRU (gated recurrent unit) is an RNN with gates that decide what to keep.
  • Softmax (Episode 23): turns any list of scores into positive numbers that add up to 1. softmax([1.8, 0.1]) = [0.85, 0.15].
  • Cross-entropy loss (Episodes 22-23): small when the model gives high probability to the right answer.
  • Dot product (Episode 3): multiply matching numbers and add: [1, 2] · [3, 4] = 11. Bigger when two vectors point the same way.
  • Token: one unit of a sequence: a word, a piece of a word, or a single character.

The story: word-by-word translation fails

A sequence is a list of items whose order matters. Translation is a sequence-to-sequence problem: English sentence in, French sentence out. The obvious first attempt is a dictionary:

fr = {"the": "le", "black": "noir", "cat": "chat", "i": "je", "love": "aime",
      "you": "te", "drinks": "boit", "tea": "thé", "my": "mon", "friend": "ami"}
def word_by_word(sentence):
    return " ".join(fr[w] for w in sentence.lower().split())
tests = [("the black cat", "le chat noir"),
         ("I love you", "je t'aime"),
         ("my friend drinks tea", "mon ami boit du thé")]
for en, correct in tests:
    guess = word_by_word(en)
    print(f"{en:21} -> {guess:17}", "OK" if guess == correct else "WRONG")
    print(f"{'':21}    correct: {correct}")
print("words in vs words out:", [(len(e.split()), len(c.split())) for e, c in tests])

Output:

the black cat         -> le noir chat      WRONG
                         correct: le chat noir
I love you            -> je aime te        WRONG
                         correct: je t'aime
my friend drinks tea  -> mon ami boit thé  WRONG
                         correct: mon ami boit du thé
words in vs words out: [(3, 3), (3, 2), (4, 5)]

Two things break. Word order: French puts most colours after the noun (le chat noir), and the object pronoun moves before the verb (je t'aime). Length: 3 English words became 2 French words; 4 became 5 (du, "some", has no English word here). A translator has to read the whole sentence first, then write.

Real-life example: an interpreter at a meeting

At a meeting In the model
She listens to the whole sentence before speaking Encoder: reads the whole input
She holds what it means in her head Summary vector: the encoder's last hidden state
She speaks in French, word by word, in French order Decoder: writes the output one token at a time

The encoder-decoder (seq2seq) architecture

input tokens ──► [ Encoder RNN ] ──► summary vector h ──► [ Decoder RNN ] ──► output tokens, one per step
                                                         ▲            │
                                                         └── previous output token
  • Encoder: an RNN (here a GRU) that reads the input tokens one by one. Its final hidden state is a summary of the whole input.
  • Decoder: a second RNN that starts from that summary and, at each step, predicts the next output token from the previous one.
  • Start token <s>: a special token that tells the decoder to begin. An end token </s> lets it decide when to stop, so the output length can differ from the input length.
  • Seq2seq = sequence to sequence = this whole design. Sutskever, Vinyals and Le (2014) used four-layer LSTMs and scored BLEU 34.8 on WMT'14 English to French. BLEU (bilingual evaluation understudy) is a score from 0 to 100 measuring how many word sequences a machine translation shares with human translations.

Teacher forcing, greedy decoding and exposure bias

In class In training
Write the next letter of "chai" after the letters before it A decoder step: predict the next token
The teacher corrects each letter before you continue Teacher forcing: feed the TRUE previous token
In the exam there is no teacher; one slip and you build on it Exposure bias: at test time it reads its own guesses
target = ["c", "h", "a", "i"]
dec_in = ["<s>"] + target[:-1]      # teacher forcing: the TRUE previous letter
for step, (x, y) in enumerate(zip(dec_in, target)):
    print(f"step {step}: decoder reads {x!r:6} -> must predict {y!r}")
model_guess = ["k", "h", "a", "i"]       # suppose step 0 is wrong at test time
print("training input :", dec_in)
print("test-time input:", ["<s>"] + model_guess[:-1])
print("  ('k' was its own wrong guess, and it leaks in)")
step 0: decoder reads '<s>'  -> must predict 'c'
step 1: decoder reads 'c'    -> must predict 'h'
step 2: decoder reads 'h'    -> must predict 'a'
step 3: decoder reads 'a'    -> must predict 'i'
training input : ['<s>', 'c', 'h', 'a']
test-time input: ['<s>', 'k', 'h', 'a']
  ('k' was its own wrong guess, and it leaks in)
  • Teacher forcing: during training, the decoder's input at step t is the true token from step t−1 (the target shifted right by one, with <s> in front). All steps can be trained in parallel because every input is known.
  • Greedy decoding: at test time there is no true target, so we pick the most likely token, feed it back as the next input, and repeat. (Beam search keeps the few best partial outputs instead of just one.)
  • Exposure bias: the model only ever saw correct previous tokens in training, so its own early mistakes at test time put it in situations it never practised.

Seq2seq in PyTorch, and the bottleneck measured

The task: reverse a string of letters. It has the same shape as translation, and it is hard for a summary vector: to write the first output letter, the decoder needs the last input letter. The model (attention is switched off by attn=False):

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(73); V, H, S = 27, 64, 26          # letters a..z = 0..25, start token = 26
class Seq2Seq(nn.Module):
    def __init__(s, attn):
        super().__init__(); s.attn = attn; s.emb = nn.Embedding(V, 32)
        s.enc = nn.GRU(32, H, batch_first=True); s.dec = nn.GRU(32, H, batch_first=True)
        s.out = nn.Linear(2 * H if attn else H, 26)
    def forward(s, src, dec_in):
        enc_states, h = s.enc(s.emb(src))              # h = the ONE summary vector
        dec_states, _ = s.dec(s.emb(dec_in), h)
        if not s.attn: return s.out(dec_states), None
        scores = dec_states @ enc_states.transpose(1, 2)    # Luong dot scores
        weights = F.softmax(scores, dim=-1)                 # attention weights
        context = weights @ enc_states                      # context vectors
        return s.out(torch.cat([dec_states, context], -1)), weights
def batch(n, L):
    src = torch.randint(26, (n, L)); tgt = src.flip(1)      # task: reverse the letters
    dec_in = torch.cat([torch.full((n, 1), S), tgt[:, :-1]], 1)  # teacher forcing
    return src, tgt, dec_in

Training and testing (cross-entropy with teacher forcing for 1,500 steps of 64 strings; testing on 1,000 new strings with greedy decoding):

def train(L, attn, steps=1500):
    m = Seq2Seq(attn); opt = torch.optim.Adam(m.parameters(), 3e-3)
    for _ in range(steps):
        src, tgt, dec_in = batch(64, L)
        loss = F.cross_entropy(m(src, dec_in)[0].reshape(-1, 26), tgt.reshape(-1))
        opt.zero_grad(); loss.backward(); opt.step()
    return m
@torch.no_grad()
def greedy(m, src):                                    # no teacher: feed its own guesses
    out = torch.full((len(src), 1), S)
    for _ in range(src.shape[1]):
        logits, w = m(src, out); out = torch.cat([out, logits[:, -1:].argmax(-1)], 1)
    return out[:, 1:], w
for L, attn in [(5, False), (25, False), (25, True)]:
    m = train(L, attn); src, tgt, _ = batch(1000, L)
    pred, w = greedy(m, src)
    letters, words = (pred == tgt).float().mean(), (pred == tgt).all(1).float().mean()
    print(f"length {L:2} attention={attn!s:5} letters right {letters:.1%}  whole word right {words:.1%}")
word = "learningtoreversewithgrus"; src = torch.tensor([[ord(c) - 97 for c in word]])
pred, w = greedy(m, src)
print("input :", word); print("output:", "".join(chr(97 + i) for i in pred[0]))
print("output letter i looks most at input:", w[0].argmax(-1)[:10].tolist(), "...")
print("weights of output letter 0, last 4 inputs:", [round(x, 2) for x in w[0, 0, -4:].tolist()])

Output:

length  5 attention=False letters right 96.0%  whole word right 89.8%
length 25 attention=False letters right 18.7%  whole word right 0.0%
length 25 attention=True  letters right 97.5%  whole word right 93.2%
input : learningtoreversewithgrus
output: surghtiwesreverotgninrael
output letter i looks most at input: [24, 24, 23, 22, 21, 20, 19, 18, 17, 16] ...
weights of output letter 0, last 4 inputs: [0.0, 0.0, 0.0, 1.0]

Without attention: 5 letters → 96.0% of letters and 89.8% of whole words right. 25 letters → 18.7% of letters (random guessing among 26 letters gives about 3.8%) and 0.0% of whole words. Same model, same training.

Why it fails: the information bottleneck

Everything the decoder knows about the input is one vector of 64 numbers (h). Five letters fit; 25 letters in their exact order do not. An information bottleneck is a narrow point that all the information must squeeze through, like a crowd leaving a hall through one small door. Bahdanau, Cho and Bengio: "the use of a fixed-length vector is a bottleneck". Yet the encoder produced a state for every input token (enc_states), and we threw all but the last away.

Real-life example: the interpreter's notes

At a meeting In the model
A short note for every phrase, not one memory of the whole speech Encoder states: one vector per input token
"Which note matters for my next word?" Alignment scores: how well each input matches now
Read mostly the best note, a little of the others Attention weights → context vector

The attention mechanism

At every decoder step t:

  1. Alignment scores e_t,j: how well encoder state j matches the decoder's current state. Simplest: the dot product.
  2. Attention weights α_t,j = softmax(e_t): all ≥ 0 and adding up to 1 over the input positions.
  3. Context vector c_t = Σ_j α_t,j · enc_state_j: a weighted sum of the encoder states, recomputed at every step.
  4. Predict the next token from [decoder state ; context vector] (; = joined end to end).

Alignment means which input position lines up with which output position. Attention learns it with no alignment labels.

By hand, for "the black cat ." while the decoder is about to write "chat":

import numpy as np
src = ["the", "black", "cat", "."]
enc = np.array([[0.1, 0.0, 0.2],      # one encoder state per source word
                [0.0, 1.0, 0.1],
                [1.0, 0.1, 0.9],
                [0.0, 0.0, 0.1]])
dec = np.array([0.9, 0.0, 1.0])        # decoder state while writing "chat"
scores = enc @ dec                     # 1. alignment scores (dot product)
weights = np.exp(scores) / np.exp(scores).sum()   # 2. softmax -> attention weights
context = weights @ enc                # 3. context vector = weighted sum
for w, s, a in zip(src, scores, weights):
    print(f"{w:6} score {s:4.2f}   weight {a:.2f}")
print("weights add up to", weights.sum().round(2))
print("context:", context.round(2))
the    score 0.29   weight 0.14
black  score 0.10   weight 0.12
cat    score 1.80   weight 0.63
.      score 0.10   weight 0.12
weights add up to 1.0
context: [0.64 0.18 0.62]

"cat" has the largest score (1.8), so it gets 63% of the weight, and the context vector is mostly cat's state.

Seq2seq with attention

Lines 12-15 of the model are the attention: scores = dec_states @ enc_states.transpose(1, 2) has shape (batch, output steps, input tokens); softmax(..., dim=-1) normalises over the input tokens; weights @ enc_states gives one context vector per output step. The same run printed:

  • 25 letters with attention: 97.5% of letters and 93.2% of whole words right (from 0.0%).
  • learningtoreversewithgrus → surghtiwesreverotgninrael, exactly reversed.
  • Output letter 0 puts weight 1.0 on input 24 (the last letter); then 24, 23, 22, 21, … a near-perfect reversed diagonal.

Bahdanau and Luong attention

Bahdanau et al. (2014) Luong, Pham, Manning (2015)
Score additive: v · tanh(W1·h_s + W2·s) (a one-hidden-layer feedforward network) multiplicative: dot h_t · h_s, general h_t · W·h_s (also concat)
Encoder bidirectional RNN (each state knows words before and after) top layer of a stacked LSTM
Variants global (all source words) vs local (a window)

Here h_s is an encoder state, s / h_t the decoder state, W, W1, W2 learned matrices and v a learned vector.

import numpy as np
rng = np.random.default_rng(73)
enc = rng.normal(size=(4, 3)); dec = rng.normal(size=3)   # 4 source states, 1 decoder
Wa = rng.normal(size=(3, 3)); W1, W2 = rng.normal(size=(2, 5, 3)); v = rng.normal(size=5)
softmax = lambda s: np.exp(s - s.max()) / np.exp(s - s.max()).sum()
scores = {"Luong dot     ": enc @ dec,                        # h_t . h_s
          "Luong general ": enc @ (Wa @ dec),                  # h_t . W h_s
          "Bahdanau add  ": np.tanh(enc @ W1.T + W2 @ dec) @ v}  # v . tanh(W1 h_s + W2 s)
for name, s in scores.items():
    print(name, "scores ", s.round(2))
    print(" " * 14, "weights", softmax(s).round(2))
Luong dot      scores  [ 1.66 -1.58 -2.93 -4.68]
               weights [0.95 0.04 0.01 0.  ]
Luong general  scores  [0.4  4.03 0.06 0.01]
               weights [0.03 0.94 0.02 0.02]
Bahdanau add   scores  [1.67 0.89 1.83 1.19]
               weights [0.31 0.14 0.36 0.19]

Untrained, the three score functions disagree; training makes them point at the useful input. All follow the same recipe: score → softmax → weighted sum.

Common mistakes

import torch
torch.manual_seed(73)
dec = torch.randn(2, 3, 8)       # batch 2, 3 output steps, 8 numbers
enc = torch.randn(2, 5, 8)       # batch 2, 5 input words
try:
    scores = torch.bmm(dec, enc)              # forgot to transpose enc
except RuntimeError as e:
    print("RuntimeError:", e)
scores = torch.bmm(dec, enc.transpose(1, 2))  # (2, 3, 5)
wrong = torch.softmax(scores, dim=1)          # softmax over OUTPUT steps
right = torch.softmax(scores, dim=-1)         # softmax over INPUT words
print("wrong: each step's weights sum to", [f"{x:.2f}" for x in wrong[0].sum(-1)])
print("right: each step's weights sum to", [f"{x:.2f}" for x in right[0].sum(-1)])
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 8] but got: [2, 5].
wrong: each step's weights sum to ['0.99', '2.84', '1.16']
right: each step's weights sum to ['1.00', '1.00', '1.00']
Mistake Exact message / symptom Why Fix
torch.bmm(dec, enc) RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 8] but got: [2, 5]. (2,3,8) × (2,5,8) doesn't fit torch.bmm(dec, enc.transpose(1, 2))
softmax(scores, dim=1) no error; each output step's weights sum to 0.99, 2.84, 1.16 normalised over output steps dim=-1 (over input tokens)
Word-by-word translation le noir chat order and length change encoder-decoder
One summary vector for long inputs 0.0% of 25-letter words right information bottleneck attention
Evaluating with teacher forcing on great scores, poor real outputs real use has no true previous token greedy or beam decoding
Forgetting the start token shift the decoder sees the answer it must predict dec_in = target instead of target shifted right cat([<s>], target[:-1])

Mini project: dates to ISO format

Turn "13 november 2026" into "2026-11-13". Input and output have different lengths and different orders: a real seq2seq task (with attention). Inputs are padded with spaces to 18 characters.

import torch, torch.nn as nn, torch.nn.functional as F, random, datetime
torch.manual_seed(73); random.seed(73)
chars = sorted(set("0123456789- abcdefghijklmnopqrstuvwxyz")); S = len(chars)   # S = start token
ix = {c: i for i, c in enumerate(chars)}; H = 64
def sample():
    d = datetime.date(1950, 1, 1) + datetime.timedelta(random.randrange(40000))
    return f"{d.day} {d.strftime('%B').lower()} {d.year}".ljust(18), d.isoformat()   # "13 november 2026"
class Seq2Seq(nn.Module):
    def __init__(s):
        super().__init__(); s.emb = nn.Embedding(S + 1, 32)
        s.enc = nn.GRU(32, H, batch_first=True); s.dec = nn.GRU(32, H, batch_first=True)
        s.out = nn.Linear(2 * H, S)
    def forward(s, src, dec_in):
        e, h = s.enc(s.emb(src)); d, _ = s.dec(s.emb(dec_in), h)
        w = F.softmax(d @ e.transpose(1, 2), -1)
        return s.out(torch.cat([d, w @ e], -1))
def batch(n):
    pairs = [sample() for _ in range(n)]
    src = torch.tensor([[ix[c] for c in a] for a, _ in pairs]); tgt = torch.tensor([[ix[c] for c in b] for _, b in pairs])
    return src, tgt, torch.cat([torch.full((n, 1), S), tgt[:, :-1]], 1)
m = Seq2Seq(); opt = torch.optim.Adam(m.parameters(), 3e-3)
for step in range(1500):
    src, tgt, dec_in = batch(64)
    loss = F.cross_entropy(m(src, dec_in).reshape(-1, S), tgt.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
@torch.no_grad()
def convert(texts):
    src = torch.tensor([[ix[c] for c in t.ljust(18)] for t in texts]); out = torch.full((len(texts), 1), S)
    for _ in range(10): out = torch.cat([out, m(src, out)[:, -1:].argmax(-1)], 1)
    return ["".join(chars[i] for i in row[1:]) for row in out]
src, tgt, _ = batch(1000)
pred = convert(["".join(chars[i] for i in r) for r in src])
print("test dates exactly right:", f"{sum(p == ''.join(chars[i] for i in t) for p, t in zip(pred, tgt)) / 10:.1f}%")
for t, p in zip(["13 november 2026", "2 october 1969", "26 january 1950"], convert(["13 november 2026", "2 october 1969", "26 january 1950"])):
    print(f"{t:18} -> {p}")

Output:

test dates exactly right: 100.0%
13 november 2026   -> 2026-11-13
2 october 1969     -> 1969-10-02
26 january 1950    -> 1950-01-26

Practice

1. Try lengths 10 and 40, with and without attention. Then print the attention weights for one 8-letter word.

Solution
import sys; sys.argv = ["x"]
exec(open("s2s.py").read().split("for L, attn in")[0])          # reuse the model, batch, train, greedy
for L, attn in [(10, False), (10, True), (40, False), (40, True)]:
    m = train(L, attn); src, tgt, _ = batch(1000, L); pred, w = greedy(m, src)
    print(f"length {L:2} attention={attn!s:5} whole word right {(pred == tgt).all(1).float().mean():.1%}")
word = "chaitime"; m = train(8, True); src = torch.tensor([[ord(c) - 97 for c in word]])
pred, w = greedy(m, src)
print("output:", "".join(chr(97 + i) for i in pred[0]))
print("attention weights x10 (rows = output letters, columns = input letters", list(word), "):")
for row in (w[0] * 10).round().int().clamp(max=9).tolist(): print("  ", "".join(str(v) for v in row))
length 10 attention=False whole word right 18.2%
length 10 attention=True  whole word right 97.8%
length 40 attention=False whole word right 0.0%
length 40 attention=True  whole word right 42.3%
output: emitiahc
attention weights x10 (rows = output letters, columns = input letters ['c', 'h', 'a', 'i', 't', 'i', 'm', 'e'] ):
   00000009
   00000090
   00000900
   00009000
   00082000
   00900000
   19000000
   90000000

s2s.py is the whole program from the episode (the model plus the training and testing code). Even 10 letters break the plain model (18.2%), while attention gets 97.8%. At 40 letters attention reaches 42.3% in the same 1,500 steps: longer sequences need more training, but without attention the score stays at 0.0%. The weights (×10, rounded, capped at 9) form a reversed diagonal: output letter i looks at input letter 7−i.

2. Replace the dot score with Luong's "general" score.

Solution

Add a matrix W between the two states: scores = dec_states @ s.W(enc_states).transpose(1, 2).

exec(open("s2s.py").read().split("for L, attn in")[0])          # reuse the model, batch, train, greedy
class General(Seq2Seq):                                          # Luong "general": h_t . W h_s
    def __init__(s, attn):
        super().__init__(attn); s.W = nn.Linear(H, H, bias=False)
    def forward(s, src, dec_in):
        enc_states, h = s.enc(s.emb(src)); dec_states, _ = s.dec(s.emb(dec_in), h)
        weights = F.softmax(dec_states @ s.W(enc_states).transpose(1, 2), dim=-1)
        return s.out(torch.cat([dec_states, weights @ enc_states], -1)), weights
Seq2Seq = General
m = train(25, True); src, tgt, _ = batch(1000, 25); pred, _ = greedy(m, src)
print(f"general score, length 25: whole word right {(pred == tgt).all(1).float().mean():.1%}")
general score, length 25: whole word right 90.2%

The extra matrix lets the decoder ask for something different from what the encoder state literally contains.

3. Make the encoder bidirectional.

Solution

Hint: s.enc = nn.GRU(32, H, batch_first=True, bidirectional=True) gives states of 2H numbers, so project them back with s.proj = nn.Linear(2 * H, H) (enc_states = s.proj(enc_states)), and start the decoder from h[0:1] + h[1:2], the sum of the forward and backward final states, shape (1, batch, H). Compare its accuracy at length 25 with and without attention.

4. Why can't teacher forcing be used at test time?

AnswerBecause at test time the true output is what we are trying to produce; the decoder must use its own previous predictions.

Quick quiz

dec_states: (32, 10, 64)   # batch, output steps, numbers
enc_states: (32, 25, 64)   # batch, input words, numbers
weights = softmax(dec_states @ enc_states.transpose(1, 2), dim=-1)
# weights.shape = ?
Answer

(32, 10, 25): one weight for every (output step, input word) pair, and each row of 25 adds up to 1.

Interview questions

  1. What is the information bottleneck in seq2seq? The encoder compresses the whole input into one fixed-size vector, so long inputs lose information; accuracy drops with length.
  2. How does attention fix it? At each decoder step it computes a fresh context vector as a weighted sum of all encoder states, so the decoder can read any input position directly.
  3. What is teacher forcing, and what is exposure bias? Training the decoder on the true previous token; the mismatch at test time, when it reads its own (possibly wrong) tokens.
  4. Bahdanau vs Luong attention? Additive (feedforward) scores with a bidirectional encoder vs multiplicative (dot/general) scores on the top LSTM layer; Luong also proposed local attention.
  5. Why softmax over the input dimension? Each output step needs a probability distribution over input positions (weights summing to 1).
  6. Greedy vs beam search? Greedy takes the single best token each step; beam search keeps the k best partial outputs and usually finds better whole sequences.
  7. What does an attention weight matrix tell you? Which input positions each output step relied on: a soft alignment you can plot.

Glossary

Term Meaning, with an example
Sequence Items whose order matters; the letters of "chai"
Sequence-to-sequence (seq2seq) A model that maps one sequence to another, possibly of different length; English → French
Encoder-decoder architecture An encoder network reads the input into states; a decoder network produces the output from them
Encoder The network that reads the input; here a GRU over the input letters
Decoder The network that writes the output one token per step
Hidden state The vector an RNN carries between steps, summarising what it has read (64 numbers here)
Summary (context) vector, fixed The encoder's last hidden state handed to the decoder in plain seq2seq
Start token <s> / end token </s> Special tokens that begin the output / mark where it stops
Teacher forcing Training the decoder with the true previous token as input; <s> c h a → c h a i
Greedy decoding At test time, take the most likely token and feed it back in, step by step
Beam search Keep the k best partial outputs at each step instead of one
Exposure bias Train/test mismatch: true previous tokens in training, own guesses at test time
Information bottleneck A narrow point all information must pass through; one 64-number vector for 25 letters
Attention mechanism At each output step, score all encoder states, softmax the scores, mix the states
Alignment Which input position corresponds to which output position; "chat" ↔ "cat"
Alignment score A number for how well an encoder state matches the decoder's need; cat: 1.8
Attention weights Softmax of the scores: ≥ 0, sum to 1; cat: 0.63
Context vector Weighted sum of encoder states with the attention weights; recomputed every step
Bahdanau (additive) attention Score = v · tanh(W1·h_s + W2·s), with a bidirectional encoder (2014)
Luong (multiplicative) attention Score = dot (h_t · h_s) or general (h_t · W·h_s) (2015)
Global vs local attention Attend over all input positions vs a window of them
Bidirectional RNN Two RNNs, left-to-right and right-to-left, joined so each state sees both sides
BLEU A 0-100 score of word-sequence overlap with human translations
torch.bmm Batch matrix multiply: (b, n, m) × (b, m, p) → (b, n, p)

Summary

  • Word-by-word translation fails: order and length change between languages.
  • Seq2seq: an encoder reads, a decoder writes, trained with teacher forcing and run with greedy decoding.
  • One summary vector is a bottleneck: 25 letters, 0.0% words right.
  • Attention: scores → softmax weights → context vector at every step: 93.2%, with a learned reversed diagonal.
  • Bahdanau (additive) and Luong (dot, general) differ only in the score function.

Next: Episode 74, self-attention from scratch: queries, keys and values, scaled dot-product attention, masks and multi-head attention.