SyntaxPilot
Courses/Machine Learning/Lesson 69
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 69 of 90

RNNs and sequences

sequence datarecurrent neural network (RNN)hidden stateunrolling through timebackpropagation through time (BPTT)many-to-onemany-to-manyone-to-manyvanishing gradients in RNNs

Machine Learning Full Course · Episode 69. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a CPU, and the output shown is the real output; training times depend on the machine.

The big idea: sequences (sentences, daily readings, cricket balls) come in an order and in any length. Averaging the items throws the order away, and a fixed-size input layer fits only one length. A recurrent neural network (RNN) reads one time step at a time and keeps a fixed-size hidden state, a running note, updated by the same weights at every step: h_t = tanh(Wx·x_t + Wh·h_(t-1) + b). Drawn step by step (unrolled), it is a deep network with one layer per step, trained with backpropagation through time, and that is exactly why its gradients vanish over long distances.

Sources: Jeffrey Elman, "Finding Structure in Time", 1990 (summary: Swarthmore notes on Elman's simple recurrent network: "Hidden layer activations are copied to a context layer and fed back into the network on the next time step"). Paul Werbos, "Backpropagation through time: what it does and how to do it", 1990 (Zenodo record, DOI 10.1109/5.58337). Yoshua Bengio, Patrice Simard and Paolo Frasconi, "Learning long-term dependencies with gradient descent is difficult", IEEE Transactions on Neural Networks 5(2):157-166, 1994 (bibtex).

Words you need first

  • Sequence: items that come in an order: the words of a review, the rainfall of each day, the balls of an over.
  • Time step: one position in a sequence (word 1, word 2, ...), even when it is not really time.
  • One-hot vector (Episode 16): a row with a single 1 marking which word it is: "good" = [0, 0, 0, 1, 0, 0] in our 6-word vocabulary.
  • Vocabulary: the list of all words the model knows; here ["the", "movie", "was", "good", "bad", "not"].
  • tanh (Episode 56): an activation that squashes any number into −1 ... 1.
  • Backpropagation and the chain rule (Episode 57): gradients flow backwards, multiplied layer by layer.
  • Vanishing gradient (Episode 66): the learning signal shrinking towards zero on its way back through many layers.

The story: reading a review

"The movie was good, not bad" is a happy review; "the movie was bad, not good" is an unhappy one. Same words, different order. And reviews have any length. Real-life example for the fix: reading with a running note. You read one word at a time; you keep a short note in your head ("so far good... now a not...") and update it after each word; and you use the same reading habit for every word, so any length works.

Two obvious attempts that fail

import torch, torch.nn as nn
torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
onehot = lambda s: torch.eye(6)[[vocab.index(w) for w in s.split()]]   # one row per word
model = nn.Linear(6, 1)                     # obvious attempt 1: average the words, then score
score = lambda s: model(onehot(s).mean(0)).item()
a = "the movie was good not bad"            # a positive review
b = "the movie was bad not good"            # a negative review
print(f"{a!r}: {score(a):+.4f}")
print(f"{b!r}: {score(b):+.4f}")
print("same score?", score(a) == score(b))
fixed = nn.Linear(6 * 6, 1)                 # obvious attempt 2: glue exactly 6 words together
print(f"6 words: {fixed(onehot(a).flatten()).item():+.4f}")
print("4 words:", fixed(onehot("the movie was good").flatten()).item())

Output:

'the movie was good not bad': -0.2856
'the movie was bad not good': -0.2856
same score? True
6 words: +0.4613
RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1)
  • Averaging gives exactly the same score (−0.2856) for both reviews: the average of the same rows is the same, whatever the order. No training can fix that.
  • Gluing a fixed number of words (6 × 6 = 36 inputs) works for 6 words, but a 4-word review gives 24 inputs and crashes: mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1).

The RNN cell and the hidden state

h_t = tanh( Wx · x_t  +  Wh · h_(t-1)  +  b )
  • x_t: this step's input (a one-hot word, a temperature, ...).
  • h_(t-1): the hidden state so far, a vector of fixed size (the running note); h_0 is all zeros.
  • h_t: the new hidden state. Recurrent means it is fed back in at the next step.
  • Wx, Wh, b: the weights, the same at every step (shared weights), so any length works.

Elman's 1990 simple recurrent network did exactly this: the hidden activations are copied and fed back as extra input at the next time step.

Unrolling through time

Drawing the loop as one copy of the cell per step is unrolling through time:

h0 = 0 → [cell] → h1 → [cell] → h2 → [cell] → h3 → answer
           ↑ x1          ↑ x2          ↑ x3

Every copy shares the same weights; unrolled, an RNN is a deep network with one layer per time step.

import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=3, hidden_size=2, batch_first=True)   # PyTorch's RNN layer
Wx, Wh = rnn.weight_ih_l0, rnn.weight_hh_l0                   # its weights: input, hidden
b = rnn.bias_ih_l0 + rnn.bias_hh_l0
x = torch.randn(1, 4, 3)                     # 1 sequence, 4 time steps, 3 numbers per step
h = torch.zeros(2)                           # the memory starts empty
with torch.no_grad():
    for t in range(4):                       # unrolled: the SAME weights at every step
        h = torch.tanh(Wx @ x[0, t] + Wh @ h + b)
        print(f"step {t + 1}: hidden state = {h.numpy().round(4)}")
    out, h_last = rnn(x)
print("nn.RNN, last step:   ", h_last[0, 0].detach().numpy().round(4))
print("output shape (batch, steps, hidden):", tuple(out.shape))
print("weights:", sum(p.numel() for p in rnn.parameters()), "for ANY sequence length")

Output:

step 1: hidden state = [-0.4114 -0.9595]
step 2: hidden state = [ 0.9746 -0.9355]
step 3: hidden state = [-0.0376 -0.8954]
step 4: hidden state = [ 0.8791 -0.8479]
nn.RNN, last step:    [ 0.8791 -0.8479]
output shape (batch, steps, hidden): (1, 4, 2)
weights: 14 for ANY sequence length

Our hand-written loop gives exactly nn.RNN's hidden state. nn.RNN stores weight_ih_l0 (Wx), weight_hh_l0 (Wh) and two biases: 6 + 4 + 2 + 2 = 14 numbers, for a sequence of any length.

Many-to-one, many-to-many, one-to-many

Pattern Uses Example
Many-to-one the last hidden state only a review → happy or not
Many-to-many every step's hidden state each word → its part of speech (noun, verb, ...)
One-to-many one input, then each output fed back as the next input a photo → a caption; a starting note → a melody
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=6, hidden_size=16, batch_first=True)
x = torch.randn(2, 5, 6)                  # 2 sequences, 5 steps, 6 numbers per step
out, h = rnn(x)                           # out: every step's hidden state, h: the last one
print("out:", tuple(out.shape), "| h:", tuple(h.shape))
one = nn.Linear(16, 1)                    # many-to-one: one answer per sequence
print("many-to-one  (review -> score):  ", tuple(one(h[0]).shape))
tag = nn.Linear(16, 3)                    # many-to-many: one answer per step
print("many-to-many (word -> tag):      ", tuple(tag(out).shape))
cell, back = nn.RNNCell(6, 16), nn.Linear(16, 6)    # one-to-many: one input, many outputs
inp, hs, made = torch.randn(1, 6), torch.zeros(1, 16), []
for t in range(5):
    hs = cell(inp, hs); inp = back(hs)    # each output becomes the next input
    made.append(inp)
print("one-to-many  (1 input -> 5 steps):", tuple(torch.stack(made, 1).shape))

Output:

out: (2, 5, 16) | h: (1, 2, 16)
many-to-one  (review -> score):   (2, 1)
many-to-many (word -> tag):       (2, 5, 3)
one-to-many  (1 input -> 5 steps): (1, 5, 6)

With batch_first=True, out is (batch, steps, hidden) = every step's hidden state, and h is (layers, batch, hidden) = the last step only.

Training an RNN: does order help?

3,000 random reviews of 3-8 words; the true label counts +1 per "good", −1 per "bad", and "not" flips the next one, so order matters. Short reviews get empty (all-zero) steps at the start so they fit in one batch of 8 steps: padding (more on padding and packing in Episode 70). A bag of words model (word counts, order lost) against a many-to-one RNN, both trained with 400 full-batch Adam steps:

import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review():                                # a random review and its true label
    total = 0
    while total == 0:
        words = random.choices(vocab, k=random.randint(3, 8)); total, flip = 0, 1
        for w in words:
            if w == "not": flip = -1
            elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
    return words, int(total > 0)
def encode(words):                           # one-hot rows; short reviews get empty
    x = torch.zeros(8, 6)                    # (all-zero) steps at the START
    for t, w in enumerate(words): x[8 - len(words) + t, vocab.index(w)] = 1
    return x
data = [review() for _ in range(3000)]
X = torch.stack([encode(w) for w, _ in data]); y = torch.tensor([float(l) for _, l in data])
Xtr, ytr, Xte, yte = X[:2500], y[:2500], X[2500:], y[2500:]
class RNNClassifier(nn.Module):              # many-to-one: read every word, answer once
    def __init__(self):
        super().__init__()
        self.rnn, self.out = nn.RNN(6, 32, batch_first=True), nn.Linear(32, 1)
    def forward(self, x):
        _, h = self.rnn(x)                   # h = the hidden state after the LAST step
        return self.out(h[0]).squeeze(1)
bag = nn.Linear(6, 1)
models = {"bag of words": (bag, lambda x: bag(x.sum(1)).squeeze(1)), "RNN": (RNNClassifier(),) * 2}
for name, (model, f) in models.items():
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    for step in range(400):
        loss = nn.functional.binary_cross_entropy_with_logits(f(Xtr), ytr)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        print(f"{name:12}: test accuracy {((f(Xte) > 0).float() == yte).float().mean():.1%}")
rnn = models["RNN"][0]
with torch.no_grad():
    for s in ("the movie was good not bad", "the movie was bad not good"):
        p = torch.sigmoid(rnn(encode(s.split()).unsqueeze(0))).item()
        print(f"RNN: {s!r} -> {p:.2f} chance positive")

Output:

bag of words: test accuracy 66.8%
RNN         : test accuracy 99.8%
RNN: 'the movie was good not bad' -> 1.00 chance positive
RNN: 'the movie was bad not good' -> 0.00 chance positive

The bag of words cannot see where the "not" is (66.8%); the RNN gets 99.8% and scores our two reviews correctly.

Backpropagation through time (BPTT)

Unroll the RNN and run ordinary backpropagation on the unrolled network: the error flows from the last step back to the first. Because the weights are shared, each weight's gradient is the sum of its gradients at every time step. Werbos (1990) described this as backpropagation through time. Each step back multiplies the gradient by Wh and by tanh's slope (at most 1), so over many steps it can shrink (vanish) or grow (explode).

import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=1, hidden_size=16, batch_first=True)
for steps in (5, 10, 20, 50, 100):
    x = torch.randn(1, steps, 1, requires_grad=True)
    out, h = rnn(x)
    h.sum().backward()                        # a "loss" at the very last step
    g = x.grad[0, :, 0].abs()
    print(f"{steps:3} steps: gradient at the last step {g[-1]:.1e}, at step 1 {g[0]:.1e}")

Output:

  5 steps: gradient at the last step 8.9e-02, at step 1 5.5e-02
 10 steps: gradient at the last step 8.6e-02, at step 1 1.0e-03
 20 steps: gradient at the last step 5.8e-02, at step 1 9.4e-06
 50 steps: gradient at the last step 6.6e-02, at step 1 5.2e-12
100 steps: gradient at the last step 7.5e-02, at step 1 2.1e-23

At the last step the gradient stays around 0.06-0.09; at step 1 it falls from 5.5e-02 (5 steps) to 2.1e-23 (100 steps).

Vanishing gradients in RNNs

The learning signal for early steps shrinks towards zero, so a plain RNN struggles to connect something at the start of a sequence with an answer at the end: a long-term dependency. Bengio, Simard and Frasconi (1994) showed that learning long-term dependencies with gradient descent is difficult. The opposite problem, exploding gradients, is usually handled with gradient clipping: rescale the gradient whenever its size (norm) is above a limit (Practice 3).

The memory test: the label is the sign of the first number; every later number is just noise. Three fresh RNNs per length, 200 full-batch Adam steps each:

import torch, torch.nn as nn
torch.manual_seed(69)
def task(steps, n=1000):                      # remember the FIRST number's sign; then only noise
    x = torch.randn(n, steps, 1)
    y = (x[:, 0, 0] > 0).float()
    return x, y
class Net(nn.Module):
    def __init__(self):
        super().__init__(); self.rnn, self.out = nn.RNN(1, 16, batch_first=True), nn.Linear(16, 1)
    def forward(self, x): return self.out(self.rnn(x)[1][0]).squeeze(1)
for steps in (10, 30, 60, 100):
    accs = []
    for run in range(3):                      # 3 fresh networks per length, then the average
        X, y = task(steps); Xt, yt = task(steps, 500)
        net = Net(); opt = torch.optim.Adam(net.parameters(), lr=0.01)
        for i in range(200):
            loss = nn.functional.binary_cross_entropy_with_logits(net(X), y)
            opt.zero_grad(); loss.backward(); opt.step()
        with torch.no_grad(): accs.append(((net(Xt) > 0).float() == yt).float().mean().item())
    print(f"remember the first of {steps:3} numbers: test accuracy",
          " ".join(f"{a:.0%}" for a in accs), f"| average {sum(accs) / 3:.1%}")

Output:

remember the first of  10 numbers: test accuracy 91% 99% 99% | average 96.4%
remember the first of  30 numbers: test accuracy 98% 50% 48% | average 65.1%
remember the first of  60 numbers: test accuracy 96% 49% 96% | average 80.3%
remember the first of 100 numbers: test accuracy 52% 51% 50% | average 51.0%

At 10 steps all three networks learn the task (91-99%). At 30 and 60 steps some runs learn it and others stay stuck at about 50%, a coin flip; at 100 steps all three are stuck (52%, 51%, 50%). The signal from step 1 is too weak for gradient descent to find reliably. Episode 70's LSTM and GRU add gates to fix exactly this.

Common mistakes

import torch, torch.nn as nn
rnn = nn.RNN(input_size=6, hidden_size=16, batch_first=True)
x = torch.randn(2, 5, 4)                  # but each step has only 4 numbers
out, h = rnn(x)
RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 4
Mistake Exact message / symptom Why Fix
Averaging the words the same score (−0.2856) for "good not bad" and "bad not good" order is thrown away read the sequence in order (RNN)
A fixed-size input layer RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1) only one length fits one cell reused at every step
Wrong number of features per step RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 4 the last dimension of x ≠ input_size input_size = x.shape[-1]
Forgetting batch_first=True batch and time silently swapped the default layout is (steps, batch, features) batch_first=True or x.transpose(0, 1)
Using out when you meant h (or the reverse) wrong shapes: (batch, steps, hidden) vs (1, batch, hidden) out = every step, h = last step many-to-one: h[-1]; many-to-many: out
Padding at the end for a many-to-one model the answer is read after many empty steps the last step is padding pad at the start, or pack sequences (Episode 70)
Expecting long memory from a plain RNN runs stuck at 50% when remembering far back vanishing gradients LSTM / GRU (Episode 70)

Mini project: a word-by-word sentiment tracker (many-to-many)

Label every step with the sign of the running total so far (negative / neutral / positive) and train a many-to-many tagger that uses every hidden state:

import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review():                     # label EVERY step: the running total's sign so far (+1 / -1 / 0)
    words = random.choices(vocab, k=8); total, flip, tags = 0, 1, []
    for w in words:
        if w == "not": flip = -1
        elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
        tags.append(0 if total < 0 else 1 if total == 0 else 2)     # 3 classes: negative, neutral, positive
    return words, tags
data = [review() for _ in range(3000)]
X = torch.stack([torch.eye(6)[[vocab.index(w) for w in ws]] for ws, _ in data])
Y = torch.tensor([t for _, t in data])                              # (3000, 8): one label per step
class Tagger(nn.Module):                                            # many-to-many
    def __init__(self):
        super().__init__(); self.rnn, self.out = nn.RNN(6, 32, batch_first=True), nn.Linear(32, 3)
    def forward(self, x): return self.out(self.rnn(x)[0])           # use EVERY step's hidden state
net = Tagger(); opt = torch.optim.Adam(net.parameters(), lr=0.01)
for step in range(300):
    loss = nn.functional.cross_entropy(net(X[:2500]).reshape(-1, 3), Y[:2500].reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    pred = net(X[2500:]).argmax(2)
print("per-step test accuracy:", f"{(pred == Y[2500:]).float().mean():.1%}")
print("words:", data[2500][0]); print("true :", data[2500][1]); print("pred :", pred[0].tolist())

Output:

per-step test accuracy: 99.2%
words: ['was', 'movie', 'the', 'bad', 'not', 'good', 'good', 'the']
true : [1, 1, 1, 0, 0, 0, 0, 0]
pred : [1, 1, 1, 0, 0, 0, 0, 0]

Practice

1. Run the cell loop for 10 steps by hand and compare with nn.RNN.

Solution
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=3, hidden_size=2, batch_first=True)
Wx, Wh, b = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0 + rnn.bias_hh_l0
x = torch.randn(1, 10, 3)                    # 10 time steps this time
h = torch.zeros(2)
with torch.no_grad():
    for t in range(10):
        h = torch.tanh(Wx @ x[0, t] + Wh @ h + b)
    out, h_last = rnn(x)
print("our loop, step 10:", h.numpy().round(4))
print("nn.RNN,   step 10:", h_last[0, 0].numpy().round(4))
print("same?", torch.allclose(h, h_last[0, 0], atol=1e-6))

Output:

our loop, step 10: [ 0.5263 -0.8005]
nn.RNN,   step 10: [ 0.5263 -0.8005]
same? True

2. Make the review task many-to-many: label every step. See the mini project above.

3. Clip gradients with torch.nn.utils.clip_grad_norm_.

Solution
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(1, 16, batch_first=True)
with torch.no_grad(): rnn.weight_hh_l0.mul_(4)        # big recurrent weights: gradients can explode
x = torch.randn(32, 50, 1)
out, h = rnn(x)
(out ** 2).sum().backward()
before = torch.nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0)   # returns the norm BEFORE clipping
after = torch.sqrt(sum((p.grad ** 2).sum() for p in rnn.parameters()))
print(f"gradient norm before clipping: {before:.1f}")
print(f"gradient norm after clipping:  {after:.3f}")

Output:

gradient norm before clipping: 96325.1
gradient norm after clipping:  1.000

clip_grad_norm_ returns the total norm before clipping and rescales every gradient so the total norm is at most max_norm. Call it between loss.backward() and opt.step().

4. Try hidden_size 4 and 64 on the review task.

Solution
import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review():                                # a random review and its true label
    total = 0
    while total == 0:
        words = random.choices(vocab, k=random.randint(3, 8)); total, flip = 0, 1
        for w in words:
            if w == "not": flip = -1
            elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
    return words, int(total > 0)
def encode(words):                           # one-hot rows; short reviews get empty
    x = torch.zeros(8, 6)                    # (all-zero) steps at the START
    for t, w in enumerate(words): x[8 - len(words) + t, vocab.index(w)] = 1
    return x
data = [review() for _ in range(3000)]
X = torch.stack([encode(w) for w, _ in data]); y = torch.tensor([float(l) for _, l in data])
Xtr, ytr, Xte, yte = X[:2500], y[:2500], X[2500:], y[2500:]
class RNNClassifier(nn.Module):
    def __init__(self, hidden):
        super().__init__()
        self.rnn, self.out = nn.RNN(6, hidden, batch_first=True), nn.Linear(hidden, 1)
    def forward(self, x):
        return self.out(self.rnn(x)[1][0]).squeeze(1)
for hidden in (4, 64):
    torch.manual_seed(69); model = RNNClassifier(hidden)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    for step in range(400):
        loss = nn.functional.binary_cross_entropy_with_logits(model(Xtr), ytr)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        acc = ((model(Xte) > 0).float() == yte).float().mean()
    n = sum(p.numel() for p in model.parameters())
    print(f"hidden_size {hidden:2}: {n:5} weights, test accuracy {acc:.1%}")

Output:

hidden_size  4:    53 weights, test accuracy 84.0%
hidden_size 64:  4673 weights, test accuracy 98.2%

Quick quiz

rnn = nn.RNN(input_size=10, hidden_size=32, batch_first=True), x = torch.randn(8, 20, 10), out, h = rnn(x). out.shape? (a) (8, 32) (b) (8, 20, 32) (c) (1, 8, 32) (d) (20, 8, 10)

Answer

(b) (8, 20, 32): every step's hidden state. (c) is h, the last step only.

  1. Why can't averaging word vectors tell "good not bad" from "bad not good"?
    AnswerThe average of the same rows is the same in any order.
  2. What is the hidden state?
    AnswerA fixed-size vector that summarises everything read so far; updated at every step and fed into the next.
  3. How many weights does nn.RNN(3, 2) have?
    Answer14: 2×3 + 2×2 + 2 + 2.
  4. What is backpropagation through time?
    AnswerOrdinary backpropagation on the unrolled RNN; shared weights sum their gradients over all steps.
  5. Why do gradients vanish in an RNN?
    AnswerEach step back multiplies by Wh and tanh's slope (≤ 1); many such factors shrink the gradient towards 0.

Interview questions

  1. What makes RNNs different from feed-forward networks? They process a sequence step by step with a hidden state carried forward and weights shared across time, so they handle any length and depend on order.
  2. Write the vanilla RNN update. h_t = tanh(W_xh x_t + W_hh h_(t-1) + b); outputs y_t = W_hy h_t + c when needed.
  3. Explain BPTT and truncated BPTT. BPTT backpropagates through the unrolled sequence and sums gradients for shared weights; truncated BPTT only goes back a fixed number of steps to save memory and time.
  4. Why are long-term dependencies hard for vanilla RNNs? Gradients through many steps are products of many Jacobians; they vanish (or explode), so the effect of early inputs is hard to learn (Bengio, Simard, Frasconi, 1994).
  5. How do you handle exploding gradients? Gradient clipping by norm (clip_grad_norm_), smaller learning rates, careful initialisation.
  6. Give examples of many-to-one, many-to-many and one-to-many. Sentiment classification; part-of-speech tagging; image captioning or music generation.

Glossary

Term Meaning (plain words + example)
Sequence data Items in an order: words of a review, daily rainfall, cricket balls
Time step One position in a sequence: word 1, word 2, ...
Vocabulary The list of words a model knows: our 6 words
One-hot vector A row with one 1 marking the item: "good" = [0, 0, 0, 1, 0, 0]
Bag of words Representing text by word counts only; order is lost
Recurrent neural network (RNN) A network that reads one step at a time and feeds its hidden state back in
Hidden state The fixed-size running note h_t carried from step to step
RNN cell The one-step update h_t = tanh(Wx·x_t + Wh·h_(t-1) + b)
Shared weights The same Wx, Wh, b used at every time step
Simple recurrent network (Elman network) Elman's 1990 RNN: hidden activations copied back as input at the next step
Unrolling through time Drawing the RNN as one copy of the cell per step: a deep network
nn.RNN PyTorch's RNN layer; returns out (every step) and h (last step)
batch_first=True Input laid out as (batch, steps, features) instead of (steps, batch, features)
Many-to-one Read a sequence, answer once: review → sentiment
Many-to-many An answer at every step: word → tag
One-to-many One input, a sequence out, each output fed back in: photo → caption
Padding Filling short sequences with empty (all-zero) steps so a batch has one length
Backpropagation through time (BPTT) Backpropagation on the unrolled RNN; shared weights sum their gradients over steps
Vanishing gradient (in RNNs) The gradient reaching early steps shrinks towards 0: 2.1e-23 at 100 steps back
Long-term dependency An answer that depends on something many steps earlier
Exploding gradient The gradient growing huge through many steps
Gradient clipping Rescaling the gradient when its norm is above a limit: clip_grad_norm_

Summary

  • Sequences have order and any length: averaging loses the order (same score −0.2856), fixed inputs crash on other lengths.
  • An RNN keeps a hidden state and applies the same cell at every step; our loop matched nn.RNN exactly with 14 weights.
  • Unrolled, it is a deep network; wire it many-to-one, many-to-many or one-to-many.
  • On order-dependent reviews: bag of words 66.8%, RNN 99.8%.
  • BPTT trains it; the gradient at step 1 falls to 2.1e-23 at 100 steps: vanishing gradients, so long memory is unreliable.

Next episode: LSTMs and GRUs: the cell state and forget, input and output gates, GRUs, bidirectional and stacked RNNs, padding and packing.