Machine Learning Full Course · Episode 57. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output. Timings (milliseconds, minutes) are measured live and change from machine to machine and run to run; the gradients do not.
The big idea: training needs the gradient of the loss for every weight. Nudging each weight and re-running the network (the numerical gradient) costs two forward passes per weight: for a 794,000-weight network that is 1,588,000 forward passes, about an hour, for one step. Instead, multiply local gradients along the network with the chain rule, starting at the loss and walking backwards, reusing every result: backpropagation gets all 794,000 gradients in milliseconds, and a numerical gradient check confirms they agree to 7 digits. With the weight update rule w ← w − lr × gradient, a 2-3-1 network learns XOR in plain Python (0.009, 0.989, 0.989, 0.014), as long as the weights start random: all-zero weights stay identical and get stuck (loss 0.183). In deep sigmoid networks the gradient shrinks layer by layer (0.2 → 0.000004 over 10 layers): vanishing gradients; ReLU keeps it near 0.5.
Source: D. E. Rumelhart, G. E. Hinton and R. J. Williams, "Learning representations by back-propagating errors", Nature 323, 533-536 (October 1986), which describes repeatedly adjusting connection weights to reduce the difference between actual and desired outputs, with hidden units coming to represent important features of the task (record, abstract).
Words you need first
- Neuron, weights, bias, sigmoid, ReLU, hidden layer, forward pass (Episode 56).
- Loss (Episode 8): one number saying how wrong the model is; here the squared error (output − target)².
- Derivative / slope (Episode 6): how much a function's output changes per unit change of its input.
- Chain rule (Episode 6): if a depends on z and z depends on w, then da/dw = da/dz × dz/dw.
- Gradient (Episode 7): the list of partial derivatives of the loss, one per weight.
- Gradient descent, learning rate (Episode 8): step each weight against its gradient, scaled by the learning rate.
- Numerical derivative (Episode 6): (f(w + h) − f(w − h)) / 2h for a tiny h.
The slow way: nudge every weight
- Weight update rule: w_new = w − learning_rate × dLoss/dw. To use it we need dLoss/dw for every weight.
- Obvious way: nudge one weight up by h, measure the loss; nudge it down, measure again; slope = difference / 2h.
import numpy as np, time; rng = np.random.default_rng(57)
X = rng.random((64, 784)) # 64 small 28x28 images, flattened
W1, W2 = rng.normal(0, .03, (784, 1000)), rng.normal(0, .03, (1000, 10)) # a 784 -> 1000 -> 10 network
def loss(): return (np.maximum(0, X @ W1) @ W2).var() # any number that depends on every weight
n = W1.size + W2.size; print(f"weights: {n:,}")
t = time.time()
for i in range(20): # numerical gradient for the first 20 weights
W1[0, i] += 1e-5; up = loss(); W1[0, i] -= 2e-5; down = loss(); W1[0, i] += 1e-5
one = (time.time() - t) / 20; print(f"nudge ONE weight: 2 forward passes, {one * 1000:.1f} ms")
print(f"ALL weights: {2 * n:,} forward passes, about {n * one / 60:.0f} minutes")
print("...for ONE step. Training needs thousands of steps.")
Output (one run):
weights: 794,000
nudge ONE weight: 2 forward passes, 3.6 ms
ALL weights: 1,588,000 forward passes, about 48 minutes
...for ONE step. Training needs thousands of steps.
Why it's slow: every nudge re-runs the whole network from the inputs, although neighbouring weights share almost all that work. Cost: 2 × (number of weights) forward passes per step, and training needs thousands of steps.
Local gradients and the chain rule
Gears on a cycle: one pedal turn turns gear A 3 times, each turn of A turns the wheel 2 times, so one pedal turn turns the wheel 3 × 2 = 6 times. Multiply the local rates along the chain.
- Local gradient: how one step of a computation reacts to its own input (dz/dw, da/dz, ...).
- Chain rule: the gradient through a chain is the product of the local gradients.
One neuron as a chain: w, b → z = w·x + b → a = sigmoid(z) → L = (a − y)².
| Link | Local gradient |
|---|---|
| L = (a − y)² | dL/da = 2(a − y) |
| a = sigmoid(z) | da/dz = a(1 − a) |
| z = w·x + b | dz/dw = x, dz/db = 1 |
dL/dw = dL/da × da/dz × dz/dw. Every local gradient uses values already computed in the forward pass.
import math # one neuron: x -> z = w*x + b -> a = sigmoid(z) -> loss
x, y, w, b = 2.0, 1.0, 0.5, -1.0 # one example (x = 2, target 1) and two parameters
sig = lambda z: 1 / (1 + math.exp(-z))
def loss(w, b): return (sig(w * x + b) - y) ** 2
z = w * x + b; a = sig(z); L = (a - y) ** 2 # forward pass: keep z and a
dL_da = 2 * (a - y) # local gradient of the loss
da_dz = a * (1 - a) # local gradient of sigmoid
dz_dw, dz_db = x, 1 # local gradients of z = w*x + b
print(f"z = {z}, a = {a:.4f}, loss = {L:.4f}")
print(f"chain rule : dL/dw = {dL_da * da_dz * dz_dw:.6f} dL/db = {dL_da * da_dz * dz_db:.6f}")
h = 1e-5 # numerical check: nudge each way and measure
num_w, num_b = (loss(w + h, b) - loss(w - h, b)) / (2 * h), (loss(w, b + h) - loss(w, b - h)) / (2 * h)
print(f"numerical : dL/dw = {num_w:.6f} dL/db = {num_b:.6f}")
Output:
z = 0.0, a = 0.5000, loss = 0.2500
chain rule : dL/dw = -0.500000 dL/db = -0.250000
numerical : dL/dw = -0.500000 dL/db = -0.250000
Backpropagation
- Forward pass: compute and save every z and a (the cache).
- Start at the loss: its gradient with respect to the output.
- Go back one layer at a time: multiply by the layer's local gradient (the activation's slope), then pass it through the weights (× Wᵀ) to the layer below.
- Each weight's gradient = (gradient arriving at its neuron) × (the input that weight saw). In matrix form: dW = (layer input)ᵀ @ (gradient at the layer's z).
Every intermediate gradient is reused for everything beneath it, so nothing is computed twice: all gradients cost about as much as a couple of forward passes. Backpropagation = sending the blame backwards from the loss, layer by layer. Rumelhart, Hinton and Williams made it famous in Nature in 1986.
import numpy as np, time; rng = np.random.default_rng(57)
X = rng.random((64, 784)) # the same network as before
W1, W2 = rng.normal(0, .03, (784, 1000)), rng.normal(0, .03, (1000, 10))
t = time.time()
Z = X @ W1; H = np.maximum(0, Z); O = H @ W2; L = O.var() # forward pass, keep every middle value
dO = 2 * (O - O.mean()) / O.size # backward pass: start at the loss...
dW2 = H.T @ dO; dH = dO @ W2.T # ...through layer 2
dZ = dH * (Z > 0); dW1 = X.T @ dZ # ...through ReLU and layer 1
print(f"backprop: all {W1.size + W2.size:,} gradients in {(time.time() - t) * 1000:.0f} ms")
W1[0, 0] += 1e-5; up = (np.maximum(0, X @ W1) @ W2).var(); W1[0, 0] -= 2e-5
down = (np.maximum(0, X @ W1) @ W2).var(); W1[0, 0] += 1e-5
print(f"weight [0,0]: backprop {dW1[0, 0]:.6e} numerical {(up - down) / 2e-5:.6e}")
Output (one run):
backprop: all 794,000 gradients in 13 ms
weight [0,0]: backprop -2.142100e-04 numerical -2.142100e-04
| Method | Work for all 794,000 gradients | Time here |
|---|---|---|
| Numerical (nudge each weight) | 1,588,000 forward passes | about an hour |
| Backpropagation | 1 forward + 1 backward pass | a few milliseconds |
The numerical gradient check
- Backprop code is easy to get subtly wrong (one missing factor) and the network may still train, just badly.
- Gradient check: compute (L(w + h) − L(w − h)) / 2h with h = 0.00001 for a few weights and compare with backprop. They should agree to many digits; if not, the backward pass has a bug.
- The slow method is the test; the fast method does the training.
Training XOR in plain Python
For each example: forward pass (hidden layer, output), backward pass (d2 at the output, d1 for each hidden neuron, using W2 before it is updated), then the update rule for every weight and bias.
import math, random; random.seed(57)
X, Y, H = [(0, 0), (0, 1), (1, 0), (1, 1)], [0, 1, 1, 0], 3 # XOR, and 3 hidden neurons
sig = lambda z: 1 / (1 + math.exp(-z))
def train(init, lr=1.0, epochs=5000):
W1 = [[init(), init()] for _ in range(H)]; b1 = [0.0] * H; W2 = [init() for _ in range(H)]; b2 = 0.0
for epoch in range(1, epochs + 1):
loss = 0.0
for x, y in zip(X, Y):
h = [sig(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)] # forward: hidden layer
out = sig(sum(W2[j] * h[j] for j in range(H)) + b2); loss += (out - y) ** 2 / 4
d2 = 2 * (out - y) * out * (1 - out) # backward: output layer
d1 = [d2 * W2[j] * h[j] * (1 - h[j]) for j in range(H)] # backward: hidden layer
for j in range(H): # update: w -= lr * gradient
W2[j] -= lr * d2 * h[j]; b1[j] -= lr * d1[j]
W1[j][0] -= lr * d1[j] * x[0]; W1[j][1] -= lr * d1[j] * x[1]
b2 -= lr * d2
if epoch in (1, 100, 1000, 5000): print(f" epoch {epoch:4}: loss {loss:.4f}")
return [round(sig(sum(W2[j] * sig(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)) + b2), 3) for x in X]
for name, init in [("all zeros", lambda: 0.0), ("random", lambda: random.uniform(-1, 1))]: print(name, "->", train(init))
Output:
epoch 1: loss 0.3091
epoch 100: loss 0.2859
epoch 1000: loss 0.1855
epoch 5000: loss 0.1830
all zeros -> [0.016, 0.653, 0.653, 0.654]
epoch 1: loss 0.3286
epoch 100: loss 0.1683
epoch 1000: loss 0.0008
epoch 5000: loss 0.0001
random -> [0.009, 0.989, 0.989, 0.014]
Why all-zero weights fail: symmetry
- Same starting weights → every hidden neuron computes the same value → same gradient → same update → identical for ever.
- Three identical hidden neurons act like one: one straight line, and Episode 56 showed one line can't do XOR. The all-zero network outputs 0.016 for (0,0) but 0.65 for the other three, including (1,1).
- Symmetry breaking: small random starting weights make the neurons different so they can learn different features.
Vanishing gradients
- Backprop multiplies in one local gradient per layer. Sigmoid's slope a(1 − a) is at most 0.25, so 10 layers can shrink the gradient by up to 0.25¹⁰ ≈ 0.000001.
import numpy as np; rng = np.random.default_rng(57)
x = rng.normal(size=(1, 50)) # one input with 50 features
for name, f, df in [("sigmoid", lambda z: 1 / (1 + np.exp(-z)), lambda a, z: a * (1 - a)),
("ReLU", lambda z: np.maximum(0, z), lambda a, z: (z > 0) * 1.0)]:
Ws = [rng.normal(0, np.sqrt(2 / 50), (50, 50)) for _ in range(10)] # 10 layers, 50 neurons each
a, cache = x, []
for W in Ws: z = a @ W; a = f(z); cache.append((a, z)) # forward pass through 10 layers
g, sizes = np.ones_like(a), [] # backward: start with gradient 1 at the top
for W, (a, z) in zip(Ws[::-1], cache[::-1]):
g = g * df(a, z); sizes.append(np.abs(g).mean()); g = g @ W.T # chain rule, one layer down
print(name, "gradient size\n layers 10-6:", *(f"{s:.0e}" for s in sizes[:5]))
print(" layers 5-1:", *(f"{s:.0e}" for s in sizes[5:]))
Output:
sigmoid gradient size
layers 10-6: 2e-01 5e-02 2e-02 5e-03 2e-03
layers 5-1: 5e-04 2e-04 5e-05 1e-05 4e-06
ReLU gradient size
layers 10-6: 4e-01 4e-01 5e-01 5e-01 6e-01
layers 5-1: 4e-01 5e-01 6e-01 6e-01 6e-01
- Vanishing gradient problem: the early layers of a deep network receive tiny gradients and learn very slowly (here 50,000 times smaller at layer 1 with sigmoid).
- Exploding gradients: the opposite, gradients growing huge from layer to layer.
- ReLU's slope is exactly 1 for positive inputs, so the gradient passes through (0.4 to 0.6 at every layer). Careful random initialisation helps too; normalisation layers and residual connections come later in the course.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Training with numerical gradients | ALL weights: 1,588,000 forward passes, about 50-130 minutes (depends on the machine and its load) |
2 passes per weight | backpropagation |
| Not saving forward values | recompute the whole network in the backward pass | backward needs z and a | cache them in the forward pass |
| Updating W2 before computing d1 | gradients slightly wrong, gradient check fails | d1 must use the old W2 | compute all gradients, then update |
| Skipping the gradient check | trains badly with no error message | one wrong term | compare with the numerical gradient |
| All weights start at 0 | epoch 5000: loss 0.1830, outputs [0.016, 0.653, 0.653, 0.654] |
symmetry | small random initial weights |
| Deep sigmoid network | layers 5-1: 5e-04 2e-04 5e-05 1e-05 4e-06 (sigmoid) |
slopes ≤ 0.25 multiply | ReLU, good initialisation |
| Too small a learning rate | lr 0.1: loss 0.0021 after 5000 epochs (vs 0.0001 at lr 1.0) |
tiny steps | tune the learning rate |
Mini project: vectorised backprop with NumPy
The same XOR network, with the whole batch as matrices:
import numpy as np; rng = np.random.default_rng(57)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]); y = np.array([[0], [1], [1], [0]])
W1, b1 = rng.uniform(-1, 1, (2, 3)), np.zeros(3); W2, b2 = rng.uniform(-1, 1, (3, 1)), np.zeros(1)
sig = lambda z: 1 / (1 + np.exp(-z)); lr = 2.0
for epoch in range(1, 5001):
H = sig(X @ W1 + b1); out = sig(H @ W2 + b2) # forward, whole batch
loss = ((out - y) ** 2).mean()
d2 = 2 * (out - y) / len(X) * out * (1 - out) # backward: output layer
d1 = (d2 @ W2.T) * H * (1 - H) # backward: hidden layer
W2 -= lr * H.T @ d2; b2 -= lr * d2.sum(0) # update
W1 -= lr * X.T @ d1; b1 -= lr * d1.sum(0)
if epoch in (1, 1000, 5000): print(f"epoch {epoch}: loss {loss:.4f}")
print("outputs:", out.ravel().round(3))
Output:
epoch 1: loss 0.2505
epoch 1000: loss 0.0026
epoch 5000: loss 0.0002
outputs: [0.016 0.981 0.986 0.014]
Practice
1. XOR with only 2 hidden neurons: does it learn?
Solution
import math, random; random.seed(57)
X, Y = [(0, 0), (0, 1), (1, 0), (1, 1)], [0, 1, 1, 0]
sig = lambda z: 1 / (1 + math.exp(-z))
def train(H=3, lr=1.0, epochs=5000, act=sig, dact=lambda h: h * (1 - h)):
W1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(H)]; b1 = [0.0] * H
W2 = [random.uniform(-1, 1) for _ in range(H)]; b2 = 0.0
for epoch in range(1, epochs + 1):
loss = 0.0
for x, y in zip(X, Y):
h = [act(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)]
out = sig(sum(W2[j] * h[j] for j in range(H)) + b2); loss += (out - y) ** 2 / 4
d2 = 2 * (out - y) * out * (1 - out)
d1 = [d2 * W2[j] * dact(h[j]) for j in range(H)]
for j in range(H):
W2[j] -= lr * d2 * h[j]; b1[j] -= lr * d1[j]
W1[j][0] -= lr * d1[j] * x[0]; W1[j][1] -= lr * d1[j] * x[1]
b2 -= lr * d2
preds = [round(sig(sum(W2[j] * act(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)) + b2), 3) for x in X]
return round(loss, 4), preds
for run in range(5): print("2 hidden neurons, run", run + 1, "-> loss, outputs:", train(H=2))
Output:
2 hidden neurons, run 1 -> loss, outputs: (0.1423, [0.011, 0.989, 0.467, 0.467])
2 hidden neurons, run 2 -> loss, outputs: (0.0001, [0.012, 0.987, 0.989, 0.01])
2 hidden neurons, run 3 -> loss, outputs: (0.1423, [0.011, 0.467, 0.986, 0.467])
2 hidden neurons, run 4 -> loss, outputs: (0.1829, [0.014, 0.653, 0.653, 0.654])
2 hidden neurons, run 5 -> loss, outputs: (0.0001, [0.011, 0.989, 0.989, 0.014])
Two hidden neurons are enough in principle (Episode 56 solved XOR with two), but from some random starts gradient descent gets stuck in a poor solution (a local minimum or flat region). More hidden neurons make that much rarer.
2. Learning rate 0.1 and 5.0.
Solution
With the train function from Practice 1:
for lr in [0.1, 1.0, 5.0]: print(f"lr {lr}: loss, outputs after 5000 epochs:", train(lr=lr))
Output:
lr 0.1: loss, outputs after 5000 epochs: (0.0021, [0.043, 0.957, 0.956, 0.052])
lr 1.0: loss, outputs after 5000 epochs: (0.0001, [0.009, 0.989, 0.987, 0.013])
lr 5.0: loss, outputs after 5000 epochs: (0.0, [0.005, 0.995, 0.995, 0.006])
A small learning rate is still improving after 5,000 epochs; on this tiny problem a large one converges faster. On bigger problems too large a rate makes the loss jump around or blow up.
3. tanh instead of sigmoid in the hidden layer.
Solution
tanh's local gradient is 1 − tanh(z)² = 1 − h². With the train function from Practice 1:
print("tanh hidden:", train(act=math.tanh, dact=lambda h: 1 - h * h))
Output:
tanh hidden: (0.0, [0.005, 0.994, 0.994, 0.006])
4. Gradient-check b1 in the XOR network.
Solution
import math, random; random.seed(57)
X, Y, H = [(0, 0), (0, 1), (1, 0), (1, 1)], [0, 1, 1, 0], 3
sig = lambda z: 1 / (1 + math.exp(-z))
W1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(H)]; b1 = [0.1, -0.2, 0.3]
W2 = [random.uniform(-1, 1) for _ in range(H)]; b2 = 0.0
def loss(): # mean squared error over the 4 examples
L = 0.0
for x, y in zip(X, Y):
h = [sig(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)]
L += (sig(sum(W2[j] * h[j] for j in range(H)) + b2) - y) ** 2 / 4
return L
grad = [0.0] * H # backprop: dL/db1, summed over the examples
for x, y in zip(X, Y):
h = [sig(W1[j][0] * x[0] + W1[j][1] * x[1] + b1[j]) for j in range(H)]
out = sig(sum(W2[j] * h[j] for j in range(H)) + b2)
d2 = 2 * (out - y) * out * (1 - out) / 4
for j in range(H): grad[j] += d2 * W2[j] * h[j] * (1 - h[j])
for j in range(H):
b1[j] += 1e-5; up = loss(); b1[j] -= 2e-5; down = loss(); b1[j] += 1e-5
print(f"b1[{j}]: backprop {grad[j]:.8f} numerical {(up - down) / 2e-5:.8f}")
Output:
b1[0]: backprop 0.02079673 numerical 0.02079673
b1[1]: backprop 0.00531202 numerical 0.00531202
b1[2]: backprop 0.01649946 numerical 0.01649946
Backprop and the numerical gradient agree to all printed digits.
Quick quiz
z = 3w, a = 2z, L = 5a. What is dL/dw?
Answer
30. Multiply the local gradients along the chain: 5 × 2 × 3.
More questions:
- How many forward passes does a numerical gradient need for a network with 1,000 weights?
Answer
2,000 (two per weight, using the central difference). - Why must the forward pass save z and a?
Answer
The local gradients in the backward pass (for example a(1 − a)) are computed from them. - What is the gradient of a weight connecting input x to a neuron whose z-gradient is d?
Answer
d × x. - Why do all-zero initial weights fail?
Answer
All hidden neurons stay identical (same outputs, same gradients, same updates), so they act like one neuron. - Why does sigmoid make gradients vanish in deep networks?
Answer
Its slope is at most 0.25, and backprop multiplies one slope per layer.
Interview questions
- Explain backpropagation. Forward pass storing activations; then apply the chain rule from the loss backwards, layer by layer, computing each layer's gradient from the next layer's, so all parameter gradients come from one backward pass.
- Why not compute gradients numerically? Cost scales with the number of parameters (two forward passes each) and it is approximate; it is used only to check backprop.
- What is a gradient check and how do you do it? Compare backprop gradients with central differences (L(w+h) − L(w−h)) / 2h on a few parameters; relative differences should be tiny (about 1e-7 or less in float64).
- Why initialise weights randomly? To break symmetry so hidden units learn different features.
- What are vanishing and exploding gradients, and fixes? Gradients shrink or grow exponentially with depth; fixes include ReLU-family activations, careful initialisation, normalisation layers, residual connections and gradient clipping (for exploding).
- Write the weight update rule. w ← w − η ∂L/∂w, with η the learning rate.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Backpropagation | Computing every gradient by applying the chain rule from the loss backwards, layer by layer |
| Local gradient | How one step reacts to its own input: sigmoid's a(1 − a) |
| Chain rule | Gradient through a chain = product of local gradients (gears: 3 × 2 = 6) |
| Chain rule through layers | Multiply by each layer's activation slope and pass through Wᵀ to the layer below |
| Gradient | One partial derivative per weight: how the loss changes as that weight moves |
| Weight update rule | w ← w − learning rate × dLoss/dw |
| Learning rate | Size of each update step: 1.0 in our XOR network |
| Numerical gradient | (L(w + h) − L(w − h)) / 2h, measured by nudging |
| Numerical gradient check | Comparing backprop with the numerical gradient to catch bugs |
| Forward pass cache | The z and a values saved during the forward pass for the backward pass |
| Symmetry breaking | Random initial weights so hidden neurons differ |
| Initialisation | Choosing the starting weights: small random numbers |
| Vanishing gradients | Gradients shrinking towards 0 in early layers of deep networks (sigmoid: 2e-01 → 4e-06) |
| Exploding gradients | Gradients growing huge from layer to layer |
| Epoch | One pass over all training examples |
| Local minimum | A point where every small step makes the loss worse, but which isn't the best overall |
| Loss | One number for how wrong the network is: mean squared error |
Summary
- Gradient descent needs every weight's gradient; numerical gradients cost 2 forward passes per weight (about an hour per step for 794,000 weights).
- Local gradients multiplied with the chain rule, from the loss backwards, give all gradients at once: backpropagation (milliseconds).
- Always gradient-check new backprop code against the numerical gradient.
- The weight update rule trained XOR in plain Python, from random weights; all-zero weights stay symmetric and fail.
- Deep sigmoid stacks make gradients vanish; ReLU and good initialisation help.
Next episode: PyTorch 1: tensors and autograd: tensors, shapes, devices, and backpropagation done automatically.