SyntaxPilot
Courses/Machine Learning/Lesson 62
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 62 of 90

Learning rates and initialisation

learning-rate schedules (step decay, exponential, cosine annealing)warmupone-cycle policylearning-rate finderweight initialisationXavier (Glorot) initialisationHe (Kaiming) initialisationexploding and vanishing gradientsgradient clippingleaky ReLUGELU

Machine Learning Full Course · Episode 62. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU). The output shown is the real output.

The big idea: two things decide whether a network trains well before any clever architecture: the learning rate over time and the starting weights. One fixed learning rate is a compromise: early training wants big steps, the end wants small ones, so we use a schedule (step decay, exponential decay, cosine annealing, warmup, one-cycle). The LR finder picks the starting value in one short run. Starting weights with the wrong spread make a deep network's signal explode (to inf/nan) or vanish (to 0) before the first update; Xavier (for tanh/sigmoid) and He (for ReLU) initialisation choose the spread from the layer size. Gradient clipping caps the occasional giant gradient, and leaky ReLU / GELU avoid dead ReLU neurons.

Sources: Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks", AISTATS 2010, proposed a new initialisation scheme and showed the logistic sigmoid is unsuited to deep networks with random initialisation (PMLR 9:249-256). Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, "Delving Deep into Rectifiers", 2015, derived an initialisation that takes rectifiers (ReLU) into account and trains very deep models from scratch (arXiv 1502.01852). Leslie N. Smith, "Cyclical Learning Rates for Training Neural Networks", 2015, describes estimating reasonable learning-rate bounds by increasing the learning rate during a short run (arXiv 1506.01186). Dan Hendrycks and Kevin Gimpel, "Gaussian Error Linear Units (GELUs)", 2016, define GELU as x·Φ(x) (arXiv 1606.08415).

Words you need first

  • Learning rate (lr): the optimiser's step size (Episodes 8 and 61).
  • Epoch, batch, iteration: one pass over the data; a small group of examples; one optimiser step (Episode 59).
  • Hyperparameter: a setting we choose, such as lr (Episode 8).
  • Activation function / activation: the non-linear function after a layer (ReLU, tanh, sigmoid) and its output (Episode 56).
  • Standard deviation (std, Episode 10): how spread out numbers are around their mean.
  • Normal distribution N(0, σ²): the bell curve; torch.randn draws from N(0, 1).
  • Gradient norm: the length of the whole gradient vector, √(Σ g²) (Episode 3's L2 norm).
  • Backpropagation (Episode 57): gradients flow from the last layer back to the first, multiplied layer by layer.

The problem: one fixed learning rate

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)   # the circle
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
for lr in (0.0001, 0.01, 0.1, 3.0):                     # one fixed learning rate for all 30 epochs
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
    opt, loss_fn = torch.optim.Adam(model.parameters(), lr=lr), nn.BCEWithLogitsLoss()
    for epoch in range(30):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    acc = ((model(X) > 0).float() == y).float().mean().item()
    print(f"lr={lr:<7} final loss {loss_fn(model(X), y).item():.3f}   accuracy {acc:.1%}")

Output:

lr=0.0001  final loss 0.411   accuracy 80.1%
lr=0.01    final loss 0.031   accuracy 99.5%
lr=0.1     final loss 0.073   accuracy 97.6%
lr=3.0     final loss 0.471   accuracy 80.1%

80.1% is what "always predict outside the circle" scores. lr = 0.0001 barely moved; lr = 3.0 jumped around and collapsed to that baseline; 0.01 was best (99.5%); 0.1 was good but rougher (97.6%). No single value is right for the whole run: big steps help early, small steps help at the end, like driving fast on the main road and slowly into a parking spot.

Learning-rate schedules

A learning-rate schedule sets lr as a function of the epoch (or step). In PyTorch: torch.optim.lr_scheduler, and call sched.step() after opt.step().

Schedule Rule PyTorch
Step decay multiply lr by γ (gamma) every N epochs: a staircase StepLR(opt, step_size=10, gamma=0.1)
Exponential decay multiply by γ every epoch ExponentialLR(opt, gamma=0.9)
Cosine annealing lr_t = lr_min + (lr_max − lr_min)·(1 + cos(π·t/T))/2 CosineAnnealingLR(opt, T_max=30)
Warmup start small and grow for the first few epochs, then decay LinearLR + SequentialLR
One-cycle policy warm up to max_lr, then anneal to almost 0, in one cycle OneCycleLR(opt, max_lr=0.1, total_steps=...)
  • Annealing is a metalwork word: cooling slowly so the metal settles; here the step size cools down.
  • Warmup: at the start the weights are random and gradients (and Adam's v̂) are unreliable, so we begin gently.
import torch
from torch.optim.lr_scheduler import (StepLR, ExponentialLR, CosineAnnealingLR,
                                      LinearLR, SequentialLR, OneCycleLR)
def show(name, make_sched, epochs=30):
    opt = torch.optim.SGD([torch.zeros(1, requires_grad=True)], lr=0.1)
    sched, lrs = make_sched(opt), []
    for epoch in range(epochs):
        lrs.append(opt.param_groups[0]["lr"])    # the learning rate used in this epoch
        opt.step(); sched.step()                 # train one epoch ... then move the schedule on
    print(f"{name:9}", " ".join(f"{lrs[e]:.3f}" for e in (0, 4, 9, 10, 19, 20, 29)))
print(f"{'epoch:':9}", " ".join(f"{e + 1:>5}" for e in (0, 4, 9, 10, 19, 20, 29)))
show("step",     lambda o: StepLR(o, step_size=10, gamma=0.1))     # x0.1 every 10 epochs
show("exp",      lambda o: ExponentialLR(o, gamma=0.9))            # x0.9 every epoch
show("cosine",   lambda o: CosineAnnealingLR(o, T_max=30))         # half a cosine wave down to 0
show("warmup",   lambda o: SequentialLR(o, [LinearLR(o, start_factor=0.1, total_iters=5),
                                             CosineAnnealingLR(o, T_max=25)], milestones=[5]))
show("onecycle", lambda o: OneCycleLR(o, max_lr=0.1, total_steps=30))

Output (the learning rate used in epochs 1, 5, 10, 11, 20, 21 and 30):

epoch:        1     5    10    11    20    21    30
step      0.100 0.100 0.100 0.010 0.010 0.001 0.001
exp       0.100 0.066 0.039 0.035 0.014 0.012 0.005
cosine    0.100 0.096 0.079 0.075 0.030 0.025 0.000
warmup    0.010 0.082 0.094 0.090 0.041 0.035 0.000
onecycle  0.004 0.052 0.099 0.098 0.046 0.039 0.000

Step decay drops ×0.1 at epochs 11 and 21; exponential shrinks smoothly to 0.005; cosine stays high, then falls faster, ending at ~0 (0.00027); warmup rises 0.010 → 0.082 over 5 epochs, then follows a cosine; one-cycle starts at 0.004, peaks at ~0.1 around epoch 10 (30% of the run by default) and ends at ~0.

Does a schedule help?

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)      # 20 batches per epoch
def train(schedule):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
    opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.1), nn.BCEWithLogitsLoss()
    sched = schedule(opt) if schedule else None
    for epoch in range(30):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
            if sched: sched.step()                 # OneCycleLR moves every BATCH
    acc = ((model(X) > 0).float() == y).float().mean().item()
    return f"final loss {loss_fn(model(X), y).item():.3f}   accuracy {acc:.1%}"
print("fixed lr=0.1:      ", train(None))
from torch.optim.lr_scheduler import OneCycleLR
print("one-cycle, max 0.1:", train(lambda o: OneCycleLR(o, max_lr=0.1, total_steps=30 * 20)))

Output:

fixed lr=0.1:       final loss 0.073   accuracy 97.6%
one-cycle, max 0.1: final loss 0.014   accuracy 99.9%

Same network, same peak learning rate: one-cycle gives 99.9% and a loss five times lower than fixed lr = 0.1 (97.6%). OneCycleLR is stepped every batch, so total_steps = epochs × batches per epoch = 30 × 20.

The learning-rate finder (LR range test)

Trying ten learning rates means ten full trainings. The LR finder does one short run, raising lr a little on every batch, and watches the loss: flat while lr is too small, falling in the useful range, exploding when too big. Pick a value inside the falling region, well below the explosion.

import torch, torch.nn as nn
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.Adam(model.parameters())
smooth, best, start_lr = None, float("inf"), None          # LR finder: a bigger lr on every batch
for i in range(56):
    lr = 10 ** (-5 + i / 8)                                # 0.00001 ... 10, x10 every 8 batches
    for g in opt.param_groups: g["lr"] = lr
    idx = torch.randint(0, 1000, (100,))
    loss = loss_fn(model(X[idx]), y[idx]); opt.zero_grad(); loss.backward(); opt.step()
    smooth = loss.item() if smooth is None else 0.7 * smooth + 0.3 * loss.item()   # less jumpy
    if i == 0: first = smooth
    if start_lr is None and smooth < 0.9 * first: start_lr = lr   # loss really starts to fall
    best = min(best, smooth)
    if i % 8 == 0: print(f"lr {lr:<8.5g} smoothed loss {smooth:.3f}")
    if smooth > 4 * best: print(f"lr {lr:<8.5g} smoothed loss {smooth:.3f}  <- exploded, stop"); break
print(f"learning starts near lr {start_lr:.2g}, blows up near lr {lr:.2g}")

Output:

lr 1e-05    smoothed loss 0.624
lr 0.0001   smoothed loss 0.621
lr 0.001    smoothed loss 0.611
lr 0.01     smoothed loss 0.493
lr 0.1      smoothed loss 0.465
lr 1        smoothed loss 0.400
lr 5.6234   smoothed loss 5.354  <- exploded, stop
learning starts near lr 0.0042, blows up near lr 5.6

Too small below ~0.001 (the loss stays at 0.61–0.62), learning starts near 0.004, the loss explodes near 5.6. A value around 0.01 is inside the useful range, and our first experiment agrees (0.01 gave 99.5%). The smoothed loss is a moving average (0.7·old + 0.3·new) so one lucky or unlucky batch doesn't fool us.

Initialisation: exploding and vanishing signals

Weight initialisation: choosing the weights before training. A message passed along 50 people: if everyone repeats it louder, the end is a roar (exploding); softer, and the end hears nothing (vanishing). Every layer should keep the spread about the same.

  • Exploding gradients: gradients grow layer after layer until they are huge, inf or nan.
  • Vanishing gradients: gradients shrink layer after layer until the first layers get (almost) zero and stop learning.
  • nan ("not a number"): the result of inf − inf, 0·inf, etc. Once a nan appears it spreads everywhere.
  • Saturation: tanh/sigmoid outputs stuck near ±1 (or 0/1), where their slope is almost 0.
import torch, torch.nn as nn
def check(name, init, act):
    torch.manual_seed(62)
    layers = [nn.Linear(256, 256, bias=False) for _ in range(50)]        # 50 layers deep
    for L in layers: init(L.weight)
    h = x = torch.randn(512, 256)
    for L in layers: h = act(L(h))
    h.pow(2).mean().backward()                                           # any loss will do
    grad = layers[0].weight.grad.norm().item()
    print(f"{name:22} last std {h.std().item():<9.3g} first grad {grad:.3g}")
check("normal std=1, none",   lambda w: nn.init.normal_(w, std=1.0), lambda h: h)
check("normal std=0.01, tanh", lambda w: nn.init.normal_(w, std=0.01), torch.tanh)
check("normal std=1, tanh",   lambda w: nn.init.normal_(w, std=1.0), torch.tanh)
check("Xavier, tanh",         nn.init.xavier_normal_, torch.tanh)
check("Xavier, ReLU",         nn.init.xavier_normal_, torch.relu)
check("He (Kaiming), ReLU",   lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu"), torch.relu)

Output (50 layers of 256 → 256, 512 random inputs):

normal std=1, none     last std nan       first grad nan
normal std=0.01, tanh  last std 1.34e-40  first grad 0
normal std=1, tanh     last std 0.974     first grad inf
Xavier, tanh           last std 0.0927    first grad 0.00449
Xavier, ReLU           last std 1.64e-08  first grad 4e-16
He (Kaiming), ReLU     last std 0.551     first grad 0.318
  • std = 1, no activation: each layer multiplies the spread by about √256 = 16, so values overflow to inf and then nan.
  • std = 0.01, tanh: each layer shrinks the spread to about 0.16×; after 50 layers the activations are 1e-40 and the first layer's gradient is exactly 0 (vanished).
  • std = 1, tanh: the forward pass looks fine (0.974, but saturated at ±1), yet the gradient explodes to inf.

Xavier (Glorot) and He (Kaiming) initialisation

Fan-in: the number of inputs to a layer; fan-out: its number of outputs. For nn.Linear(256, 256) both are 256.

Name std of the normal distribution Use with PyTorch
Xavier / Glorot √(2 / (fan_in + fan_out)) = 0.0625 here tanh, sigmoid nn.init.xavier_normal_(w)
He / Kaiming √(2 / fan_in) = 0.088 here ReLU (and leaky ReLU) nn.init.kaiming_normal_(w, nonlinearity="relu")
  • Why the 2 in He? ReLU sets about half of its inputs to 0, which halves the signal's energy; doubling the variance makes up for it.
  • There are also _uniform_ versions with the same spread. nn.Linear already uses a Kaiming-uniform default; deep custom networks often set the initialisation on purpose.

Results from the same program: Xavier + tanh keeps a live signal (0.0927, gradient 0.00449); Xavier + ReLU fades (1.64e-08) because ReLU halves the signal each layer; He + ReLU stays healthy (0.551 at the last layer, gradient 0.318 at the first).

Gradient clipping

Even with good initialisation, one unusual batch can produce a giant gradient. Gradient clipping (by norm): if the gradient norm is above max_norm, scale the whole gradient down to length max_norm. The direction stays the same; only the length is capped, like a speed governor on a bus.

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # between backward and step
opt.step()
import torch
x = torch.tensor([1.0, 2.0, 3.0, 100.0]); y = 2 * x          # true rule y = 2x; one huge input (100)
for clip in (None, 1.0):
    w = torch.tensor([0.0], requires_grad=True); opt = torch.optim.SGD([w], lr=0.01)
    for step in range(1, 301):
        loss = ((w * x - y) ** 2).mean(); opt.zero_grad(); loss.backward()
        before = w.grad.norm().item()
        if clip: torch.nn.utils.clip_grad_norm_([w], max_norm=clip)   # shrink the gradient if too long
        opt.step()
        if step in (1, 2, 3, 300):
            after = w.grad.norm().item()
            label = "clipped" if clip else "plain"
            print(f"{label:7} step {step:3}  grad {before:9.4g} -> {after:9.4g}  w {w.item():.4g}")

Output:

plain   step   1  grad 1.001e+04 -> 1.001e+04  w 100.1
plain   step   2  grad 4.914e+05 -> 4.914e+05  w -4814
plain   step   3  grad 2.411e+07 -> 2.411e+07  w 2.363e+05
plain   step 300  grad       nan ->       nan  w nan
clipped step   1  grad 1.001e+04 ->         1  w 0.01
clipped step   2  grad      9964 ->         1  w 0.02
clipped step   3  grad      9914 ->         1  w 0.03
clipped step 300  grad     31.66 ->         1  w 1.996

Without clipping, the input 100 makes the first gradient 10,014; w jumps to 100.1, then −4,814, then 236,300, and is nan by step 300. With clipping every step has length 1 (here lr·1 = 0.01), so w walks steadily to 1.996 ≈ 2. clip_grad_norm_ returns the norm before clipping, handy for logging. There is also clip_grad_value_ (clip each element), less common.

Better activations: leaky ReLU and GELU

  • ReLU max(0, z) has gradient 0 for every z < 0 (PyTorch also uses 0 at z = 0).
  • Dying ReLU / dead neuron: if a neuron's input is negative for every example (say after a big update pushed its bias far down), it outputs 0, gets gradient 0 and never recovers.
  • Leaky ReLU: z if z > 0, else 0.01·z (the slope 0.01 is negative_slope): a little gradient always flows.
  • GELU (Gaussian error linear unit): z · Φ(z), where Φ is the standard normal CDF (Episode 10). Smooth, slightly negative for small negative z, widely used in transformer models.
import torch, torch.nn as nn
z = torch.tensor([-3.0, -1.0, -0.5, 0.0, 0.5, 1.0, 3.0], requires_grad=True)
for name, f in [("ReLU", nn.ReLU()), ("Leaky", nn.LeakyReLU(0.01)), ("GELU", nn.GELU())]:
    out = f(z); z.grad = None; out.sum().backward()
    print(f"{name:5}", [round(v, 3) for v in out.tolist()])         # f(z)
    print(" grad", [round(v, 3) for v in z.grad.tolist()])          # slope
torch.manual_seed(62)                             # a "dead" neuron: its bias pushed far negative
neuron = nn.Linear(2, 1); nn.init.constant_(neuron.bias, -10.0)
x = torch.randn(1000, 2)
for name, f in [("ReLU", nn.ReLU()), ("Leaky", nn.LeakyReLU(0.01))]:
    neuron.zero_grad(); f(neuron(x)).sum().backward()
    active, g = (neuron(x) > 0).sum().item(), neuron.weight.grad.abs().sum().item()
    print(f"{name:5} on 1000 inputs: active {active}, weight grad {g:.3g}")

Output:

ReLU  [0.0, 0.0, 0.0, 0.0, 0.5, 1.0, 3.0]
 grad [0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
Leaky [-0.03, -0.01, -0.005, 0.0, 0.5, 1.0, 3.0]
 grad [0.01, 0.01, 0.01, 0.01, 1.0, 1.0, 1.0]
GELU  [-0.004, -0.159, -0.154, 0.0, 0.346, 0.841, 2.996]
 grad [-0.012, -0.083, 0.133, 0.5, 0.867, 1.083, 1.012]
ReLU  on 1000 inputs: active 0, weight grad 0
Leaky on 1000 inputs: active 0, weight grad 1.22

ReLU's gradient is 0 for the four inputs ≤ 0; leaky ReLU keeps 0.01; GELU dips to −0.159 at −1 and curves smoothly. The neuron with bias −10 is never active; with ReLU its weight gradient is exactly 0 (dead), with leaky ReLU it is 1.22, so training can still move it.

Common mistakes

Mistake Exact message / symptom Why Fix
One fixed lr for the whole run 97.6% vs 99.9% with one-cycle can't take small steps at the end a schedule
sched.step() before opt.step() UserWarning: Detected call of lr_scheduler.step() before optimizer.step()... Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. wrong order opt.step() then sched.step()
Stepping OneCycleLR once per epoch with total_steps = batches the schedule ends after a few epochs it counts calls match total_steps to how often you call it
std = 1 weights in a deep network nan, first-layer grad inf exploding Xavier / He
Tiny weights (std = 0.01) activations 1.34e-40, grad 0 vanishing Xavier / He
Xavier with ReLU in a deep net last-layer std 1.64e-08 ReLU halves the signal He (Kaiming)
No clipping with rare huge inputs w = nan one giant step clip_grad_norm_
Dead ReLU neurons weight grad 0 always-negative input leaky ReLU / GELU, lower lr, good init

Mini project: put it all together

A deeper network (6 hidden layers), He initialisation, AdamW, a warmup + cosine schedule stepped per batch (LambdaLR multiplies the base lr by the function's value), and gradient clipping:

import torch, torch.nn as nn, math
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
torch.manual_seed(0)
layers = [nn.Linear(2, 64), nn.ReLU()]
for _ in range(5): layers += [nn.Linear(64, 64), nn.ReLU()]          # 6 hidden layers
model = nn.Sequential(*layers, nn.Linear(64, 1))
for m in model:
    if isinstance(m, nn.Linear): nn.init.kaiming_normal_(m.weight, nonlinearity="relu"); nn.init.zeros_(m.bias)
opt = torch.optim.AdamW(model.parameters(), lr=0.01, weight_decay=0.01)
total, warm = 30 * len(loader), 2 * len(loader)                       # 600 steps, 40 of warmup
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: (s + 1) / warm if s < warm
                                          else 0.5 * (1 + math.cos(math.pi * (s - warm) / (total - warm))))
loss_fn, biggest = nn.BCEWithLogitsLoss(), 0.0
for epoch in range(1, 31):
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward()
        biggest = max(biggest, nn.utils.clip_grad_norm_(model.parameters(), 1.0).item())
        opt.step(); sched.step()
    if epoch in (1, 2, 10, 30):
        acc = ((model(X) > 0).float() == y).float().mean().item()
        print(f"epoch {epoch:2}: lr {opt.param_groups[0]['lr']:.5f}  loss {loss_fn(model(X), y).item():.3f}  acc {acc:.1%}")
print(f"largest gradient norm before clipping: {biggest:.2f}")

Output:

epoch  1: lr 0.00525  loss 0.319  acc 80.5%
epoch  2: lr 0.01000  loss 0.144  acc 94.9%
epoch 10: lr 0.00812  loss 0.018  acc 99.4%
epoch 30: lr 0.00000  loss 0.003  acc 100.0%
largest gradient norm before clipping: 4.18

The learning rate warms up to 0.01 by the end of epoch 2, then follows a cosine down to 0; the model reaches 100% on the circle, and the largest gradient norm seen (4.18) was clipped to 1.

Practice

1. Plot the cosine and one-cycle learning-rate curves with matplotlib.

Solution
import torch, matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from torch.optim.lr_scheduler import CosineAnnealingLR, OneCycleLR
def curve(make, steps=100):
    opt = torch.optim.SGD([torch.zeros(1, requires_grad=True)], lr=0.1); s = make(opt); out = []
    for _ in range(steps): out.append(opt.param_groups[0]["lr"]); opt.step(); s.step()
    return out
cos = curve(lambda o: CosineAnnealingLR(o, T_max=100))
one = curve(lambda o: OneCycleLR(o, max_lr=0.1, total_steps=100))
plt.plot(cos, label="cosine annealing"); plt.plot(one, label="one-cycle")
plt.xlabel("step"); plt.ylabel("learning rate"); plt.legend(); plt.savefig("lr_curves.png")
print("peak of one-cycle at step", one.index(max(one)), "| saved lr_curves.png")

Output:

peak of one-cycle at step 29 | saved lr_curves.png

The PNG shows cosine falling from 0.1 to 0 like half a wave, and one-cycle rising for the first 30% and then falling.

2. Run the LR finder with SGD instead of Adam.

Solution

Change the optimiser line to opt = torch.optim.SGD(model.parameters(), lr=1e-5). Output:

lr 1e-05    smoothed loss 0.624
lr 0.0001   smoothed loss 0.622
lr 0.001    smoothed loss 0.623
lr 0.01     smoothed loss 0.610
lr 0.1      smoothed loss 0.585
lr 1        smoothed loss 0.493
lr 10       smoothed loss 0.538
lr 23.714   smoothed loss 2.340  <- exploded, stop
learning starts near lr 0.32, blows up near lr 24

SGD's useful range sits at larger learning rates than Adam's: its step is lr × gradient, while Adam's is about lr.

3. Use GELU in the 50-layer test. Which initialisation works?

Solution

Add check("Xavier, GELU", nn.init.xavier_normal_, nn.functional.gelu) and check("He, GELU", lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu"), nn.functional.gelu). Output:

normal std=1, none     last std nan       first grad nan
normal std=0.01, tanh  last std 1.34e-40  first grad 0
normal std=1, tanh     last std 0.974     first grad inf
Xavier, tanh           last std 0.0927    first grad 0.00449
Xavier, ReLU           last std 1.64e-08  first grad 4e-16
He (Kaiming), ReLU     last std 0.551     first grad 0.318
Xavier, GELU           last std 1.63e-15  first grad 0
He, GELU               last std 3.36e-06  first grad 8.61e-10

Neither keeps a 50-layer GELU network fully alive: GELU passes less of the signal than ReLU for inputs near zero, so even He fades (3.36e-06), though far less than Xavier (1.63e-15). Very deep GELU networks also rely on normalisation layers (Episode 63) and residual connections (Episode 66).

4. Add clip_grad_norm_ to Episode 61's race and print the largest gradient norm seen.

Solution

Inside the batch loop: n = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); biggest = max(biggest, n.item()) between backward() and step(). Output:

epoch:            1      5     10     20
SGD           0.647  0.537  0.493  0.475  max grad norm 0.68
SGD+momentum  0.508  0.467  0.457  0.399  max grad norm 0.66
AdaGrad       0.502  0.470  0.462  0.443  max grad norm 0.64
RMSprop       0.436  0.114  0.065  0.109  max grad norm 1.24
Adam          0.476  0.418  0.055  0.029  max grad norm 1.07

On this small, well-scaled problem the gradient norm stays below 1 for SGD, momentum and AdaGrad, so clipping never acts; it only trims RMSprop (max 1.24) and Adam (max 1.07), and RMSprop's last loss moves from 0.138 to 0.109. Clipping is insurance, not a speed-up.

Quick quiz

nn.Linear(400, 100) followed by ReLU. What std does He initialisation use?

Answer

√(2/400) ≈ 0.071. He uses fan-in only (400). √(2/500) ≈ 0.063 is Xavier's formula.

More questions:

  1. Which schedule multiplies lr by 0.1 every 10 epochs?
    AnswerStep decay, StepLR(opt, step_size=10, gamma=0.1).
  2. Why warm up?
    AnswerEarly gradients from random weights are large and unreliable; a small lr at first avoids wrecking the model, then it rises.
  3. In the LR finder, where do you pick lr?
    AnswerInside the region where the loss falls fastest, well below where it explodes.
  4. What does clip_grad_norm_ change: the direction or the length?
    AnswerOnly the length (when it's above max_norm).
  5. What is a dead ReLU neuron?
    AnswerOne whose input is negative for all examples, so its output and gradient are 0 and it never learns again.

Interview questions

  1. Why decay the learning rate? Large steps make fast progress but bounce around the minimum; smaller steps later let the parameters settle into it.
  2. Cosine annealing vs step decay? Both lower lr over time; cosine is smooth with no hand-picked milestones, step decay drops suddenly at chosen epochs.
  3. What is the one-cycle policy? Increase lr to a maximum, then decrease it to nearly zero over a single cycle (often with momentum moving the opposite way); it frequently trains faster.
  4. Explain vanishing and exploding gradients. Backpropagation multiplies by each layer's weights and activation slopes; if those factors are consistently below 1 the gradient shrinks exponentially with depth, above 1 it grows exponentially.
  5. Xavier vs He? Xavier keeps the variance with 2/(fan_in+fan_out), suited to tanh/sigmoid; He uses 2/fan_in because ReLU zeroes half its inputs.
  6. Besides initialisation, what fights vanishing/exploding gradients? ReLU-family activations, normalisation layers (Episode 63), residual connections (Episode 66), gradient clipping, and LSTM gates for sequences (Episode 70).

Glossary

Term Meaning (plain words + example)
Learning-rate schedule A plan that changes lr during training
Scheduler The PyTorch object that applies a schedule: StepLR(opt, ...)
Step decay Multiply lr by γ every N epochs: 0.1 → 0.01 → 0.001
γ (gamma) The decay factor: 0.1 or 0.9
Exponential decay Multiply lr by γ every epoch: 0.1, 0.09, 0.081 …
Cosine annealing lr follows half a cosine wave down to ~0
Annealing Cooling slowly; here, lowering the step size gradually
Warmup Starting with a small lr and raising it for the first epochs
One-cycle policy Warm up to max_lr, then anneal to ~0, once: OneCycleLR
LR finder (LR range test) One short run with lr growing each batch, to find the useful range
Smoothed loss A moving average of the loss: 0.7·old + 0.3·new
Weight initialisation Choosing the starting weights: nn.init.kaiming_normal_(w)
Fan-in / fan-out A layer's number of inputs / outputs: 256 / 256
Xavier (Glorot) initialisation std √(2/(fan_in+fan_out)), for tanh and sigmoid
He (Kaiming) initialisation std √(2/fan_in), for ReLU
Exploding gradients Gradients growing layer by layer to huge values / inf
Vanishing gradients Gradients shrinking layer by layer to ~0
nan "Not a number": the result of inf − inf; it spreads through every later calculation
Saturation tanh/sigmoid stuck at their flat ends, slope ≈ 0
Gradient norm The length of the gradient vector: 10,014
Gradient clipping Scaling the gradient down when its norm exceeds max_norm
max_norm The clipping limit: 1.0
Dying ReLU (dead neuron) A ReLU unit that always outputs 0 and gets 0 gradient
Leaky ReLU z for z > 0, else 0.01·z
negative_slope Leaky ReLU's slope below 0: 0.01
GELU Gaussian error linear unit, z·Φ(z): smooth, used in transformers
Φ (Phi) The standard normal cumulative distribution function (CDF)

Summary

  • One fixed learning rate is a compromise; schedules start big and finish small: step decay, exponential, cosine annealing, warmup, one-cycle.
  • The LR finder finds the useful range in one short run.
  • Bad starting weights make a deep network explode or vanish before training starts.
  • Xavier for tanh/sigmoid, He (Kaiming) for ReLU.
  • Gradient clipping caps rare giant gradients; leaky ReLU and GELU avoid dead neurons.

Next episode: Regularisation for deep networks: dropout, weight decay, early stopping, data augmentation, batch norm, layer norm and label smoothing.