SyntaxPilot
Courses/Machine Learning/Lesson 61
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 61 of 90

Optimisers

SGD in PyTorchmomentumNesterov momentumAdaGradRMSPropAdamfirst and second moment estimatesAdamWweight decay vs L2comparing optimisers on loss curves

Machine Learning Full Course · Episode 61. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU). The output shown is the real output.

The big idea: plain SGD uses one learning rate for every parameter and forgets every previous step. In a long, narrow valley that means it zig-zags across the steep direction and crawls along the flat one; a slightly bigger learning rate makes it explode. Two fixes, invented one after the other: momentum (a velocity that remembers past gradients, so the zig-zags cancel and the steady direction speeds up) and per-parameter step sizes (AdaGrad, then RMSProp, divide each gradient by its typical size). Adam combines both, plus bias correction, and became the default optimiser for deep learning. AdamW fixes how Adam applies weight decay.

Sources: Adam was introduced by Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization", published at ICLR 2015 (arXiv 1412.6980). AdamW comes from Ilya Loshchilov and Frank Hutter, "Decoupled Weight Decay Regularization", ICLR 2019, which shows that L2 regularisation and weight decay are equivalent for SGD but not for adaptive methods like Adam, and that decoupling the decay improves Adam's generalisation (arXiv 1711.05101).

Words you need first

  • Parameter: a number the model learns, such as a weight or a bias.
  • Gradient (Episode 7): the vector of partial derivatives; it points where the loss rises fastest. We step the other way.
  • Learning rate, lr (Episode 8): how far each step goes.
  • Loss surface / valley: picture the loss as height over the parameters; training walks downhill.
  • Optimiser: the rule that turns gradients into parameter updates (torch.optim.*). Every optimiser has zero_grad() and step() (Episode 59).
  • SGD, stochastic gradient descent: p = p − lr × gradient, computed on small random batches. Stochastic means random.
  • Mini-batch: a small group of examples used for one step (Episode 60).

The problem: plain SGD in a narrow valley

The loss 0.5 · (x² + 100·y²) is 100 times steeper in y than in x; the best point is (0, 0). We start at (−10, 1).

import torch
def loss_fn(p):                                   # a long, narrow valley: steep in y, flat in x
    x, y = p
    return 0.5 * (x**2 + 100 * y**2)
for lr in (0.019, 0.021):
    p = torch.tensor([-10.0, 1.0], requires_grad=True)
    opt = torch.optim.SGD([p], lr=lr)             # plain SGD: p = p - lr * gradient
    for step in range(1, 101):
        loss = loss_fn(p); opt.zero_grad(); loss.backward(); opt.step()
        if step in (1, 2, 3, 100):
            x, y = p.tolist()
            print(f"lr={lr} step {step:3}: x={x:8.3f}  y={y:10.3f}  loss={loss.item():.4g}")

Output:

lr=0.019 step   1: x=  -9.810  y=    -0.900  loss=100
lr=0.019 step   2: x=  -9.624  y=     0.810  loss=88.62
lr=0.019 step   3: x=  -9.441  y=    -0.729  loss=79.11
lr=0.019 step 100: x=  -1.469  y=     0.000  loss=1.121
lr=0.021 step   1: x=  -9.790  y=    -1.100  loss=100
lr=0.021 step   2: x=  -9.584  y=     1.210  loss=108.4
lr=0.021 step   3: x=  -9.383  y=    -1.331  loss=119.1
lr=0.021 step 100: x=  -1.197  y= 13780.572  loss=7.847e+09
  • lr = 0.019: y flips sign every step (−0.9, 0.81, −0.729): a zig-zag across the valley. x only crawls from −10 to −1.469 in 100 steps.
  • lr = 0.021: the y swing grows (×−1.1 per step) and the loss explodes to 7.8 billion.
  • Why the limit is 0.02: in y one SGD step multiplies y by (1 − lr·100). That factor must stay between −1 and 1, so lr < 2/100 = 0.02. In x the factor is (1 − lr), which is close to 1: slow.

Why it failed: one learning rate must serve both a steep and a flat direction, and each step forgets the previous one.

Momentum and Nesterov momentum

A cyclist coasting down a ghat road keeps their speed (memory); side-to-side wobbles cancel; the steady downhill pull adds up.

v = β · v + g        # velocity: a running memory of gradients, β (beta) ≈ 0.9
p = p − lr · v
  • Momentum: gradients that keep the same sign add up (speed along the valley); gradients that flip sign cancel (less zig-zag).
  • β (beta): the fraction of the old velocity kept each step. 0.9 means "remember about the last 10 steps".
  • Nesterov momentum: first move along the velocity (look ahead), measure the gradient there, then correct. It reacts earlier, like braking before a bend.
  • In PyTorch: torch.optim.SGD(params, lr=0.01, momentum=0.9, nesterov=True).
import torch
def loss_fn(p): return 0.5 * (p[0]**2 + 100 * p[1]**2)
def run(make_opt, steps=100):
    p = torch.tensor([-10.0, 1.0], requires_grad=True); opt = make_opt([p])
    for _ in range(steps):
        loss = loss_fn(p); opt.zero_grad(); loss.backward(); opt.step()
    return loss_fn(p).item()
# momentum from scratch: v = beta * v + g ;  p = p - lr * v
p = torch.tensor([-10.0, 1.0]); v = torch.zeros(2); lr, beta = 0.01, 0.9
for step in range(100):
    g = torch.tensor([p[0], 100 * p[1]])          # the gradient of our valley
    v = beta * v + g                              # a running "velocity"
    p = p - lr * v
print("momentum (ours) loss@100:", f"{loss_fn(p).item():.3g}")
SGD = torch.optim.SGD
print("SGD             loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01)):.3g}")
print("SGD momentum    loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01, momentum=0.9)):.3g}")
print("SGD Nesterov    loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01, momentum=0.9, nesterov=True)):.3g}")

Output:

momentum (ours) loss@100: 0.000438
SGD             loss@100: 6.7
SGD momentum    loss@100: 0.000438
SGD Nesterov    loss@100: 2.51e-05

Same learning rate (0.01), 100 steps: plain SGD 6.7, momentum 0.000438 (our two-line version gives exactly PyTorch's number), Nesterov 2.51e-05. Momentum also changes the safe learning rate: with lr = 0.019 Nesterov diverges in this valley (we tested it), so momentum and lr must be tuned together.

AdaGrad and RMSProp: a step size for every parameter

A tailor measures the sleeve, the collar and the waist separately instead of one tape length for the whole body. Here: divide each parameter's gradient by its own typical size.

AdaGrad:  s = s + g²                     (add up the squared gradients forever)
RMSProp:  s = ρ · s + (1 − ρ) · g²       (a moving average, ρ (rho) ≈ 0.9)
step = lr · g / (√s + ε)                 (ε (epsilon) ≈ 1e-8 avoids division by zero)
  • AdaGrad (adaptive gradient): parameters with big past gradients get small steps; ones with tiny gradients get big steps. Its weakness: s only grows, so steps shrink forever.
  • RMSProp (root mean square propagation): the same idea with a moving average, so old gradients fade and the step size settles.
import torch
g = torch.tensor([100.0, 0.01])                  # one steep direction, one almost flat
lr = 0.1
print("SGD step:", [round(v, 4) for v in (lr * g).tolist()], "<- 10 vs 0.001")
s_ada, s_rms, rho = torch.zeros(2), torch.zeros(2), 0.9
for t in range(1, 1001):
    s_ada += g**2                                # AdaGrad: ADD UP squared gradients forever
    s_rms = rho * s_rms + (1 - rho) * g**2       # RMSProp: a moving AVERAGE of them
    if t in (1, 10, 100, 1000):
        ada = lr * g / (s_ada.sqrt() + 1e-8); rms = lr * g / (s_rms.sqrt() + 1e-8)
        r = lambda s: [round(v, 4) for v in s.tolist()]
        print(f"step {t:4}: AdaGrad {r(ada)}   RMSProp {r(rms)}")

Output:

SGD step: [10.0, 0.001] <- 10 vs 0.001
step    1: AdaGrad [0.1, 0.1]   RMSProp [0.3162, 0.3162]
step   10: AdaGrad [0.0316, 0.0316]   RMSProp [0.1239, 0.1239]
step  100: AdaGrad [0.01, 0.01]   RMSProp [0.1, 0.1]
step 1000: AdaGrad [0.0032, 0.0032]   RMSProp [0.1, 0.1]

Plain SGD steps differ by 10,000× (10 vs 0.001). AdaGrad and RMSProp give both parameters the same step. AdaGrad's step falls 0.1 → 0.0316 → 0.01 → 0.0032 (it is lr/√t for a constant gradient); RMSProp's settles at 0.1 = lr.

Adam: first and second moment estimates, bias correction

A delivery rider remembers which way the last turns went (direction = momentum) and how bumpy each road usually is (scale = RMSProp). On the first ride of the day both memories are empty, so they shouldn't be trusted as they are: bias correction.

m = β₁ · m + (1 − β₁) · g          # first moment estimate: moving average of g      (β₁ = 0.9)
v = β₂ · v + (1 − β₂) · g²         # second moment estimate: moving average of g²    (β₂ = 0.999)
m̂ = m / (1 − β₁ᵗ),  v̂ = v / (1 − β₂ᵗ)    # bias correction; t = step number 1, 2, 3, ...
p = p − lr · m̂ / (√v̂ + ε)
  • Moment (statistics): the first moment of a quantity is its mean; the second (raw) moment is the mean of its square. Adam estimates both for every parameter.
  • Bias correction: m and v start at 0, so early on they are too small (biased towards 0). Dividing by 1 − β^t exactly removes that bias for a constant gradient.
  • Adam = adaptive moment estimation. Defaults in PyTorch: lr=0.001, betas=(0.9, 0.999), eps=1e-8.
import torch
g = torch.tensor([100.0, 0.01]); lr, b1, b2, eps = 0.1, 0.9, 0.999, 1e-8
m = torch.zeros(2); v = torch.zeros(2)           # 1st moment (mean) and 2nd moment (mean of g^2)
for t in range(1, 4):
    m = b1 * m + (1 - b1) * g                    # like momentum
    v = b2 * v + (1 - b2) * g**2                 # like RMSProp
    m_hat, v_hat = m / (1 - b1**t), v / (1 - b2**t)   # bias correction: they start at 0
    step = lr * m_hat / (v_hat.sqrt() + eps)
    st = [round(s, 4) for s in step.tolist()]
    print(f"t={t}: m={m[0].item():6.2f}  m_hat={m_hat[0].item():6.1f}  step={st}")
p = torch.tensor([1.0, 1.0], requires_grad=True)  # check against PyTorch's Adam
opt = torch.optim.Adam([p], lr=0.1)
for t in range(3):
    opt.zero_grad(); (p * g).sum().backward(); opt.step()
print("torch Adam after 3 steps, p =", [round(x, 4) for x in p.tolist()])

Output:

t=1: m= 10.00  m_hat= 100.0  step=[0.1, 0.1]
t=2: m= 19.00  m_hat= 100.0  step=[0.1, 0.1]
t=3: m= 27.10  m_hat= 100.0  step=[0.1, 0.1]
torch Adam after 3 steps, p = [0.7, 0.7]

At step 1, m = 0.1 × 100 = 10, and m̂ = 10 / (1 − 0.9) = 100, the true gradient. The step is 0.1 = lr for both parameters, whatever the gradient size: in Adam, lr is roughly the size of each step. PyTorch's Adam moves both from 1.0 to 0.7 in 3 steps of 0.1, matching ours.

The race: comparing optimisers on loss curves

A loss curve is the loss after each epoch; lower and earlier is better. A fair race uses the same data, the same starting weights (torch.manual_seed(0) before building each model) and the same batches.

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader; torch.manual_seed(61)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)   # the circle again
print(f"{'epoch:':13}", "  ".join(f"{e:>5}" for e in (1, 5, 10, 20)))
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
def race(name, make_opt):
    torch.manual_seed(0)                         # the SAME starting weights for everyone
    model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 16), nn.Tanh(), nn.Linear(16, 1))
    opt, loss_fn, curve = make_opt(model.parameters()), nn.BCEWithLogitsLoss(), []
    for epoch in range(20):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
        curve.append(loss_fn(model(X), y).item())
    print(f"{name:13}", "  ".join(f"{curve[e - 1]:.3f}" for e in (1, 5, 10, 20)))
race("SGD",          lambda p: torch.optim.SGD(p, lr=0.01))
race("SGD+momentum", lambda p: torch.optim.SGD(p, lr=0.01, momentum=0.9))
race("AdaGrad",      lambda p: torch.optim.Adagrad(p, lr=0.01))
race("RMSprop",      lambda p: torch.optim.RMSprop(p, lr=0.01))
race("Adam",         lambda p: torch.optim.Adam(p, lr=0.01))

Output (loss on all 1,000 points after epochs 1, 5, 10 and 20):

epoch:            1      5     10     20
SGD           0.647  0.537  0.493  0.475
SGD+momentum  0.508  0.467  0.457  0.399
AdaGrad       0.502  0.470  0.462  0.443
RMSprop       0.436  0.114  0.065  0.138
Adam          0.476  0.418  0.055  0.029

With lr = 0.01 for everyone: Adam wins (0.029), RMSprop is fast but bounces back up (0.065 → 0.138), SGD, momentum and AdaGrad are still near 0.4–0.48. One race, one problem, one learning rate: each optimiser has its own best learning rate (see the mini project). SGD with momentum and a tuned learning rate is still common for image models.

AdamW: weight decay vs L2

Weight decay: shrink every weight a little each step, so the model keeps small weights and a simpler function (the idea of Episode 21's ridge regression).

L2 regularisation:      g = g + λ·w            then the usual optimiser step
Decoupled weight decay: w = w − lr·λ·w         separately, after the step     (λ = lambda, the decay strength)
  • For plain SGD these are the same update.
  • For Adam they are not: with L2, the λ·w term is inside g, so Adam divides it by √v̂. Weights with big gradients (big v̂) barely decay; weights with tiny gradients decay a lot.
  • In PyTorch, torch.optim.Adam(..., weight_decay=λ) adds L2 to the gradient; torch.optim.AdamW(..., weight_decay=λ) decays the weights directly (its default weight_decay is 0.01).
import torch
torch.manual_seed(61)
def run(opt_class, **kw):
    w = torch.tensor([1.0, 1.0], requires_grad=True)   # w[0]: big noisy gradients, w[1]: tiny
    opt = opt_class([w], lr=0.001, **kw)
    for _ in range(1000):
        noise = torch.randn(2) * torch.tensor([10.0, 0.01])
        loss = (w * noise).sum()                        # gradients that average out to zero
        opt.zero_grad(); loss.backward(); opt.step()
    return [round(v, 3) for v in w.tolist()]
print("Adam,  L2 (weight_decay=0.5):", run(torch.optim.Adam,  weight_decay=0.5))
print("AdamW, decay (weight_decay=0.5):", run(torch.optim.AdamW, weight_decay=0.5))

Output:

Adam,  L2 (weight_decay=0.5): [0.912, 0.258]
AdamW, decay (weight_decay=0.5): [0.566, 0.597]

The gradients average to zero, so only the decay should move the weights. Adam + L2: the noisy weight barely moves (0.912), the quiet one collapses (0.258). AdamW: both shrink alike (0.566, 0.597; (1 − 0.001·0.5)^1000 ≈ 0.61, plus a little noise).

Which optimiser should I use?

Situation A good start
Most deep learning (MLPs, transformers, first try) AdamW(params, lr=1e-3, weight_decay=0.01)
Image models with long, well-tuned training SGD(params, lr=0.1, momentum=0.9, nesterov=True) + a schedule (Episode 62)
Sparse features (e.g. word counts) AdaGrad can still work well
Comparing optimisers same seed, same data, loss curves, and tune lr for each one

Common mistakes

Mistake Exact message / symptom Why Fix
lr too big for a steep direction loss=7.847e+09 |1 − lr·curvature| > 1 smaller lr, momentum, Adam
lr tiny so nothing explodes x still −1.469 after 100 steps the flat direction crawls momentum / adaptive optimiser
AdaGrad on a long run steps 0.1 → 0.0032 the squared-gradient sum only grows RMSProp / Adam
Adam(..., weight_decay=λ) as weight decay 0.912 vs 0.258 L2 is divided by √v̂ AdamW
Nesterov with the same lr as plain SGD loss inf at lr=0.019 momentum makes steps effectively larger lower lr when adding momentum
Comparing optimisers at one shared lr an unfair race each has its own best lr tune lr per optimiser
Forgetting opt.zero_grad() gradients add up across steps backward() accumulates zero_grad() every step

Mini project: a fair race (tune the learning rate per optimiser)

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(61)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
def final_loss(make_opt):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 16), nn.Tanh(), nn.Linear(16, 1))
    opt, loss_fn = make_opt(model.parameters()), nn.BCEWithLogitsLoss()
    for epoch in range(20):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    return loss_fn(model(X), y).item()
optimisers = {"SGD": lambda p, lr: torch.optim.SGD(p, lr=lr),
              "SGD+momentum": lambda p, lr: torch.optim.SGD(p, lr=lr, momentum=0.9),
              "RMSprop": lambda p, lr: torch.optim.RMSprop(p, lr=lr),
              "Adam": lambda p, lr: torch.optim.Adam(p, lr=lr),
              "AdamW": lambda p, lr: torch.optim.AdamW(p, lr=lr, weight_decay=0.01)}
for name, make in optimisers.items():
    results = {lr: final_loss(lambda p: make(p, lr)) for lr in (0.001, 0.01, 0.1, 1.0)}
    best = min(results, key=results.get)
    print(f"{name:13}", "  ".join(f"lr {lr}: {l:.3f}" for lr, l in results.items()), f"| best lr {best}")

Output:

SGD           lr 0.001: 0.608  lr 0.01: 0.475  lr 0.1: 0.368  lr 1.0: 0.260 | best lr 1.0
SGD+momentum  lr 0.001: 0.475  lr 0.01: 0.399  lr 0.1: 0.024  lr 1.0: 0.102 | best lr 0.1
RMSprop       lr 0.001: 0.394  lr 0.01: 0.138  lr 0.1: 0.044  lr 1.0: 1.118 | best lr 0.1
Adam          lr 0.001: 0.445  lr 0.01: 0.029  lr 0.1: 0.070  lr 1.0: 0.567 | best lr 0.01
AdamW         lr 0.001: 0.446  lr 0.01: 0.030  lr 0.1: 0.034  lr 1.0: 0.382 | best lr 0.01

Each optimiser has a different best learning rate: 1.0 for plain SGD, 0.1 for SGD with momentum and RMSprop, 0.01 for Adam and AdamW. Tuned, SGD with momentum (0.024) even edges out Adam (0.029). Tuning lr matters as much as choosing the optimiser.

Practice

1. Add Nesterov to the race and re-run everything with lr = 0.1.

Solution

Add race("Nesterov", lambda p: torch.optim.SGD(p, lr=0.1, momentum=0.9, nesterov=True)) and change every lr=0.01 to lr=0.1 in the race program. Output:

epoch:            1      5     10     20
SGD           0.493  0.466  0.454  0.368
SGD+momentum  0.468  0.179  0.050  0.024
AdaGrad       0.437  0.109  0.049  0.059
RMSprop       0.284  0.131  0.078  0.044
Adam          0.381  0.103  0.057  0.070
Nesterov      0.466  0.119  0.035  0.048

At lr = 0.1 momentum and Nesterov do far better (0.024 and 0.048 instead of ~0.4), while Adam gets worse (0.029 → 0.070), because its steps are already about lr each: the best lr depends on the optimiser.

2. Write RMSProp by hand and check it against torch.optim.RMSprop.

Solution
import torch
torch.manual_seed(0)
target = torch.tensor([3.0, -2.0])
def grad_of(p): return 2 * (p - target)                  # gradient of (p - target)^2
lr, rho, eps = 0.01, 0.99, 1e-8
p = torch.zeros(2); s = torch.zeros(2)                   # our RMSProp
for _ in range(100):
    g = grad_of(p)
    s = rho * s + (1 - rho) * g**2
    p = p - lr * g / (s.sqrt() + eps)
q = torch.zeros(2, requires_grad=True)                   # PyTorch's RMSprop
opt = torch.optim.RMSprop([q], lr=lr, alpha=rho, eps=eps)
for _ in range(100):
    opt.zero_grad(); ((q - target) ** 2).sum().backward(); opt.step()
print("ours:   ", [round(v, 5) for v in p.tolist()])
print("PyTorch:", [round(v, 5) for v in q.tolist()])

Output:

ours:    [1.73736, -1.54992]
PyTorch: [1.73736, -1.54992]

PyTorch's RMSprop uses alpha for ρ (default 0.99) and adds eps after the square root, exactly as here.

3. Set Adam's betas=(0.0, 0.999). What happens?

Solution

β₁ = 0 means no momentum: m̂ is just the current gradient, so Adam becomes RMSProp with bias correction. In the race program, race("Adam b1=0", lambda p: torch.optim.Adam(p, lr=0.01, betas=(0.0, 0.999))) gives (without momentum the loss is noisier and climbs back to 0.174):

epoch:            1      5     10     20
Adam          0.476  0.418  0.055  0.029
Adam b1=0     0.471  0.240  0.098  0.174

4. Re-run the AdamW demo with weight_decay=0.1.

Solution
Adam,  L2 (weight_decay=0.1): [0.95, 0.263]
AdamW, decay (weight_decay=0.1): [0.855, 0.895]

Less decay: AdamW shrinks both weights to about (1 − 0.001·0.1)^1000 ≈ 0.905 (0.855 and 0.895 here, plus noise). Adam + L2 still treats them very differently: the noisy weight stays at 0.95 while the quiet one falls to 0.263.

Quick quiz

Adam([p], lr=0.01), and p's gradient at step 1 is 500. How far does p move?

Answer

About 0.01. At step 1, m̂ = g = 500 and √v̂ = |g| = 500, so the step is lr · 500/500 = lr.

More questions:

  1. What does momentum's β = 0.9 mean?
    AnswerEach step keeps 90% of the previous velocity and adds the new gradient.
  2. Why do AdaGrad's steps shrink forever?
    AnswerIt divides by the square root of a running sum of squared gradients, which only grows.
  3. What problem does bias correction fix?
    Answerm and v start at zero, so their early values are too small; dividing by 1 − β^t removes that bias.
  4. When are L2 and weight decay the same?
    AnswerFor plain SGD (with the decay scaled by the learning rate). Not for Adam.
  5. Which PyTorch class gives decoupled weight decay?
    Answertorch.optim.AdamW.

Interview questions

  1. Explain momentum in one sentence. It updates with a moving average of past gradients, so consistent directions accelerate and oscillating ones cancel.
  2. Adam vs RMSProp? Adam = RMSProp's per-parameter scaling (second moment) + momentum (first moment) + bias correction for both.
  3. Why is Adam's lr easier to set? The update m̂/√v̂ is roughly unit-sized, so the step is about lr per parameter regardless of the gradient's scale.
  4. Why AdamW instead of Adam with weight_decay? In Adam, L2 is rescaled by the adaptive denominator, so decay is uneven; AdamW decouples it so every weight decays by the same fraction (Loshchilov and Hutter, ICLR 2019).
  5. When might you prefer SGD with momentum? Well-tuned long training of large image models, where it often generalises as well or better; it needs a schedule and lr tuning.
  6. What is Nesterov momentum? Momentum that evaluates the gradient at the look-ahead point p − lr·β·v, correcting earlier.

Glossary

Term Meaning (plain words + example)
Optimiser The rule that turns gradients into parameter updates: torch.optim.Adam
SGD (stochastic gradient descent) p = p − lr·g on random mini-batches
Stochastic Random; here, gradients from random batches
Learning rate (lr) The step size: 0.01
Loss surface The loss as a "landscape" over the parameters
Narrow valley (ill-conditioned) Much steeper in some directions than others: x² + 100y²
Zig-zag / oscillation Steps that flip direction every time: y = −0.9, 0.81, −0.729
Divergence (exploding) Steps that grow until the loss is huge: 7.8e9
Momentum Updating with a velocity that remembers past gradients
Velocity (v) The running sum v = β·v + g
β (beta) How much of the old memory is kept: 0.9
Nesterov momentum Momentum that measures the gradient at a look-ahead point
Per-parameter learning rate Each parameter gets its own effective step size
AdaGrad Divides by √(sum of all past squared gradients); steps shrink over time
RMSProp Divides by √(moving average of squared gradients); steps stay steady
ρ (rho) RMSProp's averaging factor: 0.9 (PyTorch's alpha, default 0.99)
ε (epsilon) A tiny number (1e-8) added to avoid dividing by zero
Moving (exponential) average s = ρ·s + (1 − ρ)·new: recent values count more
Moment (statistics) First moment = mean; second (raw) moment = mean of the square
First moment estimate (m) Adam's moving average of the gradient
Second moment estimate (v) Adam's moving average of the squared gradient
Bias correction Dividing by 1 − β^t so early estimates aren't pulled towards 0: m 10 → m̂ 100
Adam Adaptive moment estimation: momentum + RMSProp + bias correction
Loss curve The loss after each epoch, used to compare training runs
Weight decay Shrinking every weight by a small fraction each step
L2 regularisation Adding λ·Σw²/2 to the loss, i.e. λ·w to the gradient
λ (lambda) The regularisation / decay strength: weight_decay=0.5
Decoupled weight decay Weight decay applied separately from the gradient step
AdamW Adam with decoupled weight decay: torch.optim.AdamW
Hyperparameter A setting we choose, not learned: lr, β, weight_decay

Summary

  • Plain SGD uses one step size for every direction: it zig-zags in steep directions and crawls in flat ones.
  • Momentum (and Nesterov) add a velocity: oscillations cancel, steady directions speed up.
  • AdaGrad and RMSProp divide each gradient by its own typical size; RMSProp's moving average keeps steps from shrinking forever.
  • Adam combines both with bias correction; its step is about lr per parameter.
  • Compare optimisers fairly: same seed and data, loss curves, and tune lr for each.
  • Use AdamW for weight decay with Adam: Adam's weight_decay is really L2, which is unevenly rescaled.

Next episode: Learning rates and initialisation: schedules, warmup, one-cycle, the LR finder, Xavier and He initialisation, and gradient clipping.