Machine Learning Full Course · Episode 61. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU). The output shown is the real output.
The big idea: plain SGD uses one learning rate for every parameter and forgets every previous step. In a long, narrow valley that means it zig-zags across the steep direction and crawls along the flat one; a slightly bigger learning rate makes it explode. Two fixes, invented one after the other: momentum (a velocity that remembers past gradients, so the zig-zags cancel and the steady direction speeds up) and per-parameter step sizes (AdaGrad, then RMSProp, divide each gradient by its typical size). Adam combines both, plus bias correction, and became the default optimiser for deep learning. AdamW fixes how Adam applies weight decay.
Sources: Adam was introduced by Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization", published at ICLR 2015 (arXiv 1412.6980). AdamW comes from Ilya Loshchilov and Frank Hutter, "Decoupled Weight Decay Regularization", ICLR 2019, which shows that L2 regularisation and weight decay are equivalent for SGD but not for adaptive methods like Adam, and that decoupling the decay improves Adam's generalisation (arXiv 1711.05101).
Words you need first
- Parameter: a number the model learns, such as a weight or a bias.
- Gradient (Episode 7): the vector of partial derivatives; it points where the loss rises fastest. We step the other way.
- Learning rate,
lr(Episode 8): how far each step goes. - Loss surface / valley: picture the loss as height over the parameters; training walks downhill.
- Optimiser: the rule that turns gradients into parameter updates (
torch.optim.*). Every optimiser haszero_grad()andstep()(Episode 59). - SGD, stochastic gradient descent:
p = p − lr × gradient, computed on small random batches. Stochastic means random. - Mini-batch: a small group of examples used for one step (Episode 60).
The problem: plain SGD in a narrow valley
The loss 0.5 · (x² + 100·y²) is 100 times steeper in y than in x; the best point is (0, 0). We start at (−10, 1).
import torch
def loss_fn(p): # a long, narrow valley: steep in y, flat in x
x, y = p
return 0.5 * (x**2 + 100 * y**2)
for lr in (0.019, 0.021):
p = torch.tensor([-10.0, 1.0], requires_grad=True)
opt = torch.optim.SGD([p], lr=lr) # plain SGD: p = p - lr * gradient
for step in range(1, 101):
loss = loss_fn(p); opt.zero_grad(); loss.backward(); opt.step()
if step in (1, 2, 3, 100):
x, y = p.tolist()
print(f"lr={lr} step {step:3}: x={x:8.3f} y={y:10.3f} loss={loss.item():.4g}")
Output:
lr=0.019 step 1: x= -9.810 y= -0.900 loss=100
lr=0.019 step 2: x= -9.624 y= 0.810 loss=88.62
lr=0.019 step 3: x= -9.441 y= -0.729 loss=79.11
lr=0.019 step 100: x= -1.469 y= 0.000 loss=1.121
lr=0.021 step 1: x= -9.790 y= -1.100 loss=100
lr=0.021 step 2: x= -9.584 y= 1.210 loss=108.4
lr=0.021 step 3: x= -9.383 y= -1.331 loss=119.1
lr=0.021 step 100: x= -1.197 y= 13780.572 loss=7.847e+09
lr = 0.019:yflips sign every step (−0.9, 0.81, −0.729): a zig-zag across the valley.xonly crawls from −10 to −1.469 in 100 steps.lr = 0.021: theyswing grows (×−1.1 per step) and the loss explodes to 7.8 billion.- Why the limit is 0.02: in
yone SGD step multipliesyby(1 − lr·100). That factor must stay between −1 and 1, solr < 2/100 = 0.02. Inxthe factor is(1 − lr), which is close to 1: slow.
Why it failed: one learning rate must serve both a steep and a flat direction, and each step forgets the previous one.
Momentum and Nesterov momentum
A cyclist coasting down a ghat road keeps their speed (memory); side-to-side wobbles cancel; the steady downhill pull adds up.
v = β · v + g # velocity: a running memory of gradients, β (beta) ≈ 0.9
p = p − lr · v
- Momentum: gradients that keep the same sign add up (speed along the valley); gradients that flip sign cancel (less zig-zag).
- β (beta): the fraction of the old velocity kept each step. 0.9 means "remember about the last 10 steps".
- Nesterov momentum: first move along the velocity (look ahead), measure the gradient there, then correct. It reacts earlier, like braking before a bend.
- In PyTorch:
torch.optim.SGD(params, lr=0.01, momentum=0.9, nesterov=True).
import torch
def loss_fn(p): return 0.5 * (p[0]**2 + 100 * p[1]**2)
def run(make_opt, steps=100):
p = torch.tensor([-10.0, 1.0], requires_grad=True); opt = make_opt([p])
for _ in range(steps):
loss = loss_fn(p); opt.zero_grad(); loss.backward(); opt.step()
return loss_fn(p).item()
# momentum from scratch: v = beta * v + g ; p = p - lr * v
p = torch.tensor([-10.0, 1.0]); v = torch.zeros(2); lr, beta = 0.01, 0.9
for step in range(100):
g = torch.tensor([p[0], 100 * p[1]]) # the gradient of our valley
v = beta * v + g # a running "velocity"
p = p - lr * v
print("momentum (ours) loss@100:", f"{loss_fn(p).item():.3g}")
SGD = torch.optim.SGD
print("SGD loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01)):.3g}")
print("SGD momentum loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01, momentum=0.9)):.3g}")
print("SGD Nesterov loss@100:", f"{run(lambda ps: SGD(ps, lr=0.01, momentum=0.9, nesterov=True)):.3g}")
Output:
momentum (ours) loss@100: 0.000438
SGD loss@100: 6.7
SGD momentum loss@100: 0.000438
SGD Nesterov loss@100: 2.51e-05
Same learning rate (0.01), 100 steps: plain SGD 6.7, momentum 0.000438 (our two-line version gives exactly PyTorch's number), Nesterov 2.51e-05. Momentum also changes the safe learning rate: with lr = 0.019 Nesterov diverges in this valley (we tested it), so momentum and lr must be tuned together.
AdaGrad and RMSProp: a step size for every parameter
A tailor measures the sleeve, the collar and the waist separately instead of one tape length for the whole body. Here: divide each parameter's gradient by its own typical size.
AdaGrad: s = s + g² (add up the squared gradients forever)
RMSProp: s = ρ · s + (1 − ρ) · g² (a moving average, ρ (rho) ≈ 0.9)
step = lr · g / (√s + ε) (ε (epsilon) ≈ 1e-8 avoids division by zero)
- AdaGrad (adaptive gradient): parameters with big past gradients get small steps; ones with tiny gradients get big steps. Its weakness:
sonly grows, so steps shrink forever. - RMSProp (root mean square propagation): the same idea with a moving average, so old gradients fade and the step size settles.
import torch
g = torch.tensor([100.0, 0.01]) # one steep direction, one almost flat
lr = 0.1
print("SGD step:", [round(v, 4) for v in (lr * g).tolist()], "<- 10 vs 0.001")
s_ada, s_rms, rho = torch.zeros(2), torch.zeros(2), 0.9
for t in range(1, 1001):
s_ada += g**2 # AdaGrad: ADD UP squared gradients forever
s_rms = rho * s_rms + (1 - rho) * g**2 # RMSProp: a moving AVERAGE of them
if t in (1, 10, 100, 1000):
ada = lr * g / (s_ada.sqrt() + 1e-8); rms = lr * g / (s_rms.sqrt() + 1e-8)
r = lambda s: [round(v, 4) for v in s.tolist()]
print(f"step {t:4}: AdaGrad {r(ada)} RMSProp {r(rms)}")
Output:
SGD step: [10.0, 0.001] <- 10 vs 0.001
step 1: AdaGrad [0.1, 0.1] RMSProp [0.3162, 0.3162]
step 10: AdaGrad [0.0316, 0.0316] RMSProp [0.1239, 0.1239]
step 100: AdaGrad [0.01, 0.01] RMSProp [0.1, 0.1]
step 1000: AdaGrad [0.0032, 0.0032] RMSProp [0.1, 0.1]
Plain SGD steps differ by 10,000× (10 vs 0.001). AdaGrad and RMSProp give both parameters the same step. AdaGrad's step falls 0.1 → 0.0316 → 0.01 → 0.0032 (it is lr/√t for a constant gradient); RMSProp's settles at 0.1 = lr.
Adam: first and second moment estimates, bias correction
A delivery rider remembers which way the last turns went (direction = momentum) and how bumpy each road usually is (scale = RMSProp). On the first ride of the day both memories are empty, so they shouldn't be trusted as they are: bias correction.
m = β₁ · m + (1 − β₁) · g # first moment estimate: moving average of g (β₁ = 0.9)
v = β₂ · v + (1 − β₂) · g² # second moment estimate: moving average of g² (β₂ = 0.999)
m̂ = m / (1 − β₁ᵗ), v̂ = v / (1 − β₂ᵗ) # bias correction; t = step number 1, 2, 3, ...
p = p − lr · m̂ / (√v̂ + ε)
- Moment (statistics): the first moment of a quantity is its mean; the second (raw) moment is the mean of its square. Adam estimates both for every parameter.
- Bias correction:
mandvstart at 0, so early on they are too small (biased towards 0). Dividing by1 − β^texactly removes that bias for a constant gradient. - Adam = adaptive moment estimation. Defaults in PyTorch:
lr=0.001,betas=(0.9, 0.999),eps=1e-8.
import torch
g = torch.tensor([100.0, 0.01]); lr, b1, b2, eps = 0.1, 0.9, 0.999, 1e-8
m = torch.zeros(2); v = torch.zeros(2) # 1st moment (mean) and 2nd moment (mean of g^2)
for t in range(1, 4):
m = b1 * m + (1 - b1) * g # like momentum
v = b2 * v + (1 - b2) * g**2 # like RMSProp
m_hat, v_hat = m / (1 - b1**t), v / (1 - b2**t) # bias correction: they start at 0
step = lr * m_hat / (v_hat.sqrt() + eps)
st = [round(s, 4) for s in step.tolist()]
print(f"t={t}: m={m[0].item():6.2f} m_hat={m_hat[0].item():6.1f} step={st}")
p = torch.tensor([1.0, 1.0], requires_grad=True) # check against PyTorch's Adam
opt = torch.optim.Adam([p], lr=0.1)
for t in range(3):
opt.zero_grad(); (p * g).sum().backward(); opt.step()
print("torch Adam after 3 steps, p =", [round(x, 4) for x in p.tolist()])
Output:
t=1: m= 10.00 m_hat= 100.0 step=[0.1, 0.1]
t=2: m= 19.00 m_hat= 100.0 step=[0.1, 0.1]
t=3: m= 27.10 m_hat= 100.0 step=[0.1, 0.1]
torch Adam after 3 steps, p = [0.7, 0.7]
At step 1, m = 0.1 × 100 = 10, and m̂ = 10 / (1 − 0.9) = 100, the true gradient. The step is 0.1 = lr for both parameters, whatever the gradient size: in Adam, lr is roughly the size of each step. PyTorch's Adam moves both from 1.0 to 0.7 in 3 steps of 0.1, matching ours.
The race: comparing optimisers on loss curves
A loss curve is the loss after each epoch; lower and earlier is better. A fair race uses the same data, the same starting weights (torch.manual_seed(0) before building each model) and the same batches.
import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader; torch.manual_seed(61)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1) # the circle again
print(f"{'epoch:':13}", " ".join(f"{e:>5}" for e in (1, 5, 10, 20)))
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
def race(name, make_opt):
torch.manual_seed(0) # the SAME starting weights for everyone
model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 16), nn.Tanh(), nn.Linear(16, 1))
opt, loss_fn, curve = make_opt(model.parameters()), nn.BCEWithLogitsLoss(), []
for epoch in range(20):
for xb, yb in loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
curve.append(loss_fn(model(X), y).item())
print(f"{name:13}", " ".join(f"{curve[e - 1]:.3f}" for e in (1, 5, 10, 20)))
race("SGD", lambda p: torch.optim.SGD(p, lr=0.01))
race("SGD+momentum", lambda p: torch.optim.SGD(p, lr=0.01, momentum=0.9))
race("AdaGrad", lambda p: torch.optim.Adagrad(p, lr=0.01))
race("RMSprop", lambda p: torch.optim.RMSprop(p, lr=0.01))
race("Adam", lambda p: torch.optim.Adam(p, lr=0.01))
Output (loss on all 1,000 points after epochs 1, 5, 10 and 20):
epoch: 1 5 10 20
SGD 0.647 0.537 0.493 0.475
SGD+momentum 0.508 0.467 0.457 0.399
AdaGrad 0.502 0.470 0.462 0.443
RMSprop 0.436 0.114 0.065 0.138
Adam 0.476 0.418 0.055 0.029
With lr = 0.01 for everyone: Adam wins (0.029), RMSprop is fast but bounces back up (0.065 → 0.138), SGD, momentum and AdaGrad are still near 0.4–0.48. One race, one problem, one learning rate: each optimiser has its own best learning rate (see the mini project). SGD with momentum and a tuned learning rate is still common for image models.
AdamW: weight decay vs L2
Weight decay: shrink every weight a little each step, so the model keeps small weights and a simpler function (the idea of Episode 21's ridge regression).
L2 regularisation: g = g + λ·w then the usual optimiser step
Decoupled weight decay: w = w − lr·λ·w separately, after the step (λ = lambda, the decay strength)
- For plain SGD these are the same update.
- For Adam they are not: with L2, the
λ·wterm is insideg, so Adam divides it by√v̂. Weights with big gradients (bigv̂) barely decay; weights with tiny gradients decay a lot. - In PyTorch,
torch.optim.Adam(..., weight_decay=λ)adds L2 to the gradient;torch.optim.AdamW(..., weight_decay=λ)decays the weights directly (its defaultweight_decayis 0.01).
import torch
torch.manual_seed(61)
def run(opt_class, **kw):
w = torch.tensor([1.0, 1.0], requires_grad=True) # w[0]: big noisy gradients, w[1]: tiny
opt = opt_class([w], lr=0.001, **kw)
for _ in range(1000):
noise = torch.randn(2) * torch.tensor([10.0, 0.01])
loss = (w * noise).sum() # gradients that average out to zero
opt.zero_grad(); loss.backward(); opt.step()
return [round(v, 3) for v in w.tolist()]
print("Adam, L2 (weight_decay=0.5):", run(torch.optim.Adam, weight_decay=0.5))
print("AdamW, decay (weight_decay=0.5):", run(torch.optim.AdamW, weight_decay=0.5))
Output:
Adam, L2 (weight_decay=0.5): [0.912, 0.258]
AdamW, decay (weight_decay=0.5): [0.566, 0.597]
The gradients average to zero, so only the decay should move the weights. Adam + L2: the noisy weight barely moves (0.912), the quiet one collapses (0.258). AdamW: both shrink alike (0.566, 0.597; (1 − 0.001·0.5)^1000 ≈ 0.61, plus a little noise).
Which optimiser should I use?
| Situation | A good start |
|---|---|
| Most deep learning (MLPs, transformers, first try) | AdamW(params, lr=1e-3, weight_decay=0.01) |
| Image models with long, well-tuned training | SGD(params, lr=0.1, momentum=0.9, nesterov=True) + a schedule (Episode 62) |
| Sparse features (e.g. word counts) | AdaGrad can still work well |
| Comparing optimisers | same seed, same data, loss curves, and tune lr for each one |
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
lr too big for a steep direction |
loss=7.847e+09 |
|1 − lr·curvature| > 1 |
smaller lr, momentum, Adam |
lr tiny so nothing explodes |
x still −1.469 after 100 steps |
the flat direction crawls | momentum / adaptive optimiser |
| AdaGrad on a long run | steps 0.1 → 0.0032 | the squared-gradient sum only grows | RMSProp / Adam |
Adam(..., weight_decay=λ) as weight decay |
0.912 vs 0.258 | L2 is divided by √v̂ |
AdamW |
Nesterov with the same lr as plain SGD |
loss inf at lr=0.019 |
momentum makes steps effectively larger | lower lr when adding momentum |
Comparing optimisers at one shared lr |
an unfair race | each has its own best lr |
tune lr per optimiser |
Forgetting opt.zero_grad() |
gradients add up across steps | backward() accumulates |
zero_grad() every step |
Mini project: a fair race (tune the learning rate per optimiser)
import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(61)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
def final_loss(make_opt):
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 16), nn.Tanh(), nn.Linear(16, 1))
opt, loss_fn = make_opt(model.parameters()), nn.BCEWithLogitsLoss()
for epoch in range(20):
for xb, yb in loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
return loss_fn(model(X), y).item()
optimisers = {"SGD": lambda p, lr: torch.optim.SGD(p, lr=lr),
"SGD+momentum": lambda p, lr: torch.optim.SGD(p, lr=lr, momentum=0.9),
"RMSprop": lambda p, lr: torch.optim.RMSprop(p, lr=lr),
"Adam": lambda p, lr: torch.optim.Adam(p, lr=lr),
"AdamW": lambda p, lr: torch.optim.AdamW(p, lr=lr, weight_decay=0.01)}
for name, make in optimisers.items():
results = {lr: final_loss(lambda p: make(p, lr)) for lr in (0.001, 0.01, 0.1, 1.0)}
best = min(results, key=results.get)
print(f"{name:13}", " ".join(f"lr {lr}: {l:.3f}" for lr, l in results.items()), f"| best lr {best}")
Output:
SGD lr 0.001: 0.608 lr 0.01: 0.475 lr 0.1: 0.368 lr 1.0: 0.260 | best lr 1.0
SGD+momentum lr 0.001: 0.475 lr 0.01: 0.399 lr 0.1: 0.024 lr 1.0: 0.102 | best lr 0.1
RMSprop lr 0.001: 0.394 lr 0.01: 0.138 lr 0.1: 0.044 lr 1.0: 1.118 | best lr 0.1
Adam lr 0.001: 0.445 lr 0.01: 0.029 lr 0.1: 0.070 lr 1.0: 0.567 | best lr 0.01
AdamW lr 0.001: 0.446 lr 0.01: 0.030 lr 0.1: 0.034 lr 1.0: 0.382 | best lr 0.01
Each optimiser has a different best learning rate: 1.0 for plain SGD, 0.1 for SGD with momentum and RMSprop, 0.01 for Adam and AdamW. Tuned, SGD with momentum (0.024) even edges out Adam (0.029). Tuning lr matters as much as choosing the optimiser.
Practice
1. Add Nesterov to the race and re-run everything with lr = 0.1.
Solution
Add race("Nesterov", lambda p: torch.optim.SGD(p, lr=0.1, momentum=0.9, nesterov=True)) and change every lr=0.01 to lr=0.1 in the race program. Output:
epoch: 1 5 10 20
SGD 0.493 0.466 0.454 0.368
SGD+momentum 0.468 0.179 0.050 0.024
AdaGrad 0.437 0.109 0.049 0.059
RMSprop 0.284 0.131 0.078 0.044
Adam 0.381 0.103 0.057 0.070
Nesterov 0.466 0.119 0.035 0.048
At lr = 0.1 momentum and Nesterov do far better (0.024 and 0.048 instead of ~0.4), while Adam gets worse (0.029 → 0.070), because its steps are already about lr each: the best lr depends on the optimiser.
2. Write RMSProp by hand and check it against torch.optim.RMSprop.
Solution
import torch
torch.manual_seed(0)
target = torch.tensor([3.0, -2.0])
def grad_of(p): return 2 * (p - target) # gradient of (p - target)^2
lr, rho, eps = 0.01, 0.99, 1e-8
p = torch.zeros(2); s = torch.zeros(2) # our RMSProp
for _ in range(100):
g = grad_of(p)
s = rho * s + (1 - rho) * g**2
p = p - lr * g / (s.sqrt() + eps)
q = torch.zeros(2, requires_grad=True) # PyTorch's RMSprop
opt = torch.optim.RMSprop([q], lr=lr, alpha=rho, eps=eps)
for _ in range(100):
opt.zero_grad(); ((q - target) ** 2).sum().backward(); opt.step()
print("ours: ", [round(v, 5) for v in p.tolist()])
print("PyTorch:", [round(v, 5) for v in q.tolist()])
Output:
ours: [1.73736, -1.54992]
PyTorch: [1.73736, -1.54992]
PyTorch's RMSprop uses alpha for ρ (default 0.99) and adds eps after the square root, exactly as here.
3. Set Adam's betas=(0.0, 0.999). What happens?
Solution
β₁ = 0 means no momentum: m̂ is just the current gradient, so Adam becomes RMSProp with bias correction. In the race program, race("Adam b1=0", lambda p: torch.optim.Adam(p, lr=0.01, betas=(0.0, 0.999))) gives (without momentum the loss is noisier and climbs back to 0.174):
epoch: 1 5 10 20
Adam 0.476 0.418 0.055 0.029
Adam b1=0 0.471 0.240 0.098 0.174
4. Re-run the AdamW demo with weight_decay=0.1.
Solution
Adam, L2 (weight_decay=0.1): [0.95, 0.263]
AdamW, decay (weight_decay=0.1): [0.855, 0.895]
Less decay: AdamW shrinks both weights to about (1 − 0.001·0.1)^1000 ≈ 0.905 (0.855 and 0.895 here, plus noise). Adam + L2 still treats them very differently: the noisy weight stays at 0.95 while the quiet one falls to 0.263.
Quick quiz
Adam([p], lr=0.01), and p's gradient at step 1 is 500. How far does p move?
Answer
About 0.01. At step 1, m̂ = g = 500 and √v̂ = |g| = 500, so the step is lr · 500/500 = lr.
More questions:
- What does momentum's β = 0.9 mean?
Answer
Each step keeps 90% of the previous velocity and adds the new gradient. - Why do AdaGrad's steps shrink forever?
Answer
It divides by the square root of a running sum of squared gradients, which only grows. - What problem does bias correction fix?
Answer
mandvstart at zero, so their early values are too small; dividing by1 − β^tremoves that bias. - When are L2 and weight decay the same?
Answer
For plain SGD (with the decay scaled by the learning rate). Not for Adam. - Which PyTorch class gives decoupled weight decay?
Answer
torch.optim.AdamW.
Interview questions
- Explain momentum in one sentence. It updates with a moving average of past gradients, so consistent directions accelerate and oscillating ones cancel.
- Adam vs RMSProp? Adam = RMSProp's per-parameter scaling (second moment) + momentum (first moment) + bias correction for both.
- Why is Adam's
lreasier to set? The updatem̂/√v̂is roughly unit-sized, so the step is aboutlrper parameter regardless of the gradient's scale. - Why AdamW instead of Adam with
weight_decay? In Adam, L2 is rescaled by the adaptive denominator, so decay is uneven; AdamW decouples it so every weight decays by the same fraction (Loshchilov and Hutter, ICLR 2019). - When might you prefer SGD with momentum? Well-tuned long training of large image models, where it often generalises as well or better; it needs a schedule and
lrtuning. - What is Nesterov momentum? Momentum that evaluates the gradient at the look-ahead point
p − lr·β·v, correcting earlier.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Optimiser | The rule that turns gradients into parameter updates: torch.optim.Adam |
| SGD (stochastic gradient descent) | p = p − lr·g on random mini-batches |
| Stochastic | Random; here, gradients from random batches |
Learning rate (lr) |
The step size: 0.01 |
| Loss surface | The loss as a "landscape" over the parameters |
| Narrow valley (ill-conditioned) | Much steeper in some directions than others: x² + 100y² |
| Zig-zag / oscillation | Steps that flip direction every time: y = −0.9, 0.81, −0.729 |
| Divergence (exploding) | Steps that grow until the loss is huge: 7.8e9 |
| Momentum | Updating with a velocity that remembers past gradients |
Velocity (v) |
The running sum v = β·v + g |
| β (beta) | How much of the old memory is kept: 0.9 |
| Nesterov momentum | Momentum that measures the gradient at a look-ahead point |
| Per-parameter learning rate | Each parameter gets its own effective step size |
| AdaGrad | Divides by √(sum of all past squared gradients); steps shrink over time |
| RMSProp | Divides by √(moving average of squared gradients); steps stay steady |
| ρ (rho) | RMSProp's averaging factor: 0.9 (PyTorch's alpha, default 0.99) |
| ε (epsilon) | A tiny number (1e-8) added to avoid dividing by zero |
| Moving (exponential) average | s = ρ·s + (1 − ρ)·new: recent values count more |
| Moment (statistics) | First moment = mean; second (raw) moment = mean of the square |
First moment estimate (m) |
Adam's moving average of the gradient |
Second moment estimate (v) |
Adam's moving average of the squared gradient |
| Bias correction | Dividing by 1 − β^t so early estimates aren't pulled towards 0: m 10 → m̂ 100 |
| Adam | Adaptive moment estimation: momentum + RMSProp + bias correction |
| Loss curve | The loss after each epoch, used to compare training runs |
| Weight decay | Shrinking every weight by a small fraction each step |
| L2 regularisation | Adding λ·Σw²/2 to the loss, i.e. λ·w to the gradient |
| λ (lambda) | The regularisation / decay strength: weight_decay=0.5 |
| Decoupled weight decay | Weight decay applied separately from the gradient step |
| AdamW | Adam with decoupled weight decay: torch.optim.AdamW |
| Hyperparameter | A setting we choose, not learned: lr, β, weight_decay |
Summary
- Plain SGD uses one step size for every direction: it zig-zags in steep directions and crawls in flat ones.
- Momentum (and Nesterov) add a velocity: oscillations cancel, steady directions speed up.
- AdaGrad and RMSProp divide each gradient by its own typical size; RMSProp's moving average keeps steps from shrinking forever.
- Adam combines both with bias correction; its step is about
lrper parameter. - Compare optimisers fairly: same seed and data, loss curves, and tune
lrfor each. - Use AdamW for weight decay with Adam: Adam's
weight_decayis really L2, which is unevenly rescaled.
Next episode: Learning rates and initialisation: schedules, warmup, one-cycle, the LR finder, Xavier and He initialisation, and gradient clipping.