Machine Learning Full Course · Episode 8. Every code sample below was run with Python 3.13 in plain Python (no libraries); the output shown is the real output. The data is made up for teaching. You need episode 6 (derivatives) and episode 7 (partial derivatives and the gradient).
The problem: how much milk to buy?
Every morning at 5 am Raju buys milk for the day. Too little and customers walk away; too much and it spoils. He has noticed that more people walking past means more cups, and wrote down five days:
| People walking past (hundreds) | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Cups sold | 32 | 55 | 79 | 98 | 124 |
He guesses a rule, a model: cups = w × people + b.
- w: extra cups per hundred people (a weight)
- b: a fixed base number of cups (a bias)
w and b are the model's parameters: numbers the model learns from data. Which w and b fit best?
Guessing by hand
people = [1, 2, 3, 4, 5] # people walking past, in hundreds
cups = [32, 55, 79, 98, 124] # cups Raju sold that day
def predict(w, b, x):
return w * x + b
for w, b in [(20, 0), (25, 5), (22, 10)]:
print(w, b, [predict(w, b, x) for x in people])
Output:
20 0 [20, 40, 60, 80, 100]
25 5 [30, 55, 80, 105, 130]
22 10 [32, 54, 76, 98, 120]
(20, 0) is too low everywhere, (25, 5) overshoots on busy days, (22, 10) is close but 4 short on the last day. Comparing five numbers by eye can't tell us which guess is best, and guessing never ends. We need one number for "how wrong".
The obvious "how wrong" fails
For each day, the miss (error) is prediction − actual. The obvious score: average the misses. Test it on a useless model: a flat line that always predicts 77.6 cups (the average), whatever the crowd.
people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]
def avg_error(w, b): # plain average of the misses
errs = [w * x + b - c for x, c in zip(people, cups)]
return sum(errs) / len(errs)
def loss(w, b): # average of the SQUARED misses
errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
return sum(errs) / len(errs)
print("flat line 77.6:", avg_error(0, 77.6), loss(0, 77.6))
print("good line: ", avg_error(22, 10), loss(22, 10))
Output:
flat line 77.6: -5.684341886080802e-15 1032.24
good line: -1.6 5.2
- The flat line's plain average miss is −5.68 × 10⁻¹⁵: floating-point rounding noise, i.e. zero. A useless model scores "perfect".
- Why: it predicts too many cups on quiet days (positive misses) and too few on busy days (negative misses), and they cancel.
- The fix: square each miss first. A square is never negative, so nothing cancels, and big misses count much more. Flat line 1032.24, good line 5.2: now the score tells the truth.
zip(people, cups)walks through both lists in step: (1, 32), (2, 55), …
Loss function, objective, optimisation
| Term | Meaning | Here |
|---|---|---|
| Loss function (also cost function) | turns all of a model's mistakes into one number | average of the squared misses |
| Objective | what we want, stated precisely | make the loss as small as possible |
| Optimisation | searching for the parameters that achieve the objective | find w, b with the lowest loss |
Real-life example: a bowler's economy rate sums up a whole spell in one number. The average of squared errors is called mean squared error (MSE); episode 18 uses it by that name. Training a model is optimisation. Picture loss(w, b) as a landscape (w east, b north, loss = height): we want its lowest point.
First attempt: step by the full gradient (it explodes)
The gradient points uphill (episode 7), so step against it: w ← w − ∂loss/∂w, b ← b − ∂loss/∂b.
The partial derivatives of loss = (1/n) Σ (wx + b − c)² are, by the chain rule:
- ∂loss/∂w = (1/n) Σ 2(wx + b − c)·x
- ∂loss/∂b = (1/n) Σ 2(wx + b − c)
people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]
def loss(w, b):
errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
return sum(errs) / len(errs)
def grad(w, b): # partial derivatives of the loss
n = len(people)
gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
return gw, gb
w, b = 0.0, 0.0
for step in range(1, 301):
gw, gb = grad(w, b)
w, b = w - gw, b - gb # step downhill by the full gradient
if step <= 3 or step % 100 == 0:
print(step, f"w={w:.3g} b={b:.3g} loss={loss(w, b):.3g}")
Output:
1 w=556 b=155 loss=3.62e+06
2 w=-1.21e+04 b=-3.34e+03 loss=1.86e+09
3 w=2.74e+05 b=7.58e+04 loss=9.55e+11
100 w=-7.97e+136 b=-2.21e+136 loss=8.08e+274
Traceback (most recent call last):
File "main.py", line 19, in <module>
print(step, f"w={w:.3g} b={b:.3g} loss={loss(w, b):.3g}")
~~~~^^^^^^
File "main.py", line 5, in loss
errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
~~~~~~~~~~~~~~~~^^~~
OverflowError: (34, 'Numerical result out of range')
f"{w:.3g}"prints 3 significant digits, switching to e-notation for huge numbers.- Why it fails: at (0, 0) the gradient is in the hundreds. The direction is right, but jumping hundreds of units flies over the valley and lands higher on the other side; each jump is bigger. Running away like this is called divergence. Python's floats can't go above about 1.8 × 10³⁰⁸, so squaring 10¹³⁷ raises
OverflowError.
The learning rate
Real-life example: walking down a hill in thick fog. You can't see the valley, only feel the slope under your feet (the gradient), so you choose your stride. That stride is the learning rate η (Greek "eta"): a small number the gradient is multiplied by.
Gradient descent: repeat w ← w − η · ∂loss/∂w (and the same for b) until the ground is flat.
people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]
def loss(w, b):
errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
return sum(errs) / len(errs)
def grad(w, b):
n = len(people)
gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
return gw, gb
for lr in [0.001, 0.05, 0.1]: # the learning rate
w, b = 0.0, 0.0
for step in range(200):
gw, gb = grad(w, b)
w, b = w - lr * gw, b - lr * gb
print(f"lr={lr}: w={w:.4g} b={b:.4g} loss={loss(w, b):.4g}")
Output:
lr=0.001: w=23.28 b=6.657 loss=3.563
lr=0.05: w=22.73 b=9.401 loss=1.662
lr=0.1: w=-2.977e+28 b=-8.246e+27 loss=1.129e+58
| Learning rate | What happened | Name |
|---|---|---|
| 0.001 | still walking down after 200 steps | too small: slow |
| 0.05 | settled at the bottom | convergence |
| 0.1 | blew up | divergence |
Converged means the steps no longer change anything (the gradient is ≈ 0). Doubling 0.05 to 0.1 flips the result from convergence to divergence: the learning rate is the most important setting in gradient descent.
Why there is a limit: on a bowl-shaped loss, a step overshoots the bottom when η is larger than about 2 ÷ (the steepest curvature). For this data that limit is ≈ 0.0845 (see practice 1).
Parameters vs hyperparameters
| Parameters | Hyperparameters | |
|---|---|---|
| Examples | w, b (later: millions of neural-network weights) | learning rate, number of steps, tolerance |
| Who sets them | learned from the data | chosen by you before training |
| Analogy | how the cake turns out | the oven temperature you set |
Later in the course (episode 42) we tune hyperparameters automatically.
Convergence: a stopping rule
people, cups = [1, 2, 3, 4, 5], [32, 55, 79, 98, 124]
def grad(w, b):
n = len(people)
gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
return gw, gb
def gradient_descent(lr=0.05, tol=1e-6, max_steps=10_000):
w, b = 0.0, 0.0 # parameters: learned
for step in range(1, max_steps + 1): # lr, tol, max_steps: hyperparameters
gw, gb = grad(w, b)
w, b = w - lr * gw, b - lr * gb
if (gw ** 2 + gb ** 2) ** 0.5 < tol: # nearly flat: converged
return w, b, step
return w, b, max_steps
w, b, steps = gradient_descent()
print(f"w={w:.4f} b={b:.4f} after {steps} steps")
print("100 people ->", round(w * 1 + b), "cups; 600 ->", round(w * 6 + b))
Output:
w=22.7000 b=9.5000 after 815 steps
100 people -> 32 cups; 600 -> 146
- Tolerance (
tol): how flat counts as flat. We stop when the gradient's length √(gw² + gb²) < 10⁻⁶. max_stepsis a safety net in case it never converges.10_000is just 10000; Python allows underscores in numbers for readability.- The answer, w = 22.7 and b = 9.5, is the exact best line (the same as the formula used in episode 18).
Local vs global minimum
Gradient descent only feels the ground under its feet. On the hilly function from episode 6, f(x) = x⁴ − 4x² + x, the starting point decides which valley you end in:
def hills(x):
return x ** 4 - 4 * x ** 2 + x
def d_hills(x):
return 4 * x ** 3 - 8 * x + 1
for start in [1.0, -2.0]:
x = start
for _ in range(500):
x = x - 0.01 * d_hills(x)
print(f"start {start:+}: ended at x={x:.3f}, height {hills(x):.2f}")
Output:
start +1.0: ended at x=1.347, height -2.62
start -2.0: ended at x=-1.473, height -5.44
- Local minimum: lowest only in its neighbourhood (x ≈ 1.347, −2.62).
- Global minimum: lowest of all (x ≈ −1.473, −5.44).
f"{start:+}"always prints the sign.
Raju's loss (and linear and logistic regression losses) is convex, a single bowl, so any start reaches the global minimum. Neural-network losses are non-convex; common remedies are several random starts and noisy steps (stochastic gradient descent, episode 18).
Saddle points
A horse's saddle curves up from front to back and down from side to side. f(x, y) = x² − y² has this shape: at (0, 0) the gradient is zero, yet it is not a minimum. That is a saddle point.
def saddle(x, y):
return x ** 2 - y ** 2
def grad(x, y):
return 2 * x, -2 * y
for x, y in [(1.0, 0.0), (1.0, 0.001)]:
for step in range(1, 1001):
gx, gy = grad(x, y)
x, y = x - 0.01 * gx, y - 0.01 * gy
if step in (200, 600, 1000):
print(step, f"x={x:.4f} y={y:.4f} f={saddle(x, y):.4f}")
print("---")
Output:
200 x=0.0176 y=0.0000 f=0.0003
600 x=0.0000 y=0.0000 f=0.0000
1000 x=0.0000 y=0.0000 f=0.0000
---
200 x=0.0176 y=0.0525 f=-0.0024
600 x=0.0000 y=144.5783 f=-20902.8794
1000 x=0.0000 y=398264.6517 f=-158614732760.3709
---
Starting exactly on y = 0, it slides to (0, 0) and stops for good (every step is zero). A nudge of 0.001 in y grows and escapes, and f keeps falling (this function has no minimum at all). A zero gradient can be a minimum, a maximum or a saddle point. In high dimensions saddle points are very common; noise in the steps helps escape them.
Common mistakes
| Mistake | Result | Why | Fix |
|---|---|---|---|
| average the plain misses | -5.684341886080802e-15 for a useless line |
positive and negative misses cancel | square them (the loss) |
w, b = w - gw, b - gb |
OverflowError: (34, 'Numerical result out of range') |
full steps overshoot further each time: divergence | multiply by a learning rate |
lr = 0.1 on this data |
loss 1.129e+58 | above the stability limit (≈ 0.0845) | lower it; watch the loss each step |
lr = 0.001 |
loss still 3.563 after 200 steps | steps too small | raise it; try 0.01, 0.03, 0.05 |
| no stopping rule | wasted steps or an endless loop | nothing checks convergence | stop when the gradient's length < tol, plus max_steps |
| gradient ≈ 0 ⇒ best | stuck at −2.62, or on a saddle | local minimum / saddle point | several starts; check the result |
loss = sum(errs) without dividing by n |
the gradient grows with the data size, so a good lr stops working | the loss depends on n | average (divide by n) |
Mini project: Raju's best price and sugar, by gradient descent
Episode 7's profit(p, s) has two knobs. To maximise profit, minimise minus profit with a reusable gradient-descent function:
def gradient_descent(grad, start, lr, steps):
x = list(start)
for _ in range(steps):
g = grad(*x)
x = [xi - lr * gi for xi, gi in zip(x, g)]
return x
def neg_profit_grad(p, s): # gradient of MINUS profit
m = p - 5 - s
c = 200 - 5 * p + 20 * s - 5 * s ** 2
return -(c - 5 * m), -(-c + m * (20 - 10 * s))
p, s = gradient_descent(neg_profit_grad, [20, 1], lr=0.005, steps=2000)
c = 200 - 5 * p + 20 * s - 5 * s ** 2
print(f"best price Rs {p:.2f}, sugar {s:.2f} spoons, profit Rs {(p - 5 - s) * c:.2f}")
print("gradient there:", [round(g, 6) for g in neg_profit_grad(p, s)])
Output:
best price Rs 25.12, sugar 1.50 spoons, profit Rs 1734.45
gradient there: [-0.0, 0.0]
The gradient is 0 at the answer (-0.0 is just zero with a minus sign from rounding): Rs 25.12 a cup with one and a half spoons, up from Rs 1,610 at (20, 1). Try lr = 0.05: it diverges.
Practice
- Find the largest learning rate that still converges on Raju's data.
Solution
Run 1000 steps for several values:
people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]
def loss(w, b):
errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
return sum(errs) / len(errs)
def grad(w, b):
n = len(people)
gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
return gw, gb
for lr in [0.08, 0.084, 0.085, 0.09]:
w, b = 0.0, 0.0
for step in range(1000):
gw, gb = grad(w, b)
w, b = w - lr * gw, b - lr * gb
print(f"lr={lr}: loss={loss(w, b):.4g}")
Output:
lr=0.08: loss=1.66
lr=0.084: loss=1.66
lr=0.085: loss=3.762e+13
lr=0.09: loss=4.738e+109
The limit is between 0.084 and 0.085. (Theory: the loss's Hessian is [[22, 6], [6, 2]], its largest eigenvalue ≈ 23.66, and 2 ÷ 23.66 ≈ 0.0845.)
- Add a sixth day: 7 hundred people, 168 cups. Retrain.
Solution
With people = [1, 2, 3, 4, 5, 7], cups = [32, 55, 79, 98, 124, 168] and 2000 steps, both lr = 0.05 and lr = 0.03 give w=22.64 b=9.643 loss=1.393. Bigger inputs make the loss steeper, so the stability limit drops (to about 0.055 here): 0.05 is now close to the edge.
- Gradient descent on loss(w) = (w − 3)² + 2 with lr = 0.1 from w = 0.
Solution
w = 0.0
for step in range(1, 51):
w = w - 0.1 * 2 * (w - 3) # loss = (w - 3) ** 2 + 2
if step in (1, 2, 10, 50):
print(step, round(w, 4), round((w - 3) ** 2 + 2, 4))
Output:
1 0.6 7.76
2 1.08 5.6864
10 2.6779 2.1038
50 3.0 2.0
It converges to w = 3, loss 2. The "+ 2" shifts the height, not the location of the minimum.
- Run the hills from 10 random starts between −2 and 2.
Solution
import random
def d_hills(x):
return 4 * x ** 3 - 8 * x + 1
random.seed(3)
ends = []
for _ in range(10):
x = random.uniform(-2, 2)
for _ in range(500):
x = x - 0.01 * d_hills(x)
ends.append(round(x, 3))
print(ends)
print("global:", sum(e < 0 for e in ends), "local:", sum(e > 0 for e in ends))
Output:
[-1.473, 1.347, -1.473, 1.347, 1.347, -1.473, -1.473, 1.347, -1.473, -1.473]
global: 6 local: 4
Keep the best of several starts: this is the "random restarts" trick. random.seed(3) makes the "random" starts repeatable.
Quick quiz
def grad(w):
return 2 * (w - 3) # loss = (w - 3) ** 2
w = 0
for _ in range(2):
w = w - 0.5 * grad(w)
print(w)
A) 3.0 B) 1.5 C) 6.0 D) 0
Answer
A) 3.0. Step 1: grad(0) = −6, so w = 0 + 3 = 3.0, exactly the bottom. Step 2: grad(3.0) = 0, so w stays 3.0. With lr = 1, w would jump 0 → 6 → 0 → 6 forever.
Interview questions
- What is gradient descent? An iterative optimisation algorithm: repeatedly move the parameters a small step (learning rate × gradient) against the gradient of the loss until it converges.
- What happens if the learning rate is too high or too low? Too high: overshooting, oscillation or divergence (NaN/overflow). Too low: very slow convergence, possibly stopping early in a poor spot.
- Loss function vs cost function vs objective? Often used interchangeably; strictly, a loss is per example, the cost is the average over the dataset (plus any penalties), and the objective is whatever is optimised (minimised or maximised).
- Why squared error rather than plain error? Plain errors cancel; squares are always positive, punish big errors more, and are smooth (easy to differentiate).
- Parameters vs hyperparameters? Parameters are learned from data (weights, biases); hyperparameters are set before training (learning rate, number of epochs, regularisation strength).
- How do you know gradient descent has converged? The gradient norm or the change in loss falls below a tolerance, or the validation loss stops improving.
- Local minimum vs saddle point? Both have zero gradient; a local minimum curves up in every direction, a saddle curves up in some and down in others. In high dimensions saddle points are far more common.
- Does gradient descent always find the global minimum? Only for convex losses (with a suitable learning rate); otherwise it can stop in a local minimum or slow down near a saddle.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Model | A rule that turns inputs into a prediction | cups = w·x + b |
| Weight (w) | A learned multiplier for an input | 22.7 cups per hundred people |
| Bias (b) | A learned constant added at the end | 9.5 cups |
| Parameter | A number the model learns from data | w, b |
| Error (miss) | prediction − actual | 120 − 124 = −4 |
| Loss function / cost function | One number measuring how wrong the model is | average squared miss |
| Mean squared error (MSE) | The average of squared errors | 5.2 for (22, 10) |
| Objective | What we want, stated precisely | minimise the loss |
| Optimisation | Searching for the parameters that achieve the objective | gradient descent |
| Gradient descent | Repeatedly stepping against the gradient | w ← w − η·∂L/∂w |
| Learning rate (η, lr) | The step-size multiplier | 0.05 |
| Divergence | Steps growing until the numbers blow up | lr = 0.1 → 1e+58 |
| Convergence | Steps settling at a point where nothing changes | w = 22.7, b = 9.5 |
| Tolerance (tol) | How small the gradient must get to count as flat | 1e-6 |
| Hyperparameter | A setting chosen before training | lr, tol, max_steps |
OverflowError |
Python error: a number too big for a float | (34, 'Numerical result out of range') |
| Floating-point noise | Tiny rounding leftovers | −5.68e-15 ≈ 0 |
| Local minimum | Lowest only in its neighbourhood | x ≈ 1.347 |
| Global minimum | Lowest point of all | x ≈ −1.473 |
| Convex / non-convex | One bowl / many valleys | line fit / neural network |
| Saddle point | Zero gradient, curving up one way and down another | (0, 0) of x² − y² |
| Random restarts | Running from several starts and keeping the best | practice 4 |
random.seed(n) |
Makes Python's random numbers repeatable | random.seed(3) |
zip(a, b) |
Walks through two lists in step | zip(people, cups) |
Summary
- A loss function turns all the mistakes into one number; squaring stops misses from cancelling.
- Training = optimisation: find the parameters with the lowest loss.
- Gradient descent steps against the gradient: w ← w − η·gradient, repeated until it converges.
- The learning rate decides everything: too small crawls, too big diverges (real
OverflowError). - Parameters are learned; hyperparameters (lr, steps, tolerance) are chosen.
- Watch out for local minima and saddle points: a zero gradient isn't always the best.
Next episode: Maths for ML 7, probability basics: sample spaces, events, conditional probability, independence and Bayes' theorem.