Machine Learning Full Course · Episode 20. Every code sample below was run with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching.
The big idea: a model can fail in two opposite ways. Too simple, and it is wrong in the same way every time (high bias, underfitting). Too flexible, and it memorises the noise in its training data and changes wildly with every new sample (high variance, overfitting). The expected error on new data is bias² + variance + noise, and your job is to find the model capacity where that sum is smallest. Two plots tell you where you stand: the validation curve and the learning curve.
Three students, one exam
| Student | Strategy | Practice paper | Real exam | ML name |
|---|---|---|---|---|
| 1 | "the answer is always C" | bad | bad | underfitting: too simple to capture the pattern |
| 2 | memorises last year's paper, typos included | 100% | fails new questions | overfitting: memorised the training data, noise and all |
| 3 | understands the chapters | good | good | a good fit that generalises (Episode 15) |
The chai stall experiment
Made-up data: cups of chai sold against temperature. Cold days sell more, with a gentle wiggle; every day also has random noise (standard deviation 8 cups). We have 20 days to learn from (training set) and 200 new days to check on (validation set, Episode 15). Temperature is scaled to −1..1 so high powers stay small. We fit polynomials (Episode 19) of degree 1, 4 and 15:
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 300).reshape(-1, 1)
fig, ax = plt.subplots(1, 3, figsize=(15, 4.4), sharey=True)
for a, d in zip(ax, [1, 4, 15]):
m = poly(d).fit(X_tr, y_tr)
print(f"degree {d:2}: train MSE {mse(y_tr, m.predict(X_tr)):6.1f} val MSE {mse(y_va, m.predict(X_va)):10.1f}")
a.scatter(X_tr, y_tr); a.plot(grid, m.predict(grid), "r"); a.set(title=f"degree {d}", ylim=(0, 220))
plt.savefig("fits.png", dpi=110, bbox_inches="tight")
Output:
degree 1: train MSE 365.7 val MSE 360.2
degree 4: train MSE 25.0 val MSE 65.0
degree 15: train MSE 8.6 val MSE 4280828.2

| Degree | Train MSE | Validation MSE | Diagnosis |
|---|---|---|---|
| 1 | 365.7 | 360.2 | both high: underfitting |
| 4 | 25.0 | 65.0 | both low, close: good fit |
| 15 | 8.6 | 4,280,828.2 | train tiny, validation enormous: overfitting |
Reading training vs validation error
- Underfitting: training error high, validation error high.
- Overfitting: training error low, validation error much higher: a big gap.
- Good fit: both low and close together.
Irreducible error
The noise has standard deviation 8, so variance 8² = 64. Nobody can predict a surprise wedding or a cricket match, so no model can beat an average squared error of 64 on new days. That floor is the irreducible error (also called noise or Bayes error). Degree 15's training MSE of 8.6 is far below the floor: impossible honestly, so it memorised the noise.
Bias and variance
Throwing darts:
| Darts | Meaning | Model |
|---|---|---|
| tight group, left of the bullseye | consistently off | high bias: wrong on average because its assumptions are too simple |
| scattered all over, centred on average | unstable | high variance: predictions change a lot with a different training set |
| tight group on the bullseye | the goal | low bias, low variance |
To "throw" a model many times, train it on many different training sets drawn from the same source, and look at its predictions at fixed test points:
- Bias² = (average prediction − truth)², averaged over test points.
- Variance = how much the predictions spread around their own average.
- For squared error: expected test MSE = bias² + variance + noise.
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 50).reshape(-1, 1); t = grid.ravel() * 17.5 + 22.5 # 50 test days
for d in [1, 4, 7]:
preds = np.array([poly(d).fit(*sample(20)).predict(grid) for _ in range(300)]) # 300 training sets
bias2 = np.mean((preds.mean(axis=0) - truth(t)) ** 2); var = np.mean(preds.var(axis=0))
test = np.mean((preds - truth(t) - rng.normal(0, 8, preds.shape)) ** 2)
print(f"degree {d}: bias² {bias2:.1f} variance {var:.1f}")
print(f" sum + noise 64 = {bias2 + var + 64:.1f} real test MSE {test:.1f}")
Output:
degree 1: bias² 332.9 variance 50.4
sum + noise 64 = 447.3 real test MSE 450.0
degree 4: bias² 1.6 variance 51.5
sum + noise 64 = 117.1 real test MSE 119.0
degree 7: bias² 377.6 variance 57589.8
sum + noise 64 = 58031.4 real test MSE 58029.5
- Degree 1: bias² 332.9, variance 50.4. Mostly bias: underfitting in numbers.
- Degree 4: bias² 1.6, variance 51.5: both small.
- Degree 7: variance 57,589.8: overfitting in numbers. (Its bias² of 377.6 is the average of wild curves, not a systematic shape error.)
- The decomposition checks out: 447.3 vs a measured 450.0, 117.1 vs 119.0, 58,031.4 vs 58,029.5.
Capacity, complexity and the trade-off
- Capacity: how complicated a shape a model is able to fit.
- Model complexity knobs: anything that changes capacity: polynomial degree, tree depth (Episode 29), number of features, number of neural-network layers, regularisation strength (Episode 21, reversed).
- More capacity: bias ↓, variance ↑. Less capacity: bias ↑, variance ↓.
- Bias-variance trade-off: with a fixed amount of data you can't push both to zero; pick the capacity with the lowest bias² + variance, which is not the one with the lowest training error.
Validation curves
Turn the knob step by step and record both errors:
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
degrees, trs, vas = [1, 2, 3, 4, 5, 6, 8, 10, 12, 15], [], []
for d in degrees: # turn the knob
m = poly(d).fit(X_tr, y_tr); trs.append(mse(y_tr, m.predict(X_tr))); vas.append(mse(y_va, m.predict(X_va)))
print(f"degree {d:2}: train {trs[-1]:6.1f} validation {vas[-1]:10.1f}")
plt.figure(figsize=(10, 4.8)); plt.yscale("log"); plt.xlabel("polynomial degree (capacity)")
plt.plot(degrees, trs, "o-", label="train"); plt.plot(degrees, vas, "s-", label="validation")
plt.legend(); plt.savefig("validation_curve.png", dpi=110, bbox_inches="tight")
Output:
degree 1: train 365.7 validation 360.2
degree 2: train 62.4 validation 194.2
degree 3: train 61.5 validation 227.1
degree 4: train 25.0 validation 65.0
degree 5: train 24.8 validation 63.7
degree 6: train 24.2 validation 95.3
degree 8: train 22.5 validation 815.0
degree 10: train 14.4 validation 47092.4
degree 12: train 9.0 validation 589485.4
degree 15: train 8.6 validation 4280828.2

- Training error only goes down as capacity grows.
- Validation error is a U: falls, bottoms out at degree 5 (63.7, right at the noise floor of 64), then climbs to millions.
- Left side = underfitting (high bias). Right side = overfitting (high variance). Pick the bottom.
A validation curve is error (or score) against a complexity knob, for train and validation. scikit-learn's validation_curve does the same with cross-validation (Episode 26), averaging several splits:
import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 60); X = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 60)
model = make_pipeline(PolynomialFeatures(), LinearRegression())
degrees = [1, 2, 4, 6, 10]
train, val = validation_curve(model, X, y, param_name="polynomialfeatures__degree",
param_range=degrees, cv=5, scoring="neg_mean_squared_error")
for d, tr, va in zip(degrees, -train.mean(axis=1), -val.mean(axis=1)):
print(f"degree {d:2}: train {tr:8.1f} validation {va:10.1f}")
Output:
degree 1: train 359.1 validation 381.0
degree 2: train 113.4 validation 131.3
degree 4: train 50.4 validation 59.8
degree 6: train 47.9 validation 60.8
degree 10: train 43.6 validation 67.0
(scikit-learn maximises scores, so errors come back negated: neg_mean_squared_error. The parameter name is <step name>__<parameter>.)
Learning curves
"Should I note down 300 more days?" Train on growing amounts of data and watch both errors:
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
def scores(d, n): # average over 30 repeats
runs = [sample(n) for _ in range(30)]; models = [poly(d).fit(X, y) for X, y in runs]
return (np.mean([mse(y, m.predict(X)) for (X, y), m in zip(runs, models)]),
np.mean([mse(y_va, m.predict(X_va)) for m in models]))
fig, ax = plt.subplots(1, 2, figsize=(13, 4.6), sharey=True); sizes = [20, 40, 80, 160, 320]
for a, d in zip(ax, [1, 8]):
tr, va = zip(*[scores(d, n) for n in [20, 40, 80, 160, 320]])
print(f"degree {d} train:", np.round(tr).astype(int), "\n validation:", np.round(va).astype(int))
a.plot(sizes, tr, "o-", label="train"); a.plot(sizes, va, "s-", label="validation")
a.set(title=f"degree {d}", xlabel="training days", yscale="log"); a.legend()
plt.savefig("learning_curves.png", dpi=110, bbox_inches="tight")
Output:
degree 1 train: [375 380 382 383 394]
validation: [397 383 362 357 355]
degree 8 train: [34 47 61 60 62]
validation: [8026 744 70 64 59]

- Degree 1 (high bias): the curves meet quickly, but high (about 360-400) and stay there from 20 to 320 days. More data is wasted; add capacity or better features.
- Degree 8 (high variance): a huge gap at first (train 34, validation 8,026), closing as data grows (320 days: 62 vs 59, at the noise floor). More data helps.
A learning curve is error against training-set size, for train and validation. scikit-learn's helper is learning_curve:
import numpy as np
from sklearn.model_selection import learning_curve
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 400); X = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 400)
for d in [1, 8]:
sizes, train, val = learning_curve(make_pipeline(PolynomialFeatures(d), LinearRegression()), X, y,
train_sizes=[20, 40, 80, 160, 320], cv=5, shuffle=True,
random_state=0, scoring="neg_mean_squared_error")
print(f"degree {d}: train", np.round(-train.mean(axis=1)).astype(int),
" validation", np.round(-val.mean(axis=1)).astype(int))
Output:
degree 1: train [289 321 346 364 374] validation [444 435 387 376 376]
degree 8: train [37 54 54 59 64] validation [186985 57817 82 69 67]
Diagnosis table
| High bias (underfitting) | High variance (overfitting) | |
|---|---|---|
| Symptom | train high, validation high, close | train low, validation much higher |
| Learning curve | curves meet early, high | big gap that shrinks with data |
| More data | won't help | helps |
| Fix | more capacity (higher degree, deeper tree, more features), better features, less regularisation | more data, less capacity, fewer features, regularisation (Episode 21), ensembles (Episodes 33-34), early stopping |
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| picking the model with the lowest training error | degree 15: train 8.6, validation 4,280,828.2 | memorised noise | pick by validation error |
| being proud of a training error below the noise | 8.6 < 64 | irreducible error is a floor | compare with the noise level / a baseline |
| collecting more data for an underfit model | degree 1 stays near 380 from 20 to 320 days | bias, not variance | add capacity or features |
| tuning the degree on the test set | an optimistic test score | the test set leaked into your choices | tune on validation (or CV), test once at the end |
| one validation split for a small dataset | the "best" degree changes with the split | validation error is itself noisy | cross-validation (Episode 26) |
| high powers of raw inputs | numbers like 40¹⁵ ≈ 10²⁴ | huge, badly conditioned features | scale inputs first (we used −1..1) |
Mini project: a model doctor
A function that fits a model on growing training sizes and prints a diagnosis:
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def doctor(make_model, sample, X_va, y_va, noise_var, sizes=(20, 80, 320), repeats=20):
rows = []
for n in sizes:
tr = va = 0
for _ in range(repeats):
X, y = sample(n); m = make_model().fit(X, y)
tr += mse(y, m.predict(X)) / repeats; va += mse(y_va, m.predict(X_va)) / repeats
rows.append((n, tr, va)); print(f" {n:3} rows: train {tr:9.1f} validation {va:9.1f}")
n, tr, va = rows[-1]
if va > 2 * noise_var and va - tr < 0.5 * va:
print(" diagnosis: HIGH BIAS - add capacity or features; more data won't help")
elif rows[0][2] > 2 * rows[-1][2]:
print(" diagnosis: HIGH VARIANCE at small sizes - more data, less capacity or regularisation")
else:
print(" diagnosis: close to the noise floor - good fit")
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n):
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_va, y_va = sample(500)
for d in [1, 4, 8]:
print(f"degree {d}:")
doctor(lambda: make_pipeline(PolynomialFeatures(d), LinearRegression()), sample, X_va, y_va, noise_var=64)
Output:
degree 1:
20 rows: train 345.4 validation 452.2
80 rows: train 382.4 validation 386.2
320 rows: train 386.6 validation 373.7
diagnosis: HIGH BIAS - add capacity or features; more data won't help
degree 4:
20 rows: train 43.6 validation 93.7
80 rows: train 58.4 validation 68.3
320 rows: train 63.6 validation 65.5
diagnosis: close to the noise floor - good fit
degree 8:
20 rows: train 39.8 validation 92125.1
80 rows: train 56.1 validation 77.0
320 rows: train 62.3 validation 66.0
diagnosis: HIGH VARIANCE at small sizes - more data, less capacity or regularisation
Practice
- Repeat the validation curve with 60 training days instead of 20. Where is the best degree now?
- Measure bias² and variance for degrees 2, 3 and 5.
- Raise the noise standard deviation to 20. What happens to the irreducible error and to the best degree?
- Draw learning curves (as numbers) for degree 4.
Solution 1
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
X_tr, y_tr = sample(60) # 60 days this time
for d in [1, 2, 3, 4, 5, 6, 8, 10, 12, 15]:
m = poly(d).fit(X_tr, y_tr)
print(f"degree {d:2}: train {mse(y_tr, m.predict(X_tr)):6.1f} validation {mse(y_va, m.predict(X_va)):9.1f}")
Output:
degree 1: train 435.0 validation 361.4
degree 2: train 140.6 validation 117.8
degree 3: train 116.0 validation 96.0
degree 4: train 59.6 validation 61.0
degree 5: train 58.5 validation 62.6
degree 6: train 53.8 validation 73.1
degree 8: train 53.6 validation 73.7
degree 10: train 53.0 validation 74.1
degree 12: train 50.7 validation 78.3
degree 15: train 47.9 validation 103.3
With three times more data, high degrees overfit far less: the right side of the U is much lower and flatter, because more data reduces variance.
Solution 2
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 50).reshape(-1, 1); t = grid.ravel() * 17.5 + 22.5
for d in [2, 3, 5]:
preds = np.array([poly(d).fit(*sample(20)).predict(grid) for _ in range(300)])
bias2 = np.mean((preds.mean(axis=0) - truth(t)) ** 2); var = np.mean(preds.var(axis=0))
print(f"degree {d}: bias² {bias2:6.1f} variance {var:7.1f} total {bias2 + var + 64:7.1f}")
Output:
degree 2: bias² 59.8 variance 29.9 total 153.7
degree 3: bias² 60.0 variance 95.0 total 219.0
degree 5: bias² 2.4 variance 256.0 total 322.4
Bias falls as the degree rises (the wiggle needs a few powers), while variance rises; degree 4 from the episode sits near the sweet spot.
Solution 3
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n, sd):
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, sd, n)
for sd in [8, 20]:
X_tr, y_tr = sample(20, sd); X_va, y_va = sample(2000, sd)
vals = {d: mse(y_va, make_pipeline(PolynomialFeatures(d), LinearRegression()).fit(X_tr, y_tr).predict(X_va))
for d in [1, 2, 3, 4, 5, 6]}
best = min(vals, key=vals.get)
print(f"noise sd {sd}: floor {sd ** 2}, validation", {d: round(v) for d, v in vals.items()}, "best degree", best)
Output:
noise sd 8: floor 64, validation {1: 407, 2: 164, 3: 181, 4: 71, 5: 71, 6: 85} best degree 5
noise sd 20: floor 400, validation {1: 874, 2: 675, 3: 1902, 4: 3657, 5: 24917, 6: 7946} best degree 2
The floor rises from 64 to 400 (sd²). With more noise, the same 20 days carry less information, so variance grows and the best degree usually drops: simpler models win when data is noisy.
Solution 4
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # made-up recipe
rng = np.random.default_rng(21)
def sample(n): # n days: temperature scaled to -1..1, cups sold (noise sd 8)
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200) # 20 days to learn, 200 new days
for n in [20, 40, 80, 160, 320]:
tr = va = 0
for _ in range(30):
X, y = sample(n); m = poly(4).fit(X, y)
tr += mse(y, m.predict(X)) / 30; va += mse(y_va, m.predict(X_va)) / 30
print(f"{n:3} days: train {tr:6.1f} validation {va:6.1f}")
Output:
20 days: train 44.1 validation 95.9
40 days: train 62.0 validation 71.3
80 days: train 59.5 validation 64.0
160 days: train 63.9 validation 60.9
320 days: train 63.1 validation 60.1
Degree 4 starts with a small gap, and both curves settle near the noise floor of 64: low bias and, with enough data, low variance.
Quick quiz
train MSE: 4.1
validation MSE: 96.3
noise variance: about 60
What is the problem, and what will help?
A) High variance: more data or less capacity B) High bias: more capacity C) Perfect fit: ship it D) Irreducible error: nothing helps
Answer
A. Training error far below the noise floor and a big gap to validation = overfitting (high variance). More data, less capacity, or regularisation.
Interview questions
- Define bias and variance. Bias: error from wrong assumptions; the gap between the average prediction (over training sets) and the truth. Variance: how much predictions change when the training set changes.
- State the bias-variance decomposition. For squared error: E[(y − ŷ)²] = bias² + variance + irreducible noise σ².
- How do you detect overfitting? Low training error with much higher validation error; a validation curve turning up; a learning curve with a large gap.
- Will more data fix underfitting? No. Learning curves converge at a high error; add capacity or features.
- Name ways to reduce variance. More data, simpler models, feature selection, regularisation, bagging/ensembles, early stopping, data augmentation.
- What is irreducible error? Noise in the target that no model can predict from the features; a floor on expected error.
- Validation curve vs learning curve? Validation curve: score vs a hyperparameter (complexity). Learning curve: score vs training-set size.
- Why not tune hyperparameters on the test set? The choice adapts to the test data, so the test score becomes optimistic; tune on validation or cross-validation and keep the test set for one final check.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Underfitting | Model too simple; high error on train and new data | degree 1: 365.7 / 360.2 |
| Overfitting | Model memorises training noise; low train, high new-data error | degree 15: 8.6 / 4,280,828.2 |
| Generalisation | Doing well on data not seen in training | degree 4 |
| Training error | Error on the data the model learned from | train MSE |
| Validation error | Error on held-back data used for choices | validation MSE |
| Gap | Validation error − training error | large = high variance |
| Noise | Random part of the target the features can't explain | sd 8 cups |
| Irreducible error | The noise variance: a floor no model can beat | 64 |
| Bias | Error of the average prediction vs the truth (too-simple assumptions) | degree 1: bias² 332.9 |
| Variance | Spread of predictions across training sets | degree 7: 57,589.8 |
| Bias-variance decomposition | Expected MSE = bias² + variance + noise | 117.1 vs 119.0 |
| Bias-variance trade-off | Lowering one usually raises the other | degree up: bias ↓, variance ↑ |
| Capacity | How complicated a shape a model can fit | degree 15 > degree 4 |
| Model complexity | The settings that control capacity | degree, depth, features |
| Hyperparameter | A setting you choose, not learned from data | degree |
| Validation curve | Train and validation error vs a complexity knob | U-shaped |
| Learning curve | Train and validation error vs training-set size | gap closes for degree 8 |
validation_curve |
scikit-learn helper, with cross-validation | param_name="polynomialfeatures__degree" |
learning_curve |
scikit-learn helper, with cross-validation | train_sizes=[20, ..., 320] |
neg_mean_squared_error |
MSE with a minus sign, so higher is better | negate to read it |
| Log scale | An axis where each step multiplies by 10 | 10, 100, 1,000 |
Summary
- Underfitting = high error everywhere (high bias); overfitting = low training error, much higher validation error (high variance).
- Expected test error = bias² + variance + irreducible noise; our measured numbers matched (117.1 vs 119.0).
- Capacity trades bias for variance; choose it at the bottom of the validation curve (degree 5, 63.7, at the noise floor).
- Learning curves tell you whether more data helps: no for high bias (curves meet high), yes for high variance (gap closes).
Next episode: regularisation: keep a high-capacity model but calm its weights with ridge, lasso and elastic net.