SyntaxPilot
Courses/AI: Zero to Agents/Lesson 6
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

AI: Zero to Agents · Lesson 6 of 40

Overfitting and underfitting

queued

These are the notes for AI episode 6. Every score in episode 5 was measured on data the model might already have seen. Here you build a model with zero error that is useless on its first new day, then invent the fix: generalization, overfitting and underfitting, the training / validation / test split, and how to avoid data leakage. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real.

The problem: Rohan memorises last year's paper

Rohan's exam season What it means
He learns every answer of last year's paper by heart Memorising, not understanding
Practice test (the same paper): 100% Perfect on questions he has seen
The real exam: new numbers, same ideas Questions he has never seen
He fails He never learned the method

Now let's build Rohan as a machine learning model.

Demo 1: a model that memorises

# Raju's notebook: (temperature °C, cups of chai sold)
notebook = [(14, 95), (16, 87), (18, 94), (22, 82),
            (24, 68), (28, 60), (30, 60), (34, 59)]

memory = dict(notebook)          # the "model" memorises every day

def predict(temp):
    return memory[temp]

errors = [abs(predict(t) - cups) for t, cups in notebook]
print("error on the notebook days:", sum(errors) / len(errors))

print("tomorrow, 17 °C ->", predict(17))

Output:

error on the notebook days: 0.0
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    print("tomorrow, 17 °C ->", predict(17))
                                ~~~~~~~^^^^
  File "main.py", line 8, in predict
    return memory[temp]
           ~~~~~~^^^^^^
KeyError: 17

Zero error on the notebook days, and no answer at all for the first new day. A model's job is not to remember the past; it is to answer new questions well. Doing well on data you have never seen is called generalization.

Patching the memoriser: copy the nearest day?

If 17 °C is missing, copy the closest notebook day: 16 °C → 87 cups. But look at the notebook: 18 °C sold 94 cups, more than 16 °C did. Hotter, yet more chai? That is noise: random ups and downs (a wedding nearby, a rainy evening) that are not part of the pattern and won't repeat. A model that copies days copies the noise too.

Demo 2: three models, two tests

train = [(14, 95), (16, 87), (18, 94), (22, 82),
         (24, 68), (28, 60), (30, 60), (34, 59)]
new_days = [(17, 89), (21, 83), (25, 78), (31, 60)]   # never seen

def average(t):                  # model 1: one number for every day
    return sum(c for _, c in train) / len(train)

def nearest(t):                  # model 2: copy the closest notebook day
    return min(train, key=lambda day: abs(day[0] - t))[1]

def line(t):                     # model 3: a straight line
    return 124 - 2.0 * t

def mae(model, days):
    return sum(abs(model(t) - c) for t, c in days) / len(days)

for model in [average, nearest, line]:
    print(f"{model.__name__:8} train error {mae(model, train):5.2f}"
          f"   new-day error {mae(model, new_days):5.2f}")

Output:

average  train error 13.88   new-day error  9.69
nearest  train error  0.00   new-day error  3.25
line     train error  4.62   new-day error  2.00

(mae is the mean absolute error from episode 4: the average gap in cups. The line 124 - 2.0 × temp is the one Raju learned in episode 4.)

Model Train error New-day error Diagnosis
nearest 0.00 3.25 Overfitting: memorised the noise
average 13.88 9.69 Underfitting: too simple to learn "hot days sell less"
line 4.62 2.00 Good fit: learned the trend, ignored the noise

Overfitting and underfitting (three students)

In the classroom In machine learning Signs
Rohan the memoriser: 100% practice, fails the exam Overfitting low training error, much higher error on new data
Guesses "C" every time: bad at both Underfitting high error everywhere
Learns the method: good at both Good fit both errors low and close together

How to read the two errors together:

train error  very low,  new-data error much higher   →  OVERFITTING  (big gap)
train error  high,      new-data error high          →  UNDERFITTING
train error  low,       new-data error low           →  GOOD FIT     (small gap)

Holding data back: training, validation and test sets

How do we measure new-data error before the real new days arrive? Do what a good teacher does:

At school In machine learning Used for
Textbook exercises Training set the model learns from it
Class test Validation set comparing models and choosing settings
Final exam, sealed until the end Test set one honest score, used once
A leaked exam paper Data leakage gives a fake, too-good score

Why not reuse the validation set as the final score? After you have chosen your model by it, it is no longer new to you, so its score is slightly optimistic.

all data ──shuffle──> [ train 16 days | validation 4 | test 4 ]
                         learn            choose         report once

Demo 3: train, validation, test

import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
        (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
        (28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(1)
random.shuffle(days)                     # mix hot and cool days first
train, val, test = days[:16], days[16:20], days[20:]
print("train:", len(train), " validation:", len(val), " test:", len(test))
def mae(model, data):
    return sum(abs(model(t) - c) for t, c in data) / len(data)
def nearest(t):
    return min(train, key=lambda d: abs(d[0] - t))[1]
def line(t):
    return 124 - 2.0 * t
for model in [nearest, line]:
    print(f"{model.__name__:8} validation error {mae(model, val):.2f}")
best = min([nearest, line], key=lambda m: mae(m, val))
# ...and open the TEST set only once, at the very end
print("chosen:", best.__name__, "  test error:", round(mae(best, test), 2))

Output:

train: 16  validation: 4  test: 4
nearest  validation error 6.50
line     validation error 2.50
chosen: line   test error: 3.75

Validation picks the line. The honest test error (3.75) is a little worse than the validation error, which is normal. This is the number you report. random.seed(1) makes the shuffle repeatable, so you and your teammates get the same split.

Demo 4: data leakage gives a fake perfect score

import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
        (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
        (28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(1)
random.shuffle(days)
test = days[20:]
train = days                  # oops: the test days are inside train too

def nearest(t):
    return min(train, key=lambda d: abs(d[0] - t))[1]

print("test error:", sum(abs(nearest(t) - c) for t, c in test) / len(test))

Output:

test error: 0.0

Perfect, and completely fake: the model just looked up the answers. If a test score looks too good to be true, check for leakage first. Leakage also happens in subtler ways: the same customer in both train and test, or a feature that secretly contains the answer (for example "refund issued" when predicting "will this order be returned?").

How to fix overfitting (and underfitting)

Fix Why it helps
More data noise averages out; it is much harder to memorise 10,000 days than 8
A simpler model a line can't memorise every bump, so it must learn the trend
Regularization during training the model pays a penalty for being too complicated
Early stopping watch validation error while training; stop when it starts rising (episode 7 shows training step by step)

For underfitting, do the opposite: a more flexible model, better features, or training longer.

Demo 5: a lucky validation set

Change only the shuffle seed in Demo 3, from random.seed(1) to random.seed(2):

import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
        (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
        (28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(2)
random.shuffle(days)                     # mix hot and cool days first
train, val, test = days[:16], days[16:20], days[20:]
print("train:", len(train), " validation:", len(val), " test:", len(test))
def mae(model, data):
    return sum(abs(model(t) - c) for t, c in data) / len(data)
def nearest(t):
    return min(train, key=lambda d: abs(d[0] - t))[1]
def line(t):
    return 124 - 2.0 * t
for model in [nearest, line]:
    print(f"{model.__name__:8} validation error {mae(model, val):.2f}")
best = min([nearest, line], key=lambda m: mae(m, val))
# ...and open the TEST set only once, at the very end
print("chosen:", best.__name__, "  test error:", round(mae(best, test), 2))

Output:

train: 16  validation: 4  test: 4
nearest  validation error 2.50
line     validation error 3.25
chosen: nearest   test error: 4.25

Now validation picks the overfitting model, and its test error (4.25) is worse than the line would have scored. Four validation days is a tiny exam: a few easy questions and a memoriser looks brilliant. One small validation set gives a noisy verdict.

Demo 6: k-fold cross-validation

Give every day a turn in the class test. Keep the test days sealed, split the other 20 days into 5 folds of 4, let each fold be the validation set once (the other 16 days train), and average.

import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
        (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
        (28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(2)
random.shuffle(days)
pool, test = days[:20], days[20:]       # the test days stay sealed
def mae(model, data):
    return sum(abs(model(t) - c) for t, c in data) / len(data)
totals = {"nearest": 0, "line": 0}
for k in range(5):                       # 5 folds: each 4 days takes a turn
    val = pool[k * 4:(k + 1) * 4]
    train = pool[:k * 4] + pool[(k + 1) * 4:]
    nearest = lambda t: min(train, key=lambda d: abs(d[0] - t))[1]
    line = lambda t: 124 - 2.0 * t
    totals["nearest"] += mae(nearest, val) / 5
    totals["line"] += mae(line, val) / 5
for name, err in totals.items():
    print(f"{name:8} average validation error {err:.2f}")

Output:

nearest  average validation error 4.45
line     average validation error 3.50

With the same unlucky seed, the line wins again. It wins for seeds 1 to 5 as well (line 3.15 / 3.50 / 2.90 / 3.25 / 3.40 vs nearest 4.65 / 4.45 / 4.70 / 3.80 / 5.00). This is k-fold cross-validation (here k = 5). It costs k training runs, but the choice no longer depends on luck. Small data? Cross-validate.

Glossary

Term Meaning
Generalization Doing well on data the model has never seen
Memorising Storing the training examples instead of learning a pattern
Noise Random ups and downs in data that are not part of the pattern
Overfitting Learning the training data too closely, noise included: low train error, high new-data error
Underfitting A model too simple to learn the pattern: high error everywhere
Good fit Low error on both training and new data
Training set The data the model learns from
Validation set Held-back data used to compare models and choose settings
Test set Held-back data used once, at the end, for an honest score
Data leakage Test information sneaking into training, giving fake good scores
Shuffle / seed Mixing rows randomly before splitting; a seed makes the mix repeatable
Regularization A training penalty for complexity, to reduce overfitting
Early stopping Stopping training when validation error starts to rise
Cross-validation Rotating which part of the data is the validation set, then averaging

Common mistakes

Mistake Code Error
A lookup instead of a model memory[17] (17 °C never seen) KeyError: 17
Test rows inside training (leakage) train = days no error, test error 0.0
Splitting sorted data last 20% of days sorted by temperature as test no error, but the test is only the hottest days
Choosing a model by test error comparing 5 models on the test set no error, an optimistic score
Judging by training error nearest has train error 0.00 no error, picks the overfitting model

Fixes: learn a pattern, not a table; split first and keep the test set apart; shuffle (with a seed) before splitting; choose on validation, report test once.

Practice

  1. Change random.seed(1) to random.seed(2) in Demo 3. Which model does validation choose now?
  2. Add the average model to Demo 3. What is its validation error?
  3. Split the sorted days without shuffling: train, val, test = days[:16], days[16:20], days[20:]. Which days are in the test set? What's the hottest training day?
  4. Explain overfitting to a friend with your own exam story.
Solutions

1. With seed 2 the validation errors are nearest 2.50 and line 3.25, so validation picks nearest. With only 4 validation days, luck matters a lot: this is why Demo 6 uses cross-validation. Validation errors for seeds 1 to 5:

seed 1: average  9.28  nearest  6.50  line  2.50
seed 2: average 16.94  nearest  2.50  line  3.25
seed 3: average 12.72  nearest  5.75  line  1.25
seed 4: average 13.25  nearest  3.50  line  4.25
seed 5: average 13.00  nearest  5.75  line  3.50

2. With seed 1, average has a validation error of 9.28: underfitting, by far the worst.

3.

days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94),
        (19, 88), (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78),
        (26, 75), (27, 72), (28, 60), (29, 57), (30, 60), (31, 60), (32, 62),
        (33, 58), (34, 59), (35, 51)]
train, val, test = days[:16], days[16:20], days[20:]
print("test days:", test)
print("hottest training day:", max(t for t, _ in train), "°C")
test days: [(32, 62), (33, 58), (34, 59), (35, 51)]
hottest training day: 27 °C

The test set holds only the four hottest days, and the model never trained above 27 °C. The test doesn't look like real new data.

4. Example: learning the answers to a cricket quiz app's 50 questions by heart, then failing when the app adds new questions.

Quick quiz 1

A photo classifier: training accuracy 99%, validation accuracy 61%. What's going on?

  • A. Underfitting
  • B. Overfitting
  • C. A good fit
  • D. Data leakage
Answer

B, overfitting. Near-perfect on training photos, much worse on new ones: the big gap is the classic sign.

Quick quiz 2

You're choosing between 5 models. Which data should you compare them on?

  • A. The training set
  • B. The validation set
  • C. The test set
  • D. All the data together
Answer

B, the validation set. Training error rewards memorising, and the test set stays sealed until the very end.

Interview questions

  • What is overfitting? How do you detect it, and name three ways to reduce it.
  • What is underfitting, and how is it fixed?
  • Why do we need a validation set as well as a test set?
  • What is data leakage? Give two examples.
  • Why shuffle before splitting, and why set a random seed?
  • What is k-fold cross-validation, and when is it useful? Why must the test set stay out of the folds?

Summary

  • The goal of a model is generalization: doing well on data it has never seen.
  • A memorising model had 0 error and crashed on the first new day (KeyError: 17).
  • Overfitting = low training error, high new-data error (memorised noise). Underfitting = high error everywhere.
  • Train to learn, validate to choose, test once at the end.
  • Data leakage gives fake perfect scores: keep the test set apart, and shuffle before splitting.
  • A tiny validation set can pick the wrong model by luck: use k-fold cross-validation.
  • Fix overfitting with more data, simpler models, regularization and early stopping.

Next episode: so far we trained models by trying every possible line. That works with two numbers, but real models have millions. How a model improves: loss, gradient descent and the learning rate.