These are the notes for AI episode 5. A classifier can be 99% accurate and completely useless. In this episode you see that happen on screen, then invent the tools that catch it: the confusion matrix, precision, recall, the F1 score, and the threshold trade-off. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real.
The problem: a dengue test at the health camp
Imagine a health camp in a small town (the numbers are an example, not real medical data):
- 1,000 people come for a quick dengue check.
- Only 10 of them really have dengue; 990 are healthy. The sick people are rare.
- A model looks at each person's symptoms and says sick or healthy. Anyone called sick goes for a proper blood test.
- If the model misses a sick person, they go home untreated. One kind of mistake is much more costly than the other.
How do we know whether the model is any good? The tool we already know is accuracy: the share of answers that are right.
Demo 1: the 99% useless model
The laziest possible model ignores the person and always says healthy.
# 1,000 people tested for dengue; 10 really have it
actual = [1] * 10 + [0] * 990 # 1 = sick, 0 = healthy
def lazy_model(person): # always says "healthy"
return 0
pred = [lazy_model(p) for p in range(1000)]
right = sum(p == a for p, a in zip(pred, actual))
caught = sum(p == 1 and a == 1 for p, a in zip(pred, actual))
print("accuracy:", right / len(actual))
print("sick people caught:", caught, "of 10")
Output:
accuracy: 0.99
sick people caught: 0 of 10
99% accuracy, and it catches nobody. When one class is rare, always guessing the common class is right almost every time. Accuracy hides the rare class, and the rare class is usually the one we care about. Data where one class is much rarer than the other is called imbalanced data.
Demo 2: a better model gets a worse accuracy
A real model catches 8 of the 10 sick people, misses 2, and raises 30 false alarms among the 990 healthy people. Instead of one number, we count four kinds of answers.
actual = [1] * 10 + [0] * 990
# A real model: catches 8 sick people, misses 2,
# and raises 30 false alarms among the healthy
pred = [1] * 8 + [0] * 2 + [1] * 30 + [0] * 960
pairs = list(zip(pred, actual))
tp = pairs.count((1, 1)) # said sick, really sick
fn = pairs.count((0, 1)) # said healthy, really sick
fp = pairs.count((1, 0)) # said sick, really healthy
tn = pairs.count((0, 0)) # said healthy, really healthy
print("TP:", tp, " FN:", fn)
print("FP:", fp, " TN:", tn)
print("accuracy:", (tp + tn) / len(actual))
Output:
TP: 8 FN: 2
FP: 30 TN: 960
accuracy: 0.968
The model that actually saves eight people scores lower accuracy (96.8%) than the lazy model (99%). One number can't tell this story. The four counts can.
The confusion matrix (the metal detector)
Think of the metal detector at a railway station:
| At the station | In machine learning | Meaning |
|---|---|---|
| Beeps, and the person has a knife | True positive (TP) | said yes, really yes |
| Beeps, but it's only keys | False positive (FP) | said yes, really no (a false alarm) |
| Silent, but the person has a knife | False negative (FN) | said no, really yes (a miss) |
| Silent, and the person has nothing | True negative (TN) | said no, really no |
Positive means the model said yes (here: sick). True/false says whether that answer was right. Arrange the four counts in a 2×2 grid and you get the confusion matrix: it shows exactly where the model gets confused.
Our real model's matrix:
| Model said sick | Model said healthy | |
|---|---|---|
| Really sick (10) | TP = 8 | FN = 2 |
| Really healthy (990) | FP = 30 | TN = 960 |
- Accuracy = (TP + TN) / everyone = (8 + 960) / 1000 = 0.968.
- The lazy model's matrix has TP = 0, FN = 10: the matrix exposes it at once.
person ──> model ──┬── says SICK ────┬── really sick → TP (right alarm)
│ └── really healthy → FP (false alarm)
└── says HEALTHY ─┬── really sick → FN (missed!)
└── really healthy → TN (right silence)
Precision and recall: two sharp questions
| Metric | The question | Formula | Matters most when |
|---|---|---|---|
| Precision | When the model says yes, how often is it right? | TP / (TP + FP) | false alarms are costly (spam filter: a job offer lost in spam) |
| Recall | Of all the real yes cases, how many did it catch? | TP / (TP + FN) | misses are costly (disease screening, card fraud) |
Recall is also called sensitivity or the true positive rate.
Demo 3: precision and recall (and a crash)
def counts(pred, actual):
pairs = list(zip(pred, actual))
return pairs.count((1, 1)), pairs.count((1, 0)), pairs.count((0, 1))
def precision(pred, actual): # when it says "sick", how often right?
tp, fp, fn = counts(pred, actual)
return tp / (tp + fp)
def recall(pred, actual): # of all sick people, how many caught?
tp, fp, fn = counts(pred, actual)
return tp / (tp + fn)
actual = [1] * 10 + [0] * 990
real = [1] * 8 + [0] * 2 + [1] * 30 + [0] * 960
lazy = [0] * 1000
print("real model: precision", round(precision(real, actual), 2),
" recall", recall(real, actual))
print("lazy model: recall", recall(lazy, actual))
print("lazy model: precision", precision(lazy, actual))
Output:
real model: precision 0.21 recall 0.8
lazy model: recall 0.0
Traceback (most recent call last):
File "main.py", line 19, in <module>
print("lazy model: precision", precision(lazy, actual))
~~~~~~~~~^^^^^^^^^^^^^^
File "main.py", line 7, in precision
return tp / (tp + fp)
~~~^~~~~~~~~~~
ZeroDivisionError: division by zero
- The real model catches 8 in 10 sick people (recall 0.8), but only about 1 in 5 of its alarms is real (precision 8/38 = 0.21).
- The lazy model's recall is 0: exposed.
- The lazy model never said sick, so TP + FP = 0 and precision divides by zero. Precision asks about the model's yes answers, and there are none. The usual convention: if nothing was flagged, report precision as 0 (scikit-learn does the same and warns you).
Demo 4: averaging fails, F1 fixes
A new trick model, the alarm, calls everyone sick. Its recall is perfect. We need one number that combines precision and recall.
def scores(pred, actual):
pairs = list(zip(pred, actual))
tp, fp, fn = pairs.count((1, 1)), pairs.count((1, 0)), pairs.count((0, 1))
p = tp / (tp + fp) if tp + fp else 0.0 # nothing flagged -> 0
r = tp / (tp + fn) if tp + fn else 0.0
return p, r
actual = [1] * 10 + [0] * 990
models = {"lazy (all healthy)": [0] * 1000,
"alarm (all sick)": [1] * 1000,
"real model": [1] * 8 + [0] * 2 + [1] * 30 + [0] * 960}
for name, pred in models.items():
p, r = scores(pred, actual)
avg = (p + r) / 2
f1 = 2 * p * r / (p + r) if p + r else 0.0
print(f"{name:20} P={p:.2f} R={r:.2f} avg={avg:.2f} F1={f1:.2f}")
Output:
lazy (all healthy) P=0.00 R=0.00 avg=0.00 F1=0.00
alarm (all sick) P=0.01 R=1.00 avg=0.51 F1=0.02
real model P=0.21 R=0.80 avg=0.51 F1=0.33
- The plain average gives the alarm model (which sends all 1,000 people for a blood test) 0.51, exactly the same as the real model. It is fooled.
- F1 gives the alarm 0.02 and the real model 0.33.
Why the harmonic mean?
F1 = 2 × precision × recall / (precision + recall)
F1 is the harmonic mean of precision and recall. Like a chain that is only as strong as its weakest link, it is pulled towards the weaker of the two numbers. You can't cheat it by maxing out just one. Use F1 when you care about both kinds of mistakes and the positive class is rare, and always show the confusion matrix next to it.
Demo 5: the threshold trade-off
From episode 4: many classifiers output a score between 0 and 1, and a threshold turns it into a class.
# The model's score for 10 people: how likely is dengue?
scores = [0.95, 0.85, 0.70, 0.62, 0.55, 0.40, 0.35, 0.20, 0.10, 0.05]
actual = [1, 1, 0, 1, 0, 1, 0, 0, 0, 0]
for threshold in [0.8, 0.5, 0.3]:
pred = [1 if s >= threshold else 0 for s in scores]
pairs = list(zip(pred, actual))
tp, fp, fn = pairs.count((1, 1)), pairs.count((1, 0)), pairs.count((0, 1))
print(f"threshold {threshold}: precision {tp / (tp + fp):.2f}",
f" recall {tp / (tp + fn):.2f}")
Output:
threshold 0.8: precision 1.00 recall 0.50
threshold 0.5: precision 0.60 recall 0.75
threshold 0.3: precision 0.57 recall 1.00
Raising the threshold buys precision and costs recall; lowering it does the opposite. The model stays the same: you choose the threshold.
Which mistake costs more?
| False alarm is worse → watch precision, threshold high | A miss is worse → watch recall, threshold low |
|---|---|
| Spam filter: a job offer lost in spam | Disease screening: a patient sent home |
| Shop recommendations: annoying wrong picks | Card fraud: the customer's money is gone |
When both matter and the yes class is rare, report F1 together with the confusion matrix.
Mini project: a metrics report
Write one function that prints the confusion matrix and every metric from this episode.
def report(pred, actual):
pairs = list(zip(pred, actual))
tp, fp = pairs.count((1, 1)), pairs.count((1, 0))
fn, tn = pairs.count((0, 1)), pairs.count((0, 0))
p = tp / (tp + fp) if tp + fp else 0.0
r = tp / (tp + fn) if tp + fn else 0.0
f1 = 2 * p * r / (p + r) if p + r else 0.0
print(f" said sick said healthy")
print(f"really sick {tp:5} {fn:5}")
print(f"really well {fp:5} {tn:5}")
print(f"accuracy {(tp + tn) / len(actual):.3f} precision {p:.2f}"
f" recall {r:.2f} F1 {f1:.2f}")
actual = [1] * 10 + [0] * 990
report([1] * 8 + [0] * 2 + [1] * 30 + [0] * 960, actual)
Output:
said sick said healthy
really sick 8 2
really well 30 960
accuracy 0.968 precision 0.21 recall 0.80 F1 0.33
Try it on the lazy and alarm models too.
Glossary
| Term | Meaning |
|---|---|
| Accuracy | Share of all predictions that are right: (TP + TN) / everyone |
| Imbalanced data | Data where one class is much rarer than the other (10 sick vs 990 healthy) |
| Positive / negative | The model's answer: yes (the class we are looking for) or no |
| True positive (TP) | Said yes, really yes (a right alarm) |
| False positive (FP) | Said yes, really no (a false alarm) |
| False negative (FN) | Said no, really yes (a miss) |
| True negative (TN) | Said no, really no (a right silence) |
| Confusion matrix | The 2×2 table of TP, FN, FP, TN |
| Precision | TP / (TP + FP): when the model says yes, how often it is right |
| Recall (sensitivity) | TP / (TP + FN): of all real yes cases, how many were caught |
| F1 score | 2·P·R / (P + R): the harmonic mean of precision and recall |
| Harmonic mean | An average pulled towards the smaller number |
| Score | A number (often 0 to 1) a classifier outputs before choosing a class |
| Threshold | The cut-off that turns a score into a class |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Precision when the model never says yes | tp / (tp + fp) with TP = FP = 0 |
ZeroDivisionError: division by zero |
| Trusting accuracy on imbalanced data | lazy model, 10 sick vs 990 healthy | no error, just 0.99 |
| Averaging precision and recall | (p + r) / 2 |
no error, alarm model gets 0.51 |
| Mixing up FP and FN | "FN = false alarm" | no error, wrong conclusions: FP is the false alarm, FN is the miss |
Fixes: return 0.0 when nothing was flagged; look at the confusion matrix, recall and F1 for rare classes; use F1, not a plain average; remember positive/negative is what the model said.
Practice
- A spam filter's confusion matrix is TP = 40, FN = 10, FP = 10, TN = 940. Compute its recall and F1.
- Make a model that flags only one person, who really is sick. What are its precision, recall and F1?
- Add threshold
0.6to Demo 5. What are precision and recall? - Name one app where precision matters most and one where recall does.
Solutions
1. Recall = 40 / (40 + 10) = 0.8. Precision = 40 / 50 = 0.8, so F1 = 2 × 0.8 × 0.8 / 1.6 = 0.8 (Python prints 0.8000000000000002 because of floating-point rounding).
2.
actual = [1] * 10 + [0] * 990
pred = [1] + [0] * 999 # flags only person 0, who is sick
pairs = list(zip(pred, actual))
tp, fp, fn = pairs.count((1, 1)), pairs.count((1, 0)), pairs.count((0, 1))
p, r = tp / (tp + fp), tp / (tp + fn)
print(f"P={p:.2f} R={r:.2f} F1={2 * p * r / (p + r):.2f}")
P=1.00 R=0.10 F1=0.18
Perfect precision, terrible recall, and F1 stays low.
3. With for threshold in [0.8, 0.6, 0.5, 0.3]: the new line is threshold 0.6: precision 0.75 recall 0.75 (the top four scores are flagged: 3 sick, 1 healthy).
4. Precision: a spam filter, a "this video is not for kids" flag. Recall: cancer screening, card-fraud alerts, a smoke alarm.
Quick quiz 1
A bank's model flags card payments as fraud. A missed fraud means the customer loses money; a false alarm means one SMS: "Was this you?" Which metric matters most?
- A. Accuracy
- B. Precision
- C. Recall
- D. Training time
Answer
C, recall. A missed fraud (a false negative) is the expensive mistake; a false alarm only costs an SMS. Catch as many frauds as possible.
Quick quiz 2
A spam filter: TP = 40, FN = 10, FP = 10, TN = 940. What is the precision?
- A. 0.80
- B. 0.96
- C. 0.98
- D. 0.50
Answer
A, 0.80. Precision = TP / (TP + FP) = 40 / 50. Accuracy would be 980 / 1000 = 0.98: higher, but less informative.
Interview questions
- Why can accuracy be misleading on imbalanced data? Give an example.
- Draw a confusion matrix and define TP, FP, FN and TN.
- Define precision and recall. When would you prioritise each one?
- Why is F1 a harmonic mean and not an arithmetic mean?
- How does changing the decision threshold affect precision and recall?
- A model never predicts the positive class. What are its precision and recall, and how should your code handle it?
Summary
- Accuracy lies when one class is rare: the lazy model scored 99% and caught nobody.
- The confusion matrix counts TP, FP, FN and TN, and shows exactly where a model is confused.
- Precision = TP / (TP + FP): can I trust a yes? Recall = TP / (TP + FN): did we catch them all?
- If the model never says yes, precision is defined as 0, not a crash.
- F1 = the harmonic mean of precision and recall: no cheating by maxing out one of them.
- The threshold trades precision for recall. Pick the metric by the cost of each mistake.
Next episode: every score today was measured on data the model might have already seen. A student who memorises last year's paper scores 100, then fails the real exam. Overfitting and underfitting: training, validation and test sets.