SyntaxPilot
Courses/Machine Learning/Lesson 24

Machine Learning · Lesson 24 of 90 · 12 min

Classification metrics 1

confusion matrixtrue positivefalse positivetrue negativefalse negativeaccuracyaccuracy paradoxprecisionrecall (sensitivity, true positive rate)specificityfalse positive rateF1 scoreF-betatype I and type II errors

Machine Learning Full Course · Episode 24. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. All data (card payments, café orders) is made up for teaching.

The big idea: accuracy treats every mistake the same, so when one class is rare a useless model can score 98%. Count each kind of right and wrong separately in a confusion matrix (TP, FP, FN, TN). From it come precision (of what we flagged, how much was right), recall (of what was really there, how much we found), specificity and the false positive rate. Combine precision and recall with the F1 score (a harmonic mean) or F-beta when one matters more, and average over many classes with macro, micro or weighted averaging.

The problem: catching card fraud

A bank wants to flag fraud (payments made with a stolen card). We have 20,000 made-up payments with 3 features each; 1 = fraud, 0 = genuine. Fewer than 2 in 100 are fraud: imbalanced data (Episode 23).

Accuracy = correct predictions / all predictions. Someone says "our model is 98% accurate". Let's compare it with the laziest model possible: DummyClassifier(strategy="most_frequent") ignores the features and always predicts the most common class (a baseline, Episode 17).

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.dummy import DummyClassifier
print(f"fraud: {y_te.sum()} of {len(y_te)} test payments ({y_te.mean():.1%})")
lazy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)   # always says "not fraud"
print(f"lazy model:  accuracy {lazy.score(X_te, y_te):.3f}  frauds caught {lazy.predict(X_te)[y_te == 1].sum()}")
m = LogisticRegression().fit(X_tr, y_tr)
print(f"real model:  accuracy {m.score(X_te, y_te):.3f}  frauds caught {m.predict(X_te)[y_te == 1].sum()}")

Output:

fraud: 184 of 10000 test payments (1.8%)
lazy model:  accuracy 0.982  frauds caught 0
real model:  accuracy 0.983  frauds caught 29

The accuracy paradox

The lazy model scores 98.2% and catches zero frauds. The real logistic regression scores 98.3% and catches 29. Accuracy can't tell them apart.

  • With 1.8% fraud, "always genuine" is right 98.2% of the time without learning anything.
  • A model with high accuracy and no value is the accuracy paradox. It appears whenever one class is rare.
  • Cause: accuracy counts every mistake as equal, but missing a fraud and blocking a genuine payment are very different mistakes.

The confusion matrix

Real-life example: a kitchen smoke alarm. The thing we look for (fire, fraud) is the positive class.

alarm silent (predicted 0) alarm rings (predicted 1)
no fire (actual 0) true negative (TN): a normal day false positive (FP): burnt toast, a false alarm
fire (actual 1) false negative (FN): a missed fire, the dangerous one true positive (TP): it did its job

"True/false" says whether the prediction was right; "positive/negative" says what it predicted. This 2×2 table is the confusion matrix: it shows which classes the model confuses. In scikit-learn, rows are the truth and columns are the prediction, and .ravel() returns the cells in the order tn, fp, fn, tp.

Older statistical names:

  • Type I error = false positive (crying wolf when there is none).
  • Type II error = false negative (missing the wolf).
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import confusion_matrix, precision_score, recall_score
m = LogisticRegression().fit(X_tr, y_tr); pred = m.predict(X_te)   # threshold 0.5
print(confusion_matrix(y_te, pred))                 # rows = truth, columns = prediction
tn, fp, fn, tp = confusion_matrix(y_te, pred).ravel()
print(f"TP {tp}  FP {fp}  FN {fn}  TN {tn}")
print(f"precision   {tp / (tp + fp):.3f}  (sklearn {precision_score(y_te, pred):.3f})")
print(f"recall      {tp / (tp + fn):.3f}  (sklearn {recall_score(y_te, pred):.3f})")
print(f"specificity {tn / (tn + fp):.3f}   false positive rate {fp / (fp + tn):.3f}")

Output:

[[9803   13]
 [ 155   29]]
TP 29  FP 13  FN 155  TN 9803
precision   0.690  (sklearn 0.690)
recall      0.158  (sklearn 0.158)
specificity 0.999   false positive rate 0.001

At the default threshold 0.5: 9,803 genuine payments correctly let through, 13 false alarms, 155 frauds missed and only 29 caught. The matrix shows at once what accuracy hid.

Precision, recall, specificity and false positive rate

Real-life example: fishing with a net. Precision: of everything in my net, how much is fish (not old shoes)? Recall: of all the fish in the lake, how many did I catch? A bigger net catches more fish and more shoes.

Metric Formula Question it answers Ours (threshold 0.5)
Precision TP / (TP + FP) When the model says fraud, how often is it right? 0.690
Recall (sensitivity, true positive rate, TPR) TP / (TP + FN) Of all frauds, how many did we catch? 0.158
Specificity (true negative rate) TN / (TN + FP) Of all genuine payments, how many did we leave alone? 0.999
False positive rate (FPR) FP / (FP + TN) = 1 − specificity Of all genuine payments, how many did we wrongly flag? 0.001

Our hand formulas match precision_score and recall_score. This model is very cautious: precise and almost never bothers genuine customers, but it catches under 16% of fraud.

Which matters? It depends on the cost of each mistake: a cancer screening cares most about recall (a miss is terrible); a spam filter that deletes mail cares most about precision (a false alarm loses real mail).

The precision-recall trade-off

Lowering the threshold (Episode 22) flags more payments: recall goes up, precision goes down. At threshold 0.1 (next program): precision 0.265, recall 0.630. You usually can't raise both for free.

F1: one number for both

To compare models with one number, the obvious idea is the plain average (P + R) / 2. Test it against a silly model that flags every payment:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
models = {"flag all": np.ones_like(y_te), "thresh 0.5": p >= 0.5,   # "flag all" says fraud every time
          "thresh 0.1": p >= 0.1, "thresh 0.03": p >= 0.03}
for name, pred in models.items():
    P, R = precision_score(y_te, pred), recall_score(y_te, pred)
    print(f"{name:>11}  P {P:.3f} R {R:.3f}  mean {(P + R) / 2:.3f}  F1 {f1_score(y_te, pred):.3f}"
          f" F2 {fbeta_score(y_te, pred, beta=2):.3f}")

Output:

   flag all  P 0.018 R 1.000  mean 0.509  F1 0.036 F2 0.086
 thresh 0.5  P 0.690 R 0.158  mean 0.424  F1 0.257 F2 0.186
 thresh 0.1  P 0.265 R 0.630  mean 0.448  F1 0.374 F2 0.494
thresh 0.03  P 0.123 R 0.810  mean 0.466  F1 0.214 F2 0.383

The silly model has recall 1.000 and precision 0.018 (98% of its flags are innocent customers), yet the plain mean gives it 0.509, the best of all four. The plain average picked the silliest model.

F1 = 2 · P · R / (P + R)          the harmonic mean of precision and recall
  • The harmonic mean is pulled towards the smaller number: P 0.018, R 1.0 → mean 0.509 but F1 0.036.
  • Like a chain that is only as strong as its weakest link, F1 is high only when both precision and recall are high.
  • Among the four, threshold 0.1 has the best F1 (0.374).

F-beta: when one matters more

Fβ = (1 + β²) · P · R / (β² · P + R)
  • β says how many times more important recall is than precision.
  • β = 2 (F2): recall-heavy, for fraud, disease, faulty products. Our F2 column also picks threshold 0.1 (0.494).
  • β = 0.5 (F0.5): precision-heavy, for spam filters or blocking cards (practice 3).
  • β = 1: F1, both equally important.

Averaging over many classes: macro, micro, weighted

With K classes, each class gets its own precision, recall and F1 (that class as "positive", all others as "negative"). How to combine them?

Real-life example: a school with three sections of very different sizes.

Average School version Classifier version Use when
Macro average of the three section averages plain mean of the per-class scores every class matters equally (rare ones too)
Micro pool all students, then average add up all TP, FP, FN over classes, then compute every prediction matters equally
Weighted section averages weighted by section size per-class scores weighted by support in between

Support = the number of real examples of a class.

A café like Episode 23's, with made-up features and a rare drink, badam milk (60 of 1,000 test orders):

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
rng = np.random.default_rng(4); n = 3000             # made-up café orders: 0 chai, 1 coffee, 2 badam milk
X = rng.normal(0, 1, (n, 2))
Z = np.c_[np.full(n, 2.5), 1.2 * X[:, 0] + 0.8, 1.0 * X[:, 1] - 0.6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])
pred = LogisticRegression().fit(X[:2000], y[:2000]).predict(X[2000:]); yt = y[2000:]
print("orders per class:", np.bincount(yt))
print("F1 per class:   ", f1_score(yt, pred, average=None).round(3))
for avg in ["macro", "micro", "weighted"]:
    print(f"{avg:>8} F1 {f1_score(yt, pred, average=avg):.3f}")
print(f"accuracy {np.mean(pred == yt):.3f}")

Output:

orders per class: [740 200  60]
F1 per class:    [0.852 0.383 0.033]
   macro F1 0.423
   micro F1 0.756
weighted F1 0.709
accuracy 0.756
  • F1 per class: chai 0.852, coffee 0.383, badam milk 0.033: the model almost never gets badam milk right.
  • Micro F1 = 0.756 = accuracy. For single-label multiclass problems, micro-averaged precision, recall and F1 all equal accuracy, so micro hides the rare class too.
  • Weighted F1 0.709: still dominated by chai.
  • Macro F1 0.423 exposes the failure. If every class matters, report macro.

Which metric when?

Situation Watch Compare models with
misses are costly (fraud, disease, faulty cookers) recall (sensitivity) F2, recall at a fixed precision
false alarms are costly (spam deleting mail, blocking genuine cards) precision, specificity / FPR F0.5
both matter roughly equally precision and recall F1
many classes, all important per-class scores macro F1
balanced classes, equal costs accuracy is fine accuracy

Always look at the confusion matrix, never only one number.

Common mistakes

Mistake Real symptom Why Fix
reporting accuracy on a rare class lazy model: accuracy 0.982, 0 frauds caught accuracy paradox confusion matrix, recall, precision
plain average of precision and recall "flag everything": mean 0.509, the best the arithmetic mean hides a near-zero F1 (harmonic mean)
reading the matrix the wrong way round FP and FN swapped scikit-learn: rows = truth, columns = prediction tn, fp, fn, tp = confusion_matrix(y, pred).ravel()
micro F1 when the rare class matters micro F1 0.756 = accuracy, badam milk F1 0.033 big classes dominate macro F1, per-class report
wrong positive class precision of the class you didn't mean pos_label defaults to 1 code the class you hunt for as 1, or pass pos_label
precision when nothing is flagged UndefinedMetricWarning and precision 0.0 TP + FP = 0, division by zero zero_division=0 (and question the model)
comparing models at different thresholds "model A has better recall" recall depends on the threshold compare at equal thresholds or over all thresholds (Episode 25)

Mini project: choose the threshold from the cost

Suppose (made-up numbers) a missed fraud costs 50 units and a false alarm costs 1 unit. Total cost = 50 · FN + 1 · FP:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import confusion_matrix
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
MISS, ALARM = 50, 1              # made-up costs: a missed fraud costs 50 units, a false alarm 1 unit
for t in [0.5, 0.3, 0.2, 0.1, 0.05, 0.03]:
    tn, fp, fn, tp = confusion_matrix(y_te, p >= t).ravel()
    print(f"threshold {t:<4}  FN {fn:3}  FP {fp:4}  cost {MISS * fn + ALARM * fp:5}")

Output:

threshold 0.5   FN 155  FP   13  cost  7763
threshold 0.3   FN 126  FP   49  cost  6349
threshold 0.2   FN 101  FP  114  cost  5164
threshold 0.1   FN  68  FP  321  cost  3721
threshold 0.05  FN  47  FP  678  cost  3028
threshold 0.03  FN  35  FP 1062  cost  2812

The cheapest of these thresholds is 0.03, far below 0.5, because a miss is 50 times worse than a false alarm. Episode 25 does this properly with curves over every threshold.

Practice

  1. Print the confusion matrix, recall and precision of the lazy model.
  2. Print F1 for thresholds from 0.05 to 0.5. Which threshold is best?
  3. Compute F0.5 for the four models of the F1 program. Which one does it prefer, and why?
  4. Run classification_report on the café data and read the badam milk row.
Solution 1
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.dummy import DummyClassifier
from sklearn.metrics import confusion_matrix, precision_score, recall_score
pred = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr).predict(X_te)
print(confusion_matrix(y_te, pred))
print("recall", recall_score(y_te, pred), " precision", precision_score(y_te, pred, zero_division=0))

Output:

[[9816    0]
 [ 184    0]]
recall 0.0  precision 0.0

All 184 frauds are false negatives, and nothing is flagged, so precision is 0 / 0: undefined. Without zero_division=0, scikit-learn prints an UndefinedMetricWarning and returns 0.0.

Solution 2
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import f1_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
for t in [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5]:
    print(f"threshold {t:<4}  F1 {f1_score(y_te, p >= t):.3f}")

Output:

threshold 0.05  F1 0.274
threshold 0.1   F1 0.374
threshold 0.15  F1 0.429
threshold 0.2   F1 0.436
threshold 0.3   F1 0.399
threshold 0.4   F1 0.344
threshold 0.5   F1 0.257

Threshold 0.2 gives the best F1 (0.436), better than the 0.1 we used in the episode: F1 peaks where precision and recall are balanced.

Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import fbeta_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
models = {"flag everything": np.ones_like(y_te), "threshold 0.5": p >= 0.5,
          "threshold 0.1": p >= 0.1, "threshold 0.03": p >= 0.03}
for name, pred in models.items():
    print(f"{name:>15}: F0.5 {fbeta_score(y_te, pred, beta=0.5):.3f}")

Output:

flag everything: F0.5 0.023
  threshold 0.5: F0.5 0.412
  threshold 0.1: F0.5 0.300
 threshold 0.03: F0.5 0.148

F0.5 weights precision more, so it prefers the cautious threshold 0.5 (0.412), the opposite of F2's choice.

Solution 4
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
rng = np.random.default_rng(4); n = 3000             # made-up café orders: 0 chai, 1 coffee, 2 badam milk
X = rng.normal(0, 1, (n, 2))
Z = np.c_[np.full(n, 2.5), 1.2 * X[:, 0] + 0.8, 1.0 * X[:, 1] - 0.6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])
pred = LogisticRegression().fit(X[:2000], y[:2000]).predict(X[2000:]); yt = y[2000:]
print("orders per class:", np.bincount(yt))
print("F1 per class:   ", f1_score(yt, pred, average=None).round(3))
for avg in ["macro", "micro", "weighted"]:
    print(f"{avg:>8} F1 {f1_score(yt, pred, average=avg):.3f}")
print(f"accuracy {np.mean(pred == yt):.3f}")
from sklearn.metrics import classification_report
print(classification_report(yt, pred, target_names=["chai", "coffee", "badam milk"], digits=3))

Output:

orders per class: [740 200  60]
F1 per class:    [0.852 0.383 0.033]
   macro F1 0.423
   micro F1 0.756
weighted F1 0.709
accuracy 0.756
              precision    recall  f1-score   support

        chai      0.778     0.942     0.852       740
      coffee      0.563     0.290     0.383       200
  badam milk      1.000     0.017     0.033        60

    accuracy                          0.756      1000
   macro avg      0.780     0.416     0.423      1000
weighted avg      0.748     0.756     0.709      1000

Badam milk: precision 1.000 but recall 0.017. The model predicted badam milk only once (correctly) and missed the other 59. "macro avg" and "weighted avg" are the averages from the episode; the "accuracy" row is the micro average.

Quick quiz

a disease screening test, 1,000 people:
TP = 40   FN = 10
FP = 60   TN = 890
recall = ?

A) 0.80 B) 0.40 C) 0.93 D) 0.94

Answer

A) 0.80. Recall = TP / (TP + FN) = 40 / 50. B is the precision (40 / 100) and C the accuracy (930 / 1,000). Specificity is 890 / 950 = 0.937.

Interview questions

  1. What is the accuracy paradox? On imbalanced data a trivial majority-class model reaches high accuracy with no value (98.2% while catching no fraud).
  2. Define precision and recall. Precision = TP / (TP + FP): how many flagged are truly positive. Recall = TP / (TP + FN): how many positives were found.
  3. What are type I and type II errors? Type I = false positive; type II = false negative.
  4. Sensitivity vs specificity? Sensitivity = recall = TPR; specificity = TN / (TN + FP) = 1 − FPR.
  5. Why is F1 a harmonic mean? So that it is high only when both precision and recall are high; it heavily penalises a model that sacrifices one for the other.
  6. When would you use F2 or F0.5? F2 when misses (false negatives) are costlier; F0.5 when false alarms (false positives) are costlier.
  7. Macro vs micro vs weighted averaging? Macro: unweighted mean over classes (rare classes count equally). Micro: pool TP/FP/FN over classes (equals accuracy for single-label multiclass). Weighted: mean weighted by support.
  8. How do you choose a threshold? From the costs of each error type, or by maximising a chosen metric on validation data (never on the test set).
  9. What is the layout of scikit-learn's confusion matrix? Rows = true labels, columns = predicted labels, classes in sorted order; for binary, ravel() gives tn, fp, fn, tp.

Glossary

Term Meaning Example
Accuracy Correct predictions / all predictions 0.983
Accuracy paradox High accuracy, no value, because one class is rare lazy model: 98.2%, 0 frauds caught
Baseline / DummyClassifier A trivial model to beat (Episode 17) strategy="most_frequent"
Positive class The class we are looking for, coded 1 fraud
True positive (TP) Actual 1, predicted 1 fraud flagged (29)
False positive (FP) Actual 0, predicted 1: a false alarm genuine flagged (13)
False negative (FN) Actual 1, predicted 0: a miss fraud let through (155)
True negative (TN) Actual 0, predicted 0 genuine let through (9,803)
Confusion matrix Table of counts: rows = truth, columns = prediction [[9803 13] [155 29]]
Type I error A false positive burnt-toast alarm
Type II error A false negative silent alarm in a fire
Precision TP / (TP + FP) 0.690
Recall / sensitivity / true positive rate (TPR) TP / (TP + FN) 0.158
Specificity (true negative rate) TN / (TN + FP) 0.999
False positive rate (FPR) FP / (FP + TN) = 1 − specificity 0.001
Precision-recall trade-off Raising one usually lowers the other threshold 0.5 → 0.1
Harmonic mean 2ab / (a + b), pulled towards the smaller value 0.018 and 1.0 → 0.036
F1 score Harmonic mean of precision and recall 0.374 at threshold 0.1
F-beta (Fβ) Weighted harmonic mean; β > 1 favours recall F2 0.494, F0.5 0.412
Support Number of true examples of a class badam milk: 60
Macro average Plain mean of per-class scores macro F1 0.423
Micro average Pool all TP/FP/FN, then compute micro F1 0.756 = accuracy
Weighted average Per-class scores weighted by support weighted F1 0.709
classification_report scikit-learn table of per-class precision, recall, F1, support practice 4
zero_division What to return when a metric divides by 0 precision_score(..., zero_division=0)
pos_label Which label counts as positive default 1

Summary

  • Accuracy paradox: with 1.8% fraud, "never fraud" scores 98.2% and catches nothing.
  • The confusion matrix splits predictions into TP, FP, FN, TN; FP = type I, FN = type II error.
  • Precision 0.690 (flags are mostly right) but recall 0.158 (most fraud missed); specificity 0.999, FPR 0.001.
  • The plain average of P and R crowned a silly model; F1 (harmonic mean) doesn't; F2 and F0.5 weight recall or precision.
  • Macro averaging makes rare classes count (0.423); micro equals accuracy (0.756); weighted sits in between.

Next episode: classification metrics 2: threshold tuning, ROC curves and AUC, precision-recall curves, calibration, and choosing a metric from business cost.