Machine Learning Full Course · Episode 41. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1 and imbalanced-learn 0.14.2; the output shown is the real output. Kiran's 10,000 refund claims and the rupee costs are made up for teaching.
The big idea: when one class is rare (fraud, a rare disease, a machine breakdown), a model trained the normal way learns to ignore it, and accuracy hides the failure: our logistic regression scored 0.981 accuracy while catching only 22 of 75 fake claims. You can fix the data (random undersampling of the majority class, random oversampling of the minority class, or SMOTE, which makes new minority rows between real ones), fix the training (class weights, a form of cost-sensitive learning), or fix the decision (threshold moving). Whatever you do, judge by the real costs or by recall, precision and PR-AUC, resample only the training rows, and do it inside the pipeline so cross-validation stays honest.
Sources for the facts in this episode: the docstrings of the installed imbalanced-learn 0.14.2 (SMOTE implements the Synthetic Minority Over-sampling Technique of N. V. Chawla, K. W. Bowyer, L. O. Hall and W. P. Kegelmeyer, Journal of Artificial Intelligence Research, 2002, with k_neighbors=5 by default; RandomOverSampler picks minority samples at random with replacement; RandomUnderSampler picks majority samples at random), the SMOTE paper page, and the docstring of scikit-learn 1.9.1's compute_class_weight ("balanced" = n_samples / (n_classes * np.bincount(y))).
Words you need first
- Classification, class: predicting a category such as fake / genuine (Episode 1).
- Logistic regression, predicted probability, threshold: Episodes 22-23.
- Confusion matrix, true/false positive/negative, accuracy paradox, precision, recall: Episode 24.
- Precision-recall curve, average precision (PR-AUC), calibration: Episode 25.
- Cross-validation, stratified k-fold: Episode 26. Pipeline: Episode 17. Decision tree: Episodes 29-30.
- Loss function: the number training tries to make small (Episode 8).
The problem: fake refund claims
Kiran's app refunds customers who complain. Of 10,000 made-up claims, about 2 in 100 are fake (the food did arrive). Each claim has the account's age in days, the number of claims in the last 30 days, the order amount, and whether a photo of the problem was sent. Goal: flag likely fakes so a person can check them. 7,000 claims for training, 3,000 for testing.
To compare methods fairly we count mistakes in (made-up) rupees: a missed fake loses ₹1,500 of refund, and every flagged claim needs a person to check it, about ₹50 of their time. So cost = missed × 1,500 + flagged × 50. The report helper in every program prints accuracy, fakes caught, false alarms (genuine claims flagged) and this cost.
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from sklearn.dummy import DummyClassifier
print("train:", np.bincount(y[tr]), " test:", np.bincount(y[te]), " fake share:", y.mean())
report("always genuine", DummyClassifier().fit(X[tr], y[tr]).predict(X[te]))
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X[tr], y[tr])
report("logistic", model.predict(X[te]))
Output:
train: [6884 116] test: [2925 75] fake share: 0.0191
always genuine accuracy 0.975 caught 0/75 false alarms 0 Rs 112,500
logistic accuracy 0.981 caught 22/75 false alarms 5 Rs 80,850
- 116 fakes in 7,000 training claims; 75 in the 3,000 test claims.
- "Always genuine" scores 0.975 accuracy and catches 0 fakes.
- Logistic regression scores 0.981 and catches only 22 of 75; the cost is ₹80,850.
Why the model ignores the rare class
- Class imbalance: one class is far rarer than the other. The common class is the majority class (genuine, 98%); the rare one is the minority class (fake, 2%).
- Training minimises a loss added up over all rows. 98 of every 100 rows are genuine, so the cheapest way to make the total small is to lean hard towards "genuine".
- Accuracy hides it (the accuracy paradox, Episode 24). Use recall (share of fakes caught), precision (share of flags that are really fake), the precision-recall curve / average precision, or, best of all, the real cost of each kind of mistake.
Fixing the data: undersampling and oversampling
Picture a class of 50 with 49 loud students and 1 quiet one; the teacher mostly hears the loud ones.
| Classroom | Data fix | Gain | Price |
|---|---|---|---|
| Ask only 1 of the loud students | Random undersampling: randomly drop majority rows until the classes match | Balanced, small, fast | Throws away most of the data |
| Let the quiet student speak 49 times | Random oversampling: randomly copy minority rows (with repeats) until the classes match | Balanced, keeps all data | Exact copies can be memorised |
Both come from imbalanced-learn (imblearn), an open-source add-on to scikit-learn. fit_resample(X, y) returns new rows. Resample only the training rows: the test set must look like real life.
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from imblearn.under_sampling import RandomUnderSampler; from imblearn.over_sampling import RandomOverSampler
for name, sampler in [("undersampling", RandomUnderSampler(random_state=0)),
("oversampling", RandomOverSampler(random_state=0))]:
Xr, yr = sampler.fit_resample(X[tr], y[tr]) # resample the TRAINING rows only
print(name, "-> rows", len(yr), " classes", np.bincount(yr))
report(name, make_pipeline(StandardScaler(), LogisticRegression()).fit(Xr, yr).predict(X[te]))
Output:
undersampling -> rows 232 classes [116 116]
undersampling accuracy 0.881 caught 60/75 false alarms 342 Rs 42,600
oversampling -> rows 13768 classes [6884 6884]
oversampling accuracy 0.882 caught 64/75 false alarms 343 Rs 36,850
Undersampling trained on just 232 rows and caught 60 of 75; oversampling caught 64. Both raise about 340 false alarms, so accuracy falls to 0.88, yet the cost falls from ₹80,850 to ₹42,600 and ₹36,850. More fakes caught, more genuine claims checked: that is the trade.
SMOTE: new minority rows, not copies
A flexible model (for example a deep decision tree) can memorise exact copies and learn nothing about the area around them. SMOTE (Synthetic Minority Over-sampling Technique, Chawla et al., 2002) makes new minority points instead:
- Take a minority row a.
- Find its k nearest minority neighbours (k = 5 by default) and pick one, b.
- Make a new row on the line between them: new = a + λ · (b − a), with λ a random number between 0 and 1 (λ = 0 gives a, λ = 1 gives b, λ = 0.4 is 40% of the way).
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
fakes = X[tr][y[tr] == 1][["account_days", "claims_30d"]].values # the 116 fake claims in training
a = fakes[0]; d = np.linalg.norm(fakes - a, axis=1); d[0] = np.inf
b = fakes[d.argmin()] # a's nearest fake neighbour
lam = 0.4; new = a + lam * (b - a) # a point on the line from a to b
print("a:", a.round(1), " b:", b.round(1), " new:", new.round(1))
from imblearn.over_sampling import SMOTE
Xr, yr = SMOTE(random_state=0).fit_resample(X[tr], y[tr]) # k_neighbors=5 by default
print("SMOTE -> rows", len(yr), " classes", np.bincount(yr), " new fakes made:", len(yr) - 7000)
report("SMOTE", make_pipeline(StandardScaler(), LogisticRegression()).fit(Xr, yr).predict(X[te]))
Output:
a: [62.7 3. ] b: [62.5 4. ] new: [62.6 3.4]
SMOTE -> rows 13768 classes [6884 6884] new fakes made: 6768
SMOTE accuracy 0.897 caught 64/75 false alarms 299 Rs 34,650
One step by hand: a = (62.7 days, 3 claims), b = (62.5 days, 4 claims), λ = 0.4 → a new fake with 3.4 claims. SMOTE invents in-between values even for counts; for true categories imbalanced-learn has variants such as SMOTENC and SMOTEN. Then real SMOTE made 6,768 new fakes, caught 64 of 75 with 299 false alarms (fewer than plain oversampling), cost ₹34,650.
SMOTE trusts every real minority row. If one fake is an odd outlier (or a wrong label), SMOTE draws new points on lines out to it, spreading the noise.
Fixing the training: class weights (cost-sensitive learning)
Instead of changing the data, change the scoring: multiply each row's loss by the weight of its class. class_weight="balanced" sets
weight = n_samples / (n_classes × count of that class)
so fakes get 7,000 / (2 × 116) ≈ 30.17 and genuine claims 7,000 / (2 × 6,884) ≈ 0.508: missing one fake now hurts as much as about 60 genuine mistakes. Most scikit-learn classifiers accept class_weight; there are no new rows and no extra training time.
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from sklearn.utils.class_weight import compute_class_weight
w = compute_class_weight("balanced", classes=np.array([0, 1]), y=y[tr]) # n_samples / (n_classes * count)
print("weights: genuine", w[0].round(3), " fake", w[1].round(3), " check:", round(7000 / (2 * 116), 3))
model = make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced")).fit(X[tr], y[tr])
report("class weights", model.predict(X[te]))
Output:
weights: genuine 0.508 fake 30.172 check: 30.172
class weights accuracy 0.884 caught 64/75 false alarms 336 Rs 36,500
Using unequal costs of mistakes to train a model, or to make its decisions, is called cost-sensitive learning. Class weights are one form of it.
| Method | Fakes caught (of 75) | False alarms | Cost on test |
|---|---|---|---|
| Always genuine | 0 | 0 | ₹112,500 |
| Plain logistic regression (accuracy 0.981) | 22 | 5 | ₹80,850 |
| Random undersampling | 60 | 342 | ₹42,600 |
| Random oversampling | 64 | 343 | ₹36,850 |
| SMOTE | 64 | 299 | ₹34,650 |
class_weight="balanced" |
64 | 336 | ₹36,500 |
| Threshold moving (0.033), no retraining | 54 | 193 | ₹43,850 |
Fixing the decision: threshold moving
predict() flags a claim when its predicted chance of being fake is at least 0.5, which only makes sense when both mistakes cost the same. With costs: if a claim is fake with probability P, not flagging it loses ₹1,500 with chance P (P × 1,500 on average); flagging costs ₹50 and the check stops the fake. Flag when P × 1,500 > 50, that is P > 50 / 1,500 = 0.033. Changing the cut-off instead of the model is threshold moving (Episode 25).
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X[tr], y[tr])
prob = model.predict_proba(X[te])[:, 1] # chance that each claim is fake
t = CHECK / MISS; print("cost-based threshold:", round(t, 3)) # flag when P x 1500 > 50
for name, thr in [("threshold 0.5", 0.5), (f"threshold {t:.3f}", t)]:
report(name, (prob >= thr).astype(int)) # flag a claim when its chance >= threshold
Output:
cost-based threshold: 0.033
threshold 0.5 accuracy 0.981 caught 22/75 false alarms 5 Rs 80,850
threshold 0.033 accuracy 0.929 caught 54/75 false alarms 193 Rs 43,850
Same model, new cut-off: 54 fakes caught instead of 22, cost ₹43,850 instead of ₹80,850. It did not reach SMOTE or class weights here, because the formula trusts the probabilities and this plain model underestimates how likely the fakes are. Practice 3 picks the threshold on cross-validated training predictions instead; Practice 4 shows that all three models rank the claims almost equally well (same PR-AUC), so most of the gain from the fixes comes from where the decision line is drawn.
Resampling inside a pipeline
scikit-learn's Pipeline only accepts steps with fit and transform. A resampler changes the rows, so it has no transform:
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from imblearn.over_sampling import SMOTE
try: make_pipeline(SMOTE(random_state=0), LogisticRegression()).fit(X[tr], y[tr]) # scikit-learn's Pipeline
except TypeError as e: print("TypeError:", e)
from imblearn.pipeline import make_pipeline as imb_pipeline # resamplers allowed, only during fit
model = imb_pipeline(StandardScaler(), SMOTE(random_state=0), LogisticRegression()).fit(X[tr], y[tr])
report("imb pipeline", model.predict(X[te]))
Output:
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'SMOTE(random_state=0)' (type <class 'imblearn.over_sampling._smote.base.SMOTE'>) doesn't
imb pipeline accuracy 0.890 caught 63/75 false alarms 317 Rs 37,000
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'SMOTE(random_state=0)' (type <class 'imblearn.over_sampling._smote.base.SMOTE'>) doesn't. Use imblearn.pipeline.make_pipeline: it runs resamplers only while fitting and skips them when predicting, so new claims are never resampled.
The worst mistake: resampling before cross-validation
Oversampling makes about 59 copies of each fake. If you oversample first and then split into folds, copies of the same fake land in both the training folds and the validation fold, and the model is tested on claims it has already seen:
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from imblearn.over_sampling import RandomOverSampler; from imblearn.pipeline import make_pipeline as imb_pipeline
from sklearn.tree import DecisionTreeClassifier as Tree; from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
cv = StratifiedKFold(5, shuffle=True, random_state=0)
Xr, yr = RandomOverSampler(random_state=0).fit_resample(X[tr], y[tr]) # MISTAKE: copies made BEFORE the split
leaky = cross_val_score(Tree(random_state=0), Xr, yr, cv=cv, scoring="recall")
print("oversample, then CV recall:", leaky.mean().round(3))
safe = imb_pipeline(RandomOverSampler(random_state=0), Tree(random_state=0)) # copies made inside each fold
print("inside each fold recall:", cross_val_score(safe, X[tr], y[tr], cv=cv, scoring="recall").mean().round(3))
Output:
oversample, then CV recall: 1.0
inside each fold recall: 0.274
Recall 1.000 (fake) versus 0.274 (honest). The deep tree memorised every copy. Resampling is part of training: put it inside an imblearn pipeline so it happens inside each training fold only.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Trust accuracy on a rare class | 0.981 accuracy, 22 of 75 fakes caught | Majority class dominates | Recall, precision, PR-AUC, cost |
| Resample the test set | scores no longer mean anything | Test must look like real life | Resample training rows only |
SMOTE inside scikit-learn's Pipeline |
TypeError: All intermediate steps should be transformers and implement fit and transform ... |
Resamplers have no transform |
imblearn.pipeline.make_pipeline |
| Oversample, then cross-validate | recall 1.000 vs 0.274 | Copies land in both train and validation folds | Resample inside each fold |
| Keep the 0.5 threshold | ₹80,850 vs ₹43,850 | Mistakes cost different amounts | Threshold from costs or validation |
| SMOTE on noisy labels | new points spread towards outliers | SMOTE trusts every minority row | Clean labels; combine with undersampling |
Mini project: a cost dashboard
Write compare(models) that fits each model (plain, class weights, SMOTE, undersampling, threshold moving) on the training claims inside an imblearn pipeline, and prints a table of fakes caught, false alarms, recall, precision and rupee cost on the test claims, sorted by cost. Change MISS to ₹500 and CHECK to ₹100: does the winner change? (It should move towards fewer flags.)
Practice
1. Combine SMOTE with random undersampling.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from imblearn.over_sampling import SMOTE; from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import make_pipeline as imb_pipeline
model = imb_pipeline(StandardScaler(), SMOTE(sampling_strategy=0.3, random_state=0), # fakes up to 30% of genuine
RandomUnderSampler(sampling_strategy=0.6, random_state=0), LogisticRegression()) # then genuine down
model.fit(X[tr], y[tr]); report("SMOTE+under", model.predict(X[te]))
Output:
SMOTE+under accuracy 0.919 caught 55/75 false alarms 224 Rs 43,950
SMOTE first grows the fakes to 30% of the genuine count, then undersampling shrinks the genuine claims until fakes are 60% of them. A milder balance than 50/50 gives fewer false alarms (224) but also fewer fakes caught (55); with these costs the full balance was cheaper.
2. Try your own class weights {0: 1, 1: w} and compare the costs.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
for w in [1, 5, 10, 30, 60]:
model = make_pipeline(StandardScaler(), LogisticRegression(class_weight={0: 1, 1: w})).fit(X[tr], y[tr])
report(f"fake weight {w}", model.predict(X[te]))
Output:
fake weight 1 accuracy 0.981 caught 22/75 false alarms 5 Rs 80,850
fake weight 5 accuracy 0.977 caught 38/75 false alarms 33 Rs 59,050
fake weight 10 accuracy 0.967 caught 43/75 false alarms 67 Rs 53,500
fake weight 30 accuracy 0.927 caught 55/75 false alarms 199 Rs 42,700
fake weight 60 accuracy 0.883 caught 64/75 false alarms 340 Rs 36,700
The heavier the fake weight, the more fakes caught and the more false alarms. "balanced" (about 30.2 vs 0.508, a ratio of about 59) behaves like a weight of 60.
3. Pick the threshold with the lowest cost on cross-validated training predictions.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from sklearn.model_selection import cross_val_predict
model = make_pipeline(StandardScaler(), LogisticRegression())
p_cv = cross_val_predict(model, X[tr], y[tr], cv=5, method="predict_proba")[:, 1] # honest probabilities for TRAIN rows
ts = np.arange(0.01, 0.51, 0.01)
cv_cost = [((p_cv < t) & (y[tr] == 1)).sum() * MISS + (p_cv >= t).sum() * CHECK for t in ts]
best_t = ts[int(np.argmin(cv_cost))]; print("threshold with the lowest CV cost:", round(best_t, 2))
prob = model.fit(X[tr], y[tr]).predict_proba(X[te])[:, 1]
report(f"threshold {best_t:.2f}", (prob >= best_t).astype(int))
Output:
threshold with the lowest CV cost: 0.04
threshold 0.04 accuracy 0.938 caught 52/75 false alarms 164 Rs 45,300
cross_val_predict gives each training claim a probability from a model that never saw it, so the threshold is chosen without touching the test set. It chose 0.04 (₹45,300 on test), close to the formula's 0.033.
4. Compare average precision (PR-AUC) of the plain, class-weighted and SMOTE models.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(41); n = 10000 # Kiran's refund claims (made up)
y = (rng.random(n) < 0.02).astype(int) # 1 = fake claim: about 2 in 100
X = pd.DataFrame({"account_days": rng.exponential(400, n) * np.where(y, .25, 1), # fakes: newer accounts
"claims_30d": rng.poisson(np.where(y, 2.2, .6)), # more claims this month
"amount": rng.gamma(2, np.where(y, 420, 260)), # bigger orders
"photo": rng.random(n) < np.where(y, .35, .8)}).astype(float) # sent a photo of the problem?
tr, te = slice(0, 7000), slice(7000, None) # 7,000 train · 3,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression; from sklearn.metrics import confusion_matrix
MISS, CHECK = 1500, 50 # rupees: a missed fake · checking one claim
def report(name, p): # p = 0/1 predictions on the test claims
tn, fp, fn, tp = confusion_matrix(y[te], p).ravel(); cost = fn * MISS + (tp + fp) * CHECK
print(f"{name:<15} accuracy {(tp + tn) / len(p):.3f} caught {tp}/{tp + fn} false alarms {fp:>3} Rs {cost:,}")
from sklearn.metrics import average_precision_score
from imblearn.over_sampling import SMOTE; from imblearn.pipeline import make_pipeline as imb_pipeline
models = {"plain": make_pipeline(StandardScaler(), LogisticRegression()),
"class weights": make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced")),
"SMOTE": imb_pipeline(StandardScaler(), SMOTE(random_state=0), LogisticRegression())}
for name, m in models.items():
prob = m.fit(X[tr], y[tr]).predict_proba(X[te])[:, 1]
print(f"{name:<14} average precision (PR-AUC) {average_precision_score(y[te], prob):.3f}")
Output:
plain average precision (PR-AUC) 0.510
class weights average precision (PR-AUC) 0.510
SMOTE average precision (PR-AUC) 0.506
Almost identical (about 0.51). These fixes barely change how well the model ranks claims; they mostly change where the line is drawn. That is why threshold moving can get much of the benefit without retraining, and why better features are still the biggest lever.
Quick quiz
1,000 training rows: 950 normal machines and 50 broken ones. With class_weight="balanced", what is the weight of a broken machine? Answer: 10, because 1,000 / (2 × 50) = 10. Normal machines get 1,000 / (2 × 950) ≈ 0.53.
Interview questions
- Why is accuracy misleading on imbalanced data? Predicting the majority class gets high accuracy while catching none of the minority (0.975 here).
- Oversampling vs undersampling: trade-offs? Undersampling discards data; oversampling keeps it but duplicates rows that flexible models can memorise.
- How does SMOTE create a sample? new = a + λ(b − a) for a minority row a, one of its k nearest minority neighbours b, and random λ in [0, 1].
- What does
class_weight="balanced"compute? n_samples / (n_classes × class count) for each class. - What is cost-sensitive learning? Training or deciding with unequal costs for different mistakes, e.g. class weights or a cost-based threshold.
- How do you choose a threshold from costs? Flag when P × cost_of_miss > cost_of_check (here P > 0.033), or pick the threshold with the lowest cost on validation predictions.
- Why must resampling happen inside cross-validation? Otherwise copies or synthetic neighbours of the same row land in both training and validation folds and inflate scores (recall 1.000 vs 0.274).
- Why can't SMOTE go in a scikit-learn Pipeline? It has
fit_resample, nottransform; use imbalanced-learn's pipeline. - Should you resample the test set? Never: the test set must keep the real class balance.
- Which metric would you report for fraud detection? Recall and precision at the chosen threshold, PR-AUC, and the expected cost.
Glossary
| Term | Meaning, with an example |
|---|---|
| Class imbalance | One class is much rarer than another: 2% fake claims |
| Majority class | The common class: genuine claims |
| Minority class | The rare class: fake claims |
| Accuracy paradox | High accuracy while missing the class you care about (0.975 by always saying genuine) |
| Recall | Share of real positives caught: 22 / 75 = 0.293 |
| Precision | Share of flags that are really positive |
| False alarm (false positive) | A genuine claim flagged as fake |
| Resampling | Changing the training rows to rebalance the classes |
| Random undersampling | Randomly dropping majority rows (6,884 → 116) |
| Random oversampling | Randomly copying minority rows with repeats (116 → 6,884) |
| SMOTE | Synthetic Minority Over-sampling Technique: new minority rows on lines between neighbouring minority rows |
| λ (lambda) in SMOTE | Random number in [0, 1]: how far along the line from a to b the new point is |
k_neighbors |
How many nearby minority rows SMOTE chooses from (default 5) |
imbalanced-learn (imblearn) |
Open-source library of resamplers that works with scikit-learn |
fit_resample |
Method that returns resampled X and y |
| Class weight | Multiplier for a class's loss: fake 30.17, genuine 0.508 |
class_weight="balanced" |
n_samples / (n_classes × class count) |
| Cost-sensitive learning | Using unequal costs of mistakes in training or decisions |
| Cost matrix | The cost of each kind of mistake: missed fake ₹1,500, check ₹50 |
| Threshold | Probability cut-off above which a row is flagged (default 0.5) |
| Threshold moving | Changing the cut-off instead of the model: 0.5 → 0.033 |
| Calibration | Whether predicted probabilities match real frequencies (Episode 25) |
| PR-AUC (average precision) | Area under the precision-recall curve: ranking quality for the rare class |
| imblearn pipeline | A pipeline that runs resamplers only during fit |
| Resampling leakage | Resampling before splitting, so copies sit in both train and validation |
Summary
- With rare classes, accuracy lies: 0.981 accuracy caught 22 of 75 fakes.
- Fix the data (undersampling, oversampling, SMOTE), the training (class weights), or the decision (threshold moving); compare them by real costs.
- SMOTE makes new minority rows between neighbours; it can spread noisy labels.
class_weight="balanced"is one argument, no new rows.- Threshold = cost of a check / cost of a miss when probabilities are trustworthy; otherwise choose it on validation.
- Resample only training rows, inside an imblearn pipeline.
Next episode: hyperparameter tuning: grid search, random search, Optuna and Hyperband, without leaking the test set.