Machine Learning Full Course · Episode 87. Every program below was run on a Linux machine with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1, and the output shown is the real output (latency numbers depend on the computer and will differ on yours). All data is made up.
The big idea: in an ML system design round, the model is the easy part. The interviewer wants to see you turn a vague business problem into a system that works on day one and keeps working: clarify, frame, metrics, data and labels, features, model, serving, monitoring. Say each step out loud, in order, with numbers.
Words you need first
- System design: planning a whole system (data, model, serving, monitoring), not just a model.
- Threshold: the score above which we act (Episode 22).
- AUC (ROC-AUC): chance a random positive gets a higher score than a random negative; 0.5 = guessing (Episode 25).
- Recall@10: share of users whose wanted item appears in our top 10.
- Percentile: the value below which a given share of measurements fall; the 99th percentile is beaten by only 1%.
- Distribution: how often each value of a feature occurs (Episode 10).
The story: "I'd use gradient boosting"
Arjun is asked to design a fraud detector for our payments app. Ten seconds later: "I'd train gradient boosting on the data." The interviewer asks: what exactly is a label? How fast must a decision be? What does a wrong decision cost us? No answers. Jumping straight to a model fails because a real system needs everything around the model.
The 8-step framework
| Step | Question to answer | Fraud example |
|---|---|---|
| 1. Clarify requirements | users, scale, speed, constraints | UPI and card payments, millions a day, decide before the payment completes |
| 2. Frame the problem | business goal → ML task → decision | lose less to fraud without annoying good customers → P(fraud) per payment → allow / review / block |
| 3. Metrics | offline, online, business | PR-AUC offline; fraud losses and blocked good payments online; money saved |
| 4. Data and labels | where labels come from, how late | chargebacks, analyst decisions; weeks of delay |
| 5. Features | what the model sees at decision time | payments in the last 10 minutes, new device, usual amount |
| 6. Model | baseline first | rules + logistic regression → gradient-boosted trees |
| 7. Serving | online or batch, latency budget | online, 50 ms for the whole check |
| 8. Monitoring | drift, feedback, retraining | PSI on features, flag rate, delayed-label metrics |
Steps 1-2: clarify and frame
- Business objective: the goal in money or customer terms ("lose less to fraud without annoying good customers").
- ML task: what the model learns, e.g. binary classification giving P(fraud) for each payment.
- Decision: what the system does with the score: allow, send to review, or block, using two thresholds. The model gives a score; the business decides what to do with it.
Step 3: metrics
Real-life example: cricket. Runs in the nets against a bowling machine (offline metric: quick, safe, on saved data), runs in a real match (online metric: on live traffic), and whether the team won (business metric). A good design links all three.
| Kind | Measured on | Fraud examples |
|---|---|---|
| Offline | saved, labelled data | PR-AUC, recall at a fixed review budget |
| Online | live traffic | fraud losses per day, good payments blocked, complaints |
| Business | the company's results | money saved, customers kept |
Choose thresholds by cost. A missed fraud loses the payment amount; each flagged payment costs a review (say Rs 100) and a delayed customer. The obvious attempt, the default 0.5, ignores both.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(88)
n = 50000 # made-up payments, about 2% fraud
X = rng.normal(size=(n, 4))
amount = rng.lognormal(7.5, 1, n).round() # rupees
fraud = X[:, 0] + 0.8 * X[:, 1] + rng.normal(0, 1, n) > 3.0
Xtr, Xte, ytr, yte, atr, ate = train_test_split(X, fraud, amount, random_state=0)
p = LogisticRegression().fit(Xtr, ytr).predict_proba(Xte)[:, 1]
REVIEW = 100 # rupees: an analyst checks one flagged payment
def cost(t): # money lost: missed fraud + review work
flag = p >= t
return ate[yte & ~flag].sum() + REVIEW * flag.sum(), flag.sum()
for t in [0.5, 0.2, 0.05]:
c, k = cost(t)
print(f"threshold {t:4}: flagged {k:5} cost Rs {c / 1e5:6.1f} lakh")
best = min(np.arange(0.01, 0.99, 0.01), key=lambda t: cost(t)[0])
print(f"cheapest threshold {best:.2f}: cost Rs {cost(best)[0] / 1e5:.1f} lakh")
Output:
threshold 0.5: flagged 163 cost Rs 9.5 lakh
threshold 0.2: flagged 531 cost Rs 6.1 lakh
threshold 0.05: flagged 1621 cost Rs 3.6 lakh
cheapest threshold 0.02: cost Rs 3.5 lakh
Same model, same AUC: moving the threshold from 0.5 to 0.02 cuts the cost from 9.5 to 3.5 lakh rupees. (A real design would also cap the flags by how many reviews analysts can do per day.)
Step 4: data and labels
- Label sources: chargebacks (a customer disputes a payment with the bank), analyst decisions, user reports.
- Label delay: a chargeback can arrive weeks after the payment, so recent data has incomplete labels.
- Class imbalance: fraud is rare: class weights, PR-AUC, careful sampling (Episodes 24-27).
- Concept drift: the link between features and the label changes over time (fraudsters change tactics).
- Time-based split: train on the past, test on the future, like production. A random split mixes future rows into training.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(87)
n = 60000 # made-up payments over 12 months
month = np.sort(rng.integers(1, 13, n))
amount = rng.lognormal(7, 1, n) # rupees
night, new_device = rng.random(n) < 0.2, rng.random(n) < 0.1
old_trick = night & (month <= 8) # fraudsters' old pattern: night payments
new_trick = new_device & (month >= 9) # from month 9 they switch to new devices
fraud = rng.random(n) < 0.004 + 0.06 * old_trick + 0.06 * new_trick
X = np.c_[np.log(amount), night, new_device]
Xa, Xb, ya, yb = train_test_split(X, fraud, random_state=0) # random split
auc = roc_auc_score(yb, LogisticRegression().fit(Xa, ya).predict_proba(Xb)[:, 1])
print(f"random split (future mixed into training): AUC {auc:.3f}")
tr, te = month <= 9, month >= 10 # time split: past -> future
auc = roc_auc_score(fraud[te], LogisticRegression().fit(X[tr], fraud[tr]).predict_proba(X[te])[:, 1])
print(f"time split (train months 1-9, test 10-12): AUC {auc:.3f}")
Output:
random split (future mixed into training): AUC 0.788
time split (train months 1-9, test 10-12): AUC 0.697
The random split already "saw" the new-device trick from months 10-12 during training; the time split shows the real, lower performance, and argues for regular retraining.
Steps 5-6: features and model
| Feature type | Examples |
|---|---|
| Real-time | payments by this user in the last 10 minutes (velocity), new device?, new payee? |
| History | account age, usual amount, usual hours |
| Context | merchant category, amount, time of day |
- Feature store: one system that computes features the same way for training and for serving, and stores them for reuse.
- Training-serving skew: a feature computed one way in training and another way live; it silently breaks the model.
- Model: start with simple rules and a logistic regression baseline, then gradient-boosted trees; plan explanations for blocked customers (SHAP, Episode 83).
Step 7: serving and latency
- Online serving: predict per request, immediately (fraud). Batch serving: precompute predictions on a schedule (a daily offers email).
- Latency: time from request to answer. A latency budget is the maximum allowed, e.g. 50 ms for the whole fraud check, features included.
- p50 / p99 latency: half / 99% of requests are faster than this. p99 is the slow tail users feel.
- Trap: timing a whole batch and dividing by the rows; live requests arrive one at a time.
import numpy as np, time
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
X, y = make_classification(n_samples=5000, n_features=20, random_state=0) # made-up data
models = {"logistic regression": LogisticRegression(max_iter=1000).fit(X, y),
"random forest, 300 trees": RandomForestClassifier(300, random_state=0).fit(X, y)}
one = X[:1] # ONE payment arrives: predict it now
for name, m in models.items():
times = []
for _ in range(200): # 200 single requests, like live traffic
t = time.perf_counter(); m.predict_proba(one); times.append(time.perf_counter() - t)
ms = np.array(times) * 1000
print(f"{name:25} p50 {np.percentile(ms, 50):6.2f} ms p99 {np.percentile(ms, 99):6.2f} ms")
t = time.perf_counter(); models["random forest, 300 trees"].predict_proba(X[:1000])
print(f"forest, 1,000 rows in one batch: {(time.perf_counter() - t) * 1000:.1f} ms total")
Output (one run; your milliseconds will differ):
logistic regression p50 0.08 ms p99 0.15 ms
random forest, 300 trees p50 10.76 ms p99 34.43 ms
forest, 1,000 rows in one batch: 26.2 ms total
Logistic regression answers in a small fraction of a millisecond; the 300-tree forest needs several milliseconds per request, and its p99 is higher. A batch of 1,000 rows costs only a few times a single request, so "batch time ÷ 1,000" would make the forest look almost free.
Step 8: monitoring
Real-life example: a shop after a festival sale. Customers buy bigger items (data drift: a feature's distribution shifts); shoplifters learn where the cameras are (concept drift: the old warning signs stop working); the shopkeeper counts stock every week (monitoring).
Population Stability Index (PSI): split the training values of a feature into bins (here 10 equal-share bins), then
PSI = Σ over bins (live% − train%) × ln(live% ÷ train%)
A common rule of thumb: below 0.1 stable, 0.1-0.25 worth watching, above 0.25 a big shift.
import numpy as np
rng = np.random.default_rng(89)
train_amount = rng.lognormal(7.0, 1.0, 50000) # payment amounts the model was trained on
week1 = rng.lognormal(7.0, 1.0, 5000) # live traffic, a normal week
week9 = rng.lognormal(7.6, 1.0, 5000) # live traffic after a big festival sale
def psi(expected, actual, bins=10): # Population Stability Index
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
return np.sum((a - e) * np.log(a / e))
for name, live in [("week 1", week1), ("week 9", week9)]:
v = psi(train_amount, live)
flag = "stable" if v < 0.1 else "watch" if v < 0.25 else "ALERT: investigate"
print(f"{name}: median Rs {np.median(live):6.0f} PSI {v:.3f} {flag}")
Output:
week 1: median Rs 1119 PSI 0.003 stable
week 9: median Rs 2014 PSI 0.346 ALERT: investigate
Also monitor the score distribution, the flag rate, latency, and, once the delayed labels arrive, precision and recall.
Design 1: the fraud detector, end to end
Payment request → Feature store (real-time + history) → Model: P(fraud) → Decision: allow / review / block
↑ |
Retrain on a schedule (time split) ← Delayed labels: chargebacks, reviews ←┘
- Rules for known patterns run beside the model; analysts review the middle band of scores.
- Shadow mode: the new model scores live payments but doesn't act; compare with the old one safely. Then an A/B test, then full traffic.
- Feedback loop: payments we block never get a true label, so the model can't learn it was wrong. Let a tiny random sample through (or review it) to keep honest labels.
Design 2: a food-delivery recommender
Real-life example: a good waiter. From 300 dishes he quickly thinks of 20 this guest might like (candidate generation: millions → hundreds, cheaply), orders those 20 by what suits the guest tonight (ranking: a stronger model scores each), and won't suggest five paneer dishes in a row (re-ranking: diversity, freshness, business rules).
- Frame: rank dishes by P(user orders it). Implicit feedback: orders and clicks, not star ratings.
- Offline metric: recall@10. Online: orders per session in an A/B test. Coverage: how many different items ever get shown.
- Cold start: a new user or dish has no history: fall back on popularity, location and content features (cuisine).
- Item-item recommendation: score a dish by how often it was ordered together with what the user already ordered.
import numpy as np
rng = np.random.default_rng(90)
U, I, G = 2000, 300, 6 # made-up: users, items (dishes), cuisines
genre = rng.integers(0, G, I); pop = rng.zipf(1.6, I).clip(1, 50).astype(float) # a few items are hits
R = np.zeros((U, I)) # R[u, i] = 1 if user u ordered item i
for u in range(U):
likes = rng.choice(G, 2, replace=False) # each user likes 2 cuisines
w = pop * np.isin(genre, likes) + 0.02 * pop
R[u, rng.choice(I, 12, replace=False, p=w / w.sum())] = 1
test = np.array([rng.choice(np.flatnonzero(r)) for r in R]) # hide one order per user
R[np.arange(U), test] = 0
def hit_rate(scores): # recall@10: hidden item in the top 10?
scores = np.where(R > 0, -np.inf, scores) # never re-recommend what they ordered
top = np.argsort(-scores, axis=1)[:, :10]
return (top == test[:, None]).any(1).mean(), len(np.unique(top))
co = R.T @ R; np.fill_diagonal(co, 0) # item-item: ordered together how often
for name, s in [("popularity", np.tile(R.sum(0), (U, 1))), ("item-item", R @ co)]:
hr, cov = hit_rate(s)
print(f"{name:11} recall@10 {hr:.3f} distinct items shown {cov:3} of {I}")
Output:
popularity recall@10 0.212 distinct items shown 18 of 300
item-item recall@10 0.560 distinct items shown 60 of 300
Popularity shows everyone the same 18 hits (recall@10 0.212); item-item more than doubles recall (0.560) and shows 60 different dishes. Popularity remains the cold-start fallback, and a ranking model would sit on top of these candidates.
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Starting with the model | no label, latency or cost answers | skipped steps 1-4 | walk the 8 steps |
| Default threshold 0.5 | Rs 9.5 lakh vs Rs 3.5 lakh | ignores the cost of mistakes | threshold from costs |
| Random split on time data | AUC 0.788, really 0.697 | the future leaks into training | time-based split |
| Different feature code live | good offline, poor live | training-serving skew | a shared feature store |
| Batch timing for latency | the forest looks almost free | hides per-request cost | single requests, p50 and p99 |
| No monitoring | PSI 0.346 unnoticed | data drift | PSI, flag rate, label metrics |
| Training only on unblocked payments | the model never learns its mistakes | feedback loop | a small random holdout |
| Popularity for everyone | recall@10 0.212, 18 items shown | not personal | candidates + ranking |
Mini project: design spam-call detection
Write a one-page design for detecting spam phone calls, using the 8 steps: requirements (calls per second, decide before the phone rings), framing (P(spam) → allow / warn / block), metrics (offline, online, business), labels (user reports, delayed), features (calls per hour from the number, share of short calls, age of the number), model (baseline then trees), serving (latency budget), monitoring (PSI on features, report rate). Draw the pipeline as in Design 1.
Practice
1. Change REVIEW = 100 to 300 in the threshold program. Will the cheapest threshold go up or down? Why?
Solution
Up: each flag now costs more, so it pays to flag fewer payments, i.e. use a higher threshold. In our run the cheapest threshold moves from 0.02 to 0.10 (cost Rs 6.3 lakh).
2. Compute PSI for two samples from the same distribution (change week9 to rng.lognormal(7.0, 1.0, 5000)). What do you expect?
Solution
A value close to 0 (like week 1's 0.003): "stable". Random sampling alone gives small non-zero values.
3. Your fraud model's p50 latency is 8 ms but p99 is 120 ms with a 50 ms budget. What do you do?
Solution
The tail breaks the budget. Find the slow part (often feature lookups), cache or precompute features, use a smaller model (fewer or shallower trees, or a distilled model), add a timeout with a rules-only fallback, and scale out serving.
4. Add a cold-start fallback to the recommender: users with no orders get the popular list.
Solution
scores = np.where(R.sum(1, keepdims=True) == 0, popularity_scores, item_item_scores), where popularity_scores = np.tile(R.sum(0), (U, 1)) and item_item_scores = R @ co.
5. Name one offline, one online and one business metric for a recommender.
Solution
Offline: recall@10. Online: orders per session (A/B test). Business: revenue per user or retention.
Quick quiz
A loan model scores AUC 0.91 offline; live, the app computes "days since last payment" differently from the training code, and results are poor. The problem: training-serving skew. A shared feature store prevents it.
Interview questions
- Design a fraud detector for a payments app. Walk through all 8 steps.
- How do you choose the decision threshold? What information do you need from the business?
- Why can a random train-test split overestimate performance? What would you do instead?
- What is label delay, and how does it affect training and monitoring?
- What is training-serving skew, and how does a feature store help?
- Explain p50 and p99 latency. How would you meet a 50 ms budget?
- Data drift vs concept drift: how would you detect each?
- How do you launch a new model safely?
- Design a recommender: candidate generation, ranking, re-ranking, cold start, metrics.
- What is a feedback loop in a fraud or recommendation system, and how do you break it?
Glossary
| Term | Meaning, with an example |
|---|---|
| ML system design | Planning data, model, serving and monitoring together: a fraud detector. |
| Requirements | Users, scale, speed and constraints: millions of payments a day, decide in 50 ms. |
| Business objective | The goal in money/customer terms: lose less to fraud. |
| ML task | What the model learns: binary classification, P(fraud). |
| Decision | What the system does with a score: allow / review / block. |
| Offline metric | Measured on saved data: AUC 0.697 on a time split. |
| Online metric | Measured on live traffic: fraud losses per day. |
| Business metric | Company result: money saved, customers kept. |
| Cost-based threshold | Threshold minimising total cost: 0.02 → Rs 3.5 lakh. |
| Chargeback | A customer disputes a payment with the bank; a fraud label source. |
| Label delay | Labels arriving long after the event: weeks for chargebacks. |
| Concept drift | Feature-label link changing: night payments → new devices. |
| Data drift | A feature's distribution shifting: median Rs 1,119 → Rs 2,014. |
| Time-based split | Train on the past, test on the future. |
| Velocity feature | Count of recent events: payments in the last 10 minutes. |
| Feature store | One system computing features the same way for training and serving. |
| Training-serving skew | A feature computed differently live than in training. |
| Baseline | A simple first model to beat: rules + logistic regression. |
| Online serving | Predict per request, immediately. |
| Batch serving | Precompute predictions on a schedule. |
| Latency | Time from request to answer. |
| Latency budget | The maximum allowed latency: 50 ms. |
| p50 / p99 | Latency that half / 99% of requests beat. |
| Monitoring | Tracking inputs, scores, outcomes and latency over time. |
| PSI (Population Stability Index) | Σ (live% − train%)·ln(live%/train%): 0.003 stable, 0.346 alert. |
| Shadow mode | The new model scores live traffic without acting. |
| A/B test | Comparing two versions on random user groups. |
| Feedback loop | The system's own decisions shaping its future training data. |
| Recommender system | Suggests items a user is likely to want. |
| Candidate generation | Cheaply picking hundreds of items from millions. |
| Ranking | A stronger model ordering the candidates. |
| Re-ranking | Final adjustments: diversity, freshness, rules. |
| Implicit feedback | Signals like orders and clicks, not ratings. |
| Recall@10 | Share of users whose next item is in our top 10: 0.212 vs 0.560. |
| Coverage | How many different items get shown: 18 vs 60 of 300. |
| Cold start | No history for a new user or item. |
| Popularity baseline | Recommend the most ordered items to everyone. |
| Item-item recommendation | Score items by co-occurrence with the user's items. |
Summary
- Don't start with the model: clarify, frame, metrics, data and labels, features, model, serving, monitoring.
- The model gives a score; thresholds come from business costs (Rs 9.5 → 3.5 lakh).
- Evaluate on a time split (0.788 random vs 0.697 time) and plan retraining.
- Use features available at decision time, computed the same way everywhere (feature store).
- Measure latency per request with p50 and p99 against a budget.
- Monitor drift with PSI and other signals; launch with shadow mode and A/B tests; watch feedback loops.
- Recommenders: candidates → ranking → re-ranking, with popularity for cold start.
Next episode: Project 1: house price prediction.