SyntaxPilot
Courses/Machine Learning/Lesson 86
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 86 of 90

Interview prep 2: coding and maths rounds

implementing kNNk-means and logistic regression from scratchNumPy and pandas questionsprobability puzzlesstatistics questionscomplexity of algorithmsdebugging a training loop

Machine Learning Full Course · Episode 86. Every program below was run on a Linux machine with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and SciPy 1.18.1, and the output shown is the real output (timings will differ on your computer). All data is made up.

The big idea: in a coding round the interviewer watches how you get to the answer. Restate the task and ask about inputs, write the simple correct version first while talking, test it on a tiny example, then improve it (usually by vectorising) and state its time complexity. In the maths round, answer with the formula and check it by simulation.

Words you need first

  • From scratch: using only NumPy, no scikit-learn model.
  • Vectorisation: doing a whole array's maths in one NumPy operation instead of a Python loop (Episode 2).
  • Broadcasting: NumPy stretching axes of size 1 so two arrays' shapes match (Episode 2).
  • Sigmoid: σ(z) = 1 / (1 + e⁻ᶻ), squashes any number into (0, 1) (Episode 22).
  • Log loss: −mean(y·log p + (1−y)·log(1−p)), the loss of logistic regression (Episode 22).
  • Time complexity, O(...): how running time grows with input size. O(n): double the rows, double the time. O(n²): double the rows, four times the time.
  • Space complexity: how memory grows with input size.

How coding rounds are judged

  1. Restate the task; ask about inputs: shapes, sizes, edge cases (empty input, ties, missing values).
  2. Write the simple, correct version first and talk while you write.
  3. Test it on a tiny example you can check by hand.
  4. Improve it (vectorise), and state time and space complexity.

Task 1: k-nearest neighbours from scratch

Real-life example: a broker prices a new flat (2 BHK, 900 sq ft, 3rd floor): she measures how similar every sold flat is (a distance), takes the 5 most similar (k = 5 nearest neighbours) and averages their prices, or, for a yes/no label, lets them vote.

First the loop version, then the vectorised one. The vectorised distance uses ‖a − b‖² = ‖a‖² + ‖b‖² − 2a·b for all pairs at once, and np.argpartition finds the k smallest without sorting each whole row.

import numpy as np, time
rng = np.random.default_rng(86)
Xtr = rng.normal(size=(2000, 5)); ytr = (Xtr[:, 0] + Xtr[:, 1] > 0).astype(int)   # made-up data
Xte = rng.normal(size=(300, 5)); yte = (Xte[:, 0] + Xte[:, 1] > 0).astype(int)
def knn_loops(k=5):                                   # first attempt: plain Python loops
    preds = []
    for a in Xte:
        d = [sum((a[j] - b[j]) ** 2 for j in range(5)) for b in Xtr]   # distance to every row
        near = sorted(range(len(d)), key=lambda i: d[i])[:k]          # k closest
        preds.append(int(sum(ytr[i] for i in near) > k / 2))          # majority vote
    return np.array(preds)
def knn_numpy(k=5):                                   # vectorised: all distances at once
    d = (Xte ** 2).sum(1)[:, None] + (Xtr ** 2).sum(1)[None, :] - 2 * Xte @ Xtr.T
    near = np.argpartition(d, k, axis=1)[:, :k]       # k smallest per row, no full sort
    return (ytr[near].sum(1) > k / 2).astype(int)
for f in [knn_loops, knn_numpy]:
    t = time.perf_counter(); p = f()
    print(f"{f.__name__:10} accuracy {(p == yte).mean():.3f}   time {time.perf_counter() - t:.3f} s")
print("same predictions:", (knn_loops() == knn_numpy()).all())

Output:

knn_loops  accuracy 0.940   time 0.957 s
knn_numpy  accuracy 0.940   time 0.006 s
same predictions: True

Same predictions; the NumPy version is around a hundred times faster because the loop version runs Python code about three million times (300 × 2,000 × 5), while NumPy does the same maths in compiled code, in big blocks.

Complexity (n training rows, d features, m test rows):

Time Space
Training O(1): just store the data (a lazy learner) O(n·d)
Prediction O(n·d) per test row, O(m·n·d) total O(m·n) for the distance matrix (300 × 2,000 = 600,000 numbers)

Follow-up "n is 10 million": the m × n matrix no longer fits in memory, so compute distances in chunks of test rows, or use an approximate nearest-neighbour (ANN) index, the idea behind vector databases.

Task 2: a NumPy shape bug with no error

  • Shape: the size of each axis. (1000,) is a flat 1-D vector; (1000, 1) is a column (2-D).
  • (1000, 1) − (1000,) broadcasts to (1000, 1000): a million differences, no error, wrong answer.
import numpy as np
rng = np.random.default_rng(2)
X = rng.normal(size=(1000, 3))
y = X @ np.array([1.0, 2.0, 3.0]) + rng.normal(0, 0.1, 1000)   # shape (1000,)
w = np.array([[1.0], [2.0], [3.0]])                  # weights as a column: shape (3, 1)
pred = X @ w                                         # shape (1000, 1)
err = pred - y                                       # (1000, 1) minus (1000,) ... broadcasting!
print("pred", pred.shape, " y", y.shape, " err", err.shape)
print(f"MSE with the bug: {np.mean(err ** 2):.3f}   (perfect weights, should be ~0.01)")
err = pred.ravel() - y                               # fix: make both 1-D (or y.reshape(-1, 1))
print("err", err.shape, f" MSE fixed: {np.mean(err ** 2):.3f}")

Output:

pred (1000, 1)  y (1000,)  err (1000, 1000)
MSE with the bug: 27.245   (perfect weights, should be ~0.01)
err (1000,)  MSE fixed: 0.010

Habit: print .shape after every step of a from-scratch model, and use ravel() or reshape(-1, 1) so both sides agree.

Task 3: k-means from scratch

Real-life example: placing 6 delivery hubs. Drop 6 pins anywhere (initialise k centres), every shop joins its nearest hub (assign step), move each hub to the middle of its shops (update step), repeat until no hub moves (converged).

  • Inertia: the sum of squared distances from each point to its centre. Lower = tighter clusters.
  • k-means only finds a local minimum: the result depends on the starting centres.
import numpy as np
rng = np.random.default_rng(3)
centres = np.array([[0, 0], [4, 0], [8, 0], [0, 4], [4, 4], [8, 4]])   # 6 made-up groups of shops
X = np.vstack([c + rng.normal(0, 0.6, (100, 2)) for c in centres])
def kmeans(X, k, seed, iters=100):
    r = np.random.default_rng(seed)
    C = X[r.choice(len(X), k, replace=False)]                # start: k random points
    for it in range(1, iters + 1):
        d = ((X[:, None, :] - C[None, :, :]) ** 2).sum(2)    # n x k squared distances
        lab = d.argmin(1)                                    # assign to the nearest centre
        newC = np.array([X[lab == j].mean(0) for j in range(k)])   # move centre to the mean
        if np.allclose(newC, C): break                       # nothing moved: converged
        C = newC
    return lab, C, d.min(1).sum(), it                        # inertia = sum of squared distances
runs = [kmeans(X, 6, seed) for seed in range(5)]
for seed, (lab, C, inertia, it) in enumerate(runs):
    print(f"start {seed}: {it:2} steps  inertia {inertia:6.1f}  {np.bincount(lab)}")   # group sizes
print(f"fix: run several starts, keep the lowest inertia -> {min(r[2] for r in runs):.1f}")

Output:

start 0:  7 steps  inertia  423.9  [100 101 100 100  99 100]
start 1:  4 steps  inertia  424.0  [100 100 100 100 100 100]
start 2: 14 steps  inertia  424.0  [100 100 100 100 100 100]
start 3:  9 steps  inertia  423.9  [101 100  99 100 100 100]
start 4: 10 steps  inertia 1159.9  [100  46 198  54 102 100]
fix: run several starts, keep the lowest inertia -> 423.9

Start 4 got stuck: one cluster of 198 swallowed two groups and another group was split 46/54. The fix: run several starts and keep the lowest inertia (scikit-learn's n_init), and use k-means++, which spreads the first centres apart (each new centre is chosen with probability proportional to its squared distance from the centres already chosen).

Time
One iteration O(n·k·d)
Whole run O(n·k·d·iterations)

Edge cases to mention: an empty cluster (no point joins a centre, so its mean is undefined): re-seed it at a random point. Choosing k: the elbow plot of inertia or the silhouette score (Episodes on clustering).

Task 4: logistic regression from scratch, and debugging the training loop

Story: you write the sigmoid, the log loss and a gradient step, feed in made-up loan data with income in rupees, and at epoch 2 the loss is nan ("not a number": the result of an impossible calculation such as 0 × ∞).

import numpy as np
rng = np.random.default_rng(4)
income = rng.normal(60000, 15000, 1000)               # made-up: monthly income in rupees
age = rng.normal(35, 8, 1000)
y = (0.0001 * (income - 60000) - 0.1 * (age - 35) + rng.normal(0, 1, 1000) > 0).astype(int)  # repaid loan?
X = np.c_[np.ones(1000), income, age]                 # bias column + raw features
sigmoid = lambda z: 1 / (1 + np.exp(-z))
w = np.zeros(3)
for epoch in range(1, 6):
    p = sigmoid(X @ w)
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))   # log loss
    print(f"epoch {epoch}: loss {loss:.3f}")
    w -= 0.1 * X.T @ (p - y) / len(y)                 # gradient step

Output:

epoch 1: loss 0.693
epoch 2: loss nan
epoch 3: loss nan
epoch 4: loss nan
epoch 5: loss nan

Output (error):

main.py:11: RuntimeWarning: divide by zero encountered in log
  loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))   # log loss
main.py:11: RuntimeWarning: invalid value encountered in multiply
  loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))   # log loss
main.py:7: RuntimeWarning: overflow encountered in exp
  sigmoid = lambda z: 1 / (1 + np.exp(-z))

Why: income is around 60,000, so the first gradient step makes the income weight large, X @ w becomes enormous, exp overflows, and the sigmoid gives exactly 0 or 1. Then log(0) = −∞ and 0 × −∞ = nan.

Fix 1 (the cause): standardise the features (mean 0, standard deviation 1, Episode 16). Fix 2 (a safety net): clip probabilities into [1e-12, 1 − 1e-12] so we never take log(0).

import numpy as np
rng = np.random.default_rng(4)
income = rng.normal(60000, 15000, 1000)               # made-up: monthly income in rupees
age = rng.normal(35, 8, 1000)
y = (0.0001 * (income - 60000) - 0.1 * (age - 35) + rng.normal(0, 1, 1000) > 0).astype(int)  # repaid loan?
F = np.c_[income, age]
F = (F - F.mean(0)) / F.std(0)                        # fix 1: standardise the features
X = np.c_[np.ones(1000), F]
sigmoid = lambda z: 1 / (1 + np.exp(-z))
w = np.zeros(3)
for epoch in range(1, 301):
    p = np.clip(sigmoid(X @ w), 1e-12, 1 - 1e-12)     # fix 2: never take log(0)
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))   # log loss
    if epoch in (1, 2, 10, 100, 300): print(f"epoch {epoch:3}: loss {loss:.3f}")
    w -= 0.1 * X.T @ (p - y) / len(y)                 # gradient step
print(f"accuracy {((sigmoid(X @ w) > 0.5) == y).mean():.3f}   weights {np.round(w, 2)}")

Output:

epoch   1: loss 0.693
epoch   2: loss 0.681
epoch  10: loss 0.605
epoch 100: loss 0.396
epoch 300: loss 0.353
accuracy 0.836   weights [-0.17  2.13 -1.21]

The weights have the directions built into the data: more income → more likely to repay (+2.13), more age → slightly less (−1.21).

Checklist when training goes wrong

Symptom Likely causes
Loss nan or inf unscaled features, learning rate too big, log(0), division by zero
Loss goes up every step gradient sign wrong (w += ... instead of w -= ...), learning rate too big
Loss stuck learning rate too small, gradient always zero, labels not matching rows (shuffled X but not y)
Any doubt overfit one tiny batch first (a correct model drives its loss near 0); gradient check against a numerical derivative (Episode 6)

Gradient of the log loss (to write on the whiteboard): ∂L/∂w = Xᵀ(p − y) / n.

Task 5: pandas questions

  • groupby: split rows into groups, compute something per group, combine the results.
  • idxmax: the row label of the largest value, so df.loc[df.groupby(g)[col].idxmax()] gives the top row per group. On a tie it keeps the first row: ask how ties should be handled.
  • agg: several summaries at once.
  • Chained assignment: df[mask]["col"] = v assigns into a temporary copy, not df. In pandas 3 (Copy-on-Write) it never works and pandas warns with ChainedAssignmentError. Use one .loc[rows, col] = v step.
import pandas as pd
df = pd.DataFrame({"customer": ["Asha", "Ravi", "Neha", "Kabir", "Priya", "Arjun"],   # made-up orders
                   "city": ["Pune", "Pune", "Delhi", "Delhi", "Delhi", "Pune"],
                   "spend": [1200, 3400, 900, 5200, 5200, 2100]})
top = df.loc[df.groupby("city")["spend"].idxmax()]          # Q: top spender in each city
print(top.to_string(index=False))
print(df.groupby("city")["spend"].agg(["count", "mean", "max"]).round(1))   # Q: count, mean, max per city
df[df.city == "Pune"]["spend"] = 0                           # trap: chained assignment
print("Pune spend after the trap:", df[df.city == "Pune"].spend.tolist())
df.loc[df.city == "Pune", "spend"] = 0                       # fix: one .loc step
print("Pune spend after .loc:  ", df[df.city == "Pune"].spend.tolist())

Output:

customer  city  spend
   Kabir Delhi   5200
    Ravi  Pune   3400
       count    mean   max
city                      
Delhi      3  3766.7  5200
Pune       3  2233.3  3400
Pune spend after the trap: [1200, 3400, 2100]
Pune spend after .loc:   [0, 0, 0]

Output (error):

main.py:8: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  df[df.city == "Pune"]["spend"] = 0                           # trap: chained assignment

Task 6: probability puzzles

Monty Hall (conditional probability). Three doors, one car. You pick a door; the host, who knows where the car is, opens another door with a goat. Switching wins exactly when your first pick was wrong: 2/3.

The disease test (Bayes' theorem). 1% are sick; the test catches 99% of the sick and wrongly flags 5% of the healthy. P(sick | positive) = P(pos | sick)·P(sick) / P(pos) = 0.99·0.01 / (0.99·0.01 + 0.05·0.99) = 1/6 ≈ 0.167. Healthy people are so many more that their false alarms outnumber the real cases: the base-rate fallacy is ignoring that.

Birthdays (complement rule). P(at least one shared birthday) = 1 − P(all different) = 1 − (365/365)(364/365)…(343/365).

import numpy as np
rng = np.random.default_rng(6)
n = 100000                                            # Monty Hall, played 100,000 times
car, pick = rng.integers(0, 3, n), rng.integers(0, 3, n)   # host then opens a goat door
print(f"Monty Hall: stay wins {np.mean(pick == car):.3f}   switch wins {np.mean(pick != car):.3f}")
# disease test: 1% of people are sick; test catches 99% of sick, wrongly flags 5% of healthy
p_sick, p_pos_sick, p_pos_healthy = 0.01, 0.99, 0.05
p_pos = p_pos_sick * p_sick + p_pos_healthy * (1 - p_sick)          # total probability
print(f"Bayes: P(sick | positive) = {p_pos_sick * p_sick / p_pos:.3f}")
sick = rng.random(n) < p_sick                                        # check by simulation
pos = np.where(sick, rng.random(n) < p_pos_sick, rng.random(n) < p_pos_healthy)
print(f"simulated: {sick[pos].mean():.3f}")
k = np.arange(23)                                     # birthday puzzle: 23 people
print(f"birthday: P(some two share a birthday) = {1 - np.prod((365 - k) / 365):.3f}")

Output:

Monty Hall: stay wins 0.333   switch wins 0.667
Bayes: P(sick | positive) = 0.167
simulated: 0.172
birthday: P(some two share a birthday) = 0.507

Task 7: statistics: "B's conversion is higher. Ship it?"

  • A/B test: show version A to some users and B to others; compare a rate (here: the share who buy, the conversion rate).
  • p-value: if there were no real difference, the chance of seeing a gap at least this big. It is not "the chance that B is no better".
  • Significance level 0.05: we accept a 5% false-alarm rate when nothing really changed.
  • Peeking: testing again and again as data arrives and stopping at the first p < 0.05. Each look is another chance to be lucky.
  • A/A test: both groups get the same version; any "significant" result is a false alarm.
import numpy as np
from scipy import stats
rng = np.random.default_rng(7)
a, b = rng.random(5000) < 0.100, rng.random(5000) < 0.115     # made-up A/B test: 10% vs 11.5% buy
t = stats.ttest_ind(b, a)                                   # is B's rate really higher?
print(f"A {a.mean():.3f}   B {b.mean():.3f}   p-value {t.pvalue:.4f}")
false_alarm_once, false_alarm_peek = 0, 0
for _ in range(1000):                                        # 1,000 A/A tests: NO real difference
    x, y = rng.random(5000) < 0.1, rng.random(5000) < 0.1
    false_alarm_once += stats.ttest_ind(x, y).pvalue < 0.05          # look once, at the end
    peeks = [stats.ttest_ind(x[:m], y[:m]).pvalue for m in range(500, 5001, 500)]
    false_alarm_peek += min(peeks) < 0.05                            # stop at the first p < 0.05
print(f"A/A tests, look once:     {false_alarm_once / 10:4.1f}% called significant")
print(f"A/A tests, peek 10 times: {false_alarm_peek / 10:4.1f}% called significant")

Output:

A 0.101   B 0.117   p-value 0.0093
A/A tests, look once:      4.3% called significant
A/A tests, peek 10 times: 20.4% called significant

(ttest_ind on 0/1 data is a standard large-sample way to compare two rates.) Peeking ten times took false alarms from 4.3% to 20.4%. Fix the sample size in advance, or use a method designed for repeated looks (sequential testing).

Complexity of everything we wrote

Algorithm Training Prediction Memory
kNN O(1) O(n·d) per query O(n·d), keeps all data
k-means O(n·k·d) per iteration O(k·d) per point O(n·d + k·d)
Logistic regression (batch GD) O(n·d) per epoch O(d), one dot product O(d) for weights
Linear regression, normal equation O(n·d² + d³) O(d) O(d²)

Common mistakes

Mistake What you see Why Fix
Python loops over rows around 100× slower Python executes every step vectorise with NumPy
(n, 1) - (n,) err (1000, 1000), MSE 27.245 with perfect weights broadcasting ravel() / reshape; print shapes
One k-means start inertia 1159.9 instead of ~424 local minimum several starts, k-means++
Raw income into the sigmoid RuntimeWarning: overflow encountered in exp, divide by zero encountered in log, loss nan huge feature values saturate the sigmoid standardise; clip p
w += lr * grad loss rises every epoch wrong sign w -= lr * grad
df[mask]["col"] = v ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment. assigns into a copy df.loc[mask, "col"] = v
Peeking at p-values false alarms 4.3% → 20.4% many chances to be lucky fix the sample size first
Reading p as P(B no better) wrong conclusion p assumes no difference state the definition exactly

Mini project: a from-scratch toolkit

Put knn_numpy, kmeans (with k-means++ and 5 restarts) and logistic regression (with standardisation, clipping and a fit/predict pair) into one file. Test each against scikit-learn's KNeighborsClassifier, KMeans and LogisticRegression on the same made-up data and print both results side by side; they should agree closely.

Practice

1. Make knn_numpy work in chunks of 100 test rows so memory stays O(100·n).

Solution
def knn_chunked(k=5, size=100):
    out = []
    for s in range(0, len(Xte), size):
        part = Xte[s:s + size]
        d = (part ** 2).sum(1)[:, None] + (Xtr ** 2).sum(1)[None, :] - 2 * part @ Xtr.T
        near = np.argpartition(d, k, axis=1)[:, :k]
        out.append((ytr[near].sum(1) > k / 2).astype(int))
    return np.concatenate(out)

Same predictions as knn_numpy, but only a 100 × n distance block exists at a time.

2. Add a k-means++ start to kmeans.

Solution
def kpp_init(X, k, r):
    C = [X[r.integers(len(X))]]
    for _ in range(k - 1):
        d2 = ((X[:, None, :] - np.array(C)[None]) ** 2).sum(2).min(1)   # distance to nearest chosen centre
        C.append(X[r.choice(len(X), p=d2 / d2.sum())])                  # far points are more likely
    return np.array(C)

Replace the random start line with C = kpp_init(X, k, r).

3. What does np.ones((4, 1)) + np.ones(3) give?

Solution

An array of shape (4, 3), all 2.0: broadcasting stretches the size-1 axis to 3 and adds the vector to every row.

4. A test catches 90% of sick people, wrongly flags 10% of healthy ones, and 1 in 1,000 people are sick. P(sick | positive)?

Solution

0.9 × 0.001 / (0.9 × 0.001 + 0.1 × 0.999) = 0.0009 / 0.1008 ≈ 0.0089, under 1%.

5. Monty Hall with 4 doors: the host opens 2 goat doors and you switch to the last closed door. Chance of winning?

Solution

You win by switching whenever your first pick was wrong: 3/4.

Quick quiz

a = np.ones((4, 1)); b = np.ones(3); print((a + b).shape) prints (4, 3): broadcasting stretches a's size-1 axis and adds b to every row, with no error.

Interview questions

  1. Implement kNN with NumPy only. What are its training and prediction complexities?
  2. How would you scale kNN to 10 million rows?
  3. Implement k-means. Why can it give different answers on different runs, and how do you fix that?
  4. Your logistic regression loss becomes nan. Walk me through how you would debug it.
  5. Derive the gradient of the log loss for logistic regression.
  6. What is broadcasting? Give an example where it causes a silent bug.
  7. Find the top row per group in pandas. What happens on ties?
  8. Why doesn't df[df.a > 0]["b"] = 0 change df?
  9. Solve Monty Hall and the disease-test puzzle; check them by simulation.
  10. Define a p-value. What is wrong with peeking at an A/B test?

Glossary

Term Meaning, with an example
Coding round An interview where you write and run code live.
From scratch Only NumPy, no ready-made model.
Time complexity How running time grows with input size: kNN prediction O(n·d).
Big O, O(...) Notation for that growth; constants ignored: O(n), O(n²).
Space complexity How memory grows: the m × n distance matrix (600,000 numbers).
k-nearest neighbours (kNN) Predict from the k most similar training rows: 5 most similar flats.
Lazy learner Does no work at training time, all at prediction (kNN).
Distance How different two rows are; squared Euclidean distance here.
argpartition NumPy function putting the k smallest values first without a full sort.
Vectorisation One array operation instead of a Python loop: around 100× faster here.
Approximate nearest neighbour (ANN) An index that finds near neighbours fast, slightly inexactly.
Shape Size of each axis: (1000,) vs (1000, 1).
Broadcasting Stretching size-1 axes so shapes match; (1000,1)−(1000,) → (1000,1000).
ravel / reshape Flatten to 1-D / change the shape: y.reshape(-1, 1).
k-means Clustering by repeating assign and update steps.
Centre (centroid) The mean point of a cluster.
Assign step Each point joins its nearest centre.
Update step Each centre moves to the mean of its points.
Converged Nothing changes between iterations.
Inertia Sum of squared distances to the assigned centres: 423.9 vs 1159.9.
Local minimum A solution no small change improves, but not the best overall.
k-means++ A start that spreads the first centres apart.
n_init scikit-learn: number of k-means restarts; the best is kept.
Empty cluster A centre that no point joins.
Sigmoid 1/(1+e⁻ᶻ), maps any number into (0, 1).
Log loss Loss of logistic regression; 0.693 = guessing 50/50.
nan "Not a number", from impossible maths such as 0 × ∞.
Overflow A number too large for the float type: overflow encountered in exp.
Saturation The sigmoid stuck at exactly 0 or 1 for huge inputs.
Clipping Forcing values into a range: p into [1e-12, 1−1e-12].
Standardisation (x − mean) / standard deviation.
Gradient check Comparing an analytic gradient with a numerical one.
groupby Split rows into groups, compute per group, combine.
idxmax Row label of the maximum; keeps the first on a tie.
agg Several summaries at once: count, mean, max.
Chained assignment df[mask]["col"] = v: sets a copy, never the original in pandas 3.
Copy-on-Write pandas 3 rule: every derived table behaves as a copy until written.
.loc Label-based selection and assignment in one step.
Conditional probability P(A | B): chance of A given B happened.
Monty Hall problem Switching doors wins 2/3 (0.667 simulated).
Bayes' theorem P(A | B) = P(B | A)·P(A)/P(B): P(sick | positive) = 0.167.
Base rate How common something is before any evidence: 1% sick.
Base-rate fallacy Ignoring the base rate and trusting "99% accurate".
Complement rule P(at least one) = 1 − P(none): birthdays 0.507.
Simulation Estimating a probability by repeating random trials.
A/B test Comparing two versions on random user groups.
A/A test Both groups see the same version; checks the false-alarm rate.
Conversion rate Share of users who do the goal action: 0.101 vs 0.117.
p-value Chance of a gap this big if there were no real difference: 0.0093.
Significance level The false-alarm rate we accept, usually 0.05.
Peeking Stopping a test at the first p < 0.05: 20.4% false alarms.
Sequential testing Test methods designed for repeated looks at the data.
Normal equation Closed-form linear regression w = (XᵀX)⁻¹Xᵀy, O(n·d² + d³).

Summary

  • Restate, write the simple version, test it, then vectorise and state complexity.
  • kNN: training O(1), prediction O(n·d); vectorised distances are around 100× faster.
  • Print shapes: broadcasting turns a column minus a vector into a silent n × n bug.
  • k-means finds local minima: restart several times and use k-means++.
  • nan losses come from unscaled features, big learning rates and log(0): standardise and clip.
  • pandas: groupby + idxmax (watch ties), and .loc instead of chained assignment.
  • Probability: Monty Hall 2/3, Bayes 0.167, birthdays 0.507; check by simulation.
  • A p-value assumes no difference; peeking multiplies false alarms.

Next episode: Interview prep 3: ML system design.