Machine Learning Full Course · Episode 23. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. All data is made up for teaching.
The big idea: scikit-learn's LogisticRegression is Episode 22's model with an L2 penalty switched on. Its weights live in log-odds, so read them as odds ratios (e^w), not as changes in probability. C is the inverse of the penalty strength. For more than two classes, one-vs-rest trains one yes/no model per class, but their probabilities don't agree; softmax turns one score per class into probabilities that add up to 1 (multinomial logistic regression, trained with categorical cross-entropy). When one class is rare, class weights make its mistakes cost more.
The problem: who will leave the mobile plan?
A mobile network has 1,000 (made-up) customers: months with the network, number of complaints, and monthly data_gb. Column left is 1 if the customer switched to another network. Customers leaving is called churn; predicting it is one of the most common classification jobs in business. The manager wants predictions and reasons: how much does one complaint matter?
LogisticRegression and its weights
LogisticRegression follows the scikit-learn API (Episode 17): fit(X, y), predict(X) (classes), predict_proba(X) (one probability column per class, in the order of classes_; column 1 is P(left)), coef_ (weights) and intercept_ (bias). max_iter is the number of optimiser iterations allowed.
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000 # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
"data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
m = LogisticRegression(max_iter=1000).fit(X, y)
for name, w in zip(X.columns, m.coef_[0]):
print(f"{name:>10}: weight {w:+.3f} odds ratio {np.exp(w):.3f}")
same = pd.DataFrame({"months": 24, "complaints": range(5), "data_gb": 20}) # one customer, 0-4 complaints
p = m.predict_proba(same)[:, 1] # column 1 = P(left)
o = p / (1 - p) # odds
for c in range(5): print(f"{c} complaints: p {p[c]:.3f} odds {o[c]:.3f} x{o[c] / o[c - 1] if c else 1:.3f}")
Output:
months: weight -0.048 odds ratio 0.953
complaints: weight +0.748 odds ratio 2.113
data_gb: weight -0.019 odds ratio 0.981
0 complaints: p 0.241 odds 0.318 x1.000
1 complaints: p 0.402 odds 0.672 x2.113
2 complaints: p 0.587 odds 1.419 x2.113
3 complaints: p 0.750 odds 2.999 x2.113
4 complaints: p 0.864 odds 6.336 x2.113
The obvious (wrong) reading: "each complaint adds 0.748 to the probability of leaving". That would push a customer with two complaints past 100%. The output shows the probability jumps are uneven (+0.161, +0.185, +0.163, +0.114), but the odds are multiplied by exactly 2.113 every time, the number printed as the odds ratio.
Coefficients as odds ratios
Logistic regression is a straight line in log-odds (Episode 22):
ln(odds) = w1·months + w2·complaints + w3·data_gb + b
- +1 complaint → log-odds + 0.748 → odds × e^0.748 = × 2.113.
- Odds ratio = e^w, the factor by which the odds change for a one-unit increase of the feature (others held fixed).
- odds ratio > 1: raises the odds (complaints 2.113: each complaint more than doubles the odds of leaving)
- odds ratio < 1: lowers them (months 0.953: each month with the network cuts the odds by 4.7%)
- odds ratio = 1 (w = 0): no effect
- The change in probability depends on where you start, because the S-curve is steep in the middle and flat at the ends.
- Odds ratios are per unit (per complaint, per month, per GB). To compare features with each other, standardise first (mini project below).
C: the inverse regularisation strength
Real-life example: a strict teacher keeps every answer short and simple (small C: strong penalty, small weights); a lenient teacher lets anything through (large C: weak penalty).
LogisticRegressionadds an L2 penalty (ridge's penalty, Episode 21) by default.- C = 1 / regularisation strength: the opposite of Episode 21's α. Small C = more regularisation. Default C = 1.
C=np.infmeans no penalty (we used it in Episode 22).l1_ratio=1gives an L1 (lasso-style) penalty instead (with a solver that supports it, e.g.solver="saga").
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000 # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
"data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
for C in [0.0001, 0.01, 1, 100]: # C = 1 / regularisation strength
m = make_pipeline(StandardScaler(), LogisticRegression(C=C)).fit(X_tr, y_tr)
print(f"C {C:<6} weights {m[-1].coef_[0].round(3)} test accuracy {m.score(X_te, y_te):.3f}")
Output:
C 0.0001 weights [-0.01 0.011 -0.004] test accuracy 0.687
C 0.01 weights [-0.417 0.445 -0.151] test accuracy 0.730
C 1 weights [-0.827 0.867 -0.313] test accuracy 0.723
C 100 weights [-0.837 0.878 -0.317] test accuracy 0.723
- C = 0.0001 crushes every weight to about 0 and the test accuracy, 0.687, equals the share of test customers who stayed: the model predicts "stays" for everyone (underfitting).
- C = 0.01 is best here (0.730); above C = 1 there is barely any penalty left and the weights stop changing.
- Choose C on validation data, like α (cross-validation, Episode 26).
More than two classes: one-vs-rest
A café (made up) has 600 orders: temperature (°C), hour of the day, and the drink: 0 chai, 1 coffee, 2 cold drink. Predicting one of more than two classes is multiclass classification.
The obvious idea: three yes/no models, "chai or not", "coffee or not", "cold drink or not". y == k is True for drink k and False for the rest:
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600 # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)] # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P]) # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]] # 30 °C at 3 pm
total = 0
for k in range(3): # one yes/no model per drink
pk = LogisticRegression().fit(X, y == k).predict_proba(cust)[0, 1]
print(f"model '{names[k]} or not': p = {pk:.3f}"); total += pk
print(f"sum of the three = {total:.3f}")
Output:
model 'chai or not': p = 0.072
model 'coffee or not': p = 0.308
model 'cold drink or not': p = 0.383
sum of the three = 0.763
The customer will certainly order one of the three drinks, so the probabilities must add up to 1, but they sum to 0.763. Each model was trained alone and doesn't know the others exist.
This scheme is one-vs-rest (OvR), also called one-vs-all:
- K classes → K binary models; predict the class whose model is most confident.
- It works with any binary classifier. scikit-learn's wrapper
OneVsRestClassifierrescales the K outputs so they sum to 1 (practice 3). - The raw probabilities are not coordinated. Better: one model that scores all classes together.
Softmax
Real-life example: friends sharing one pizza. Each shouts how hungry they are (a score for each class, any number, even negative). Hungrier friends get much bigger slices and nobody gets a negative slice (exponentiate: e^z is always positive and grows fast). The slices must make exactly one pizza (divide by the sum).
softmax(z)_k = e^(z_k) / (e^(z_1) + e^(z_2) + ... + e^(z_K))
import numpy as np
def softmax(z): return np.exp(z) / np.exp(z).sum()
z = np.array([2.0, 1.0, -1.0]) # scores for chai, coffee, cold drink
print("scores", z, "-> softmax", softmax(z).round(3), "sum", softmax(z).sum())
big = np.array([1000.0, 999.0, 990.0])
print("big scores ->", softmax(big))
def softmax_safe(z): e = np.exp(z - z.max()); return e / e.sum() # subtract the max first
print("safe ->", softmax_safe(big).round(3), "| small:", softmax_safe(z).round(3))
Output:
scores [ 2. 1. -1.] -> softmax [0.705 0.259 0.035] sum 1.0
big scores -> [nan nan nan]
safe -> [0.731 0.269 0. ] | small: [0.705 0.259 0.035]
main.py:2: RuntimeWarning: overflow encountered in exp
def softmax(z): return np.exp(z) / np.exp(z).sum()
main.py:2: RuntimeWarning: invalid value encountered in divide
def softmax(z): return np.exp(z) / np.exp(z).sum()
- Scores [2, 1, −1] → [0.705, 0.259, 0.035], summing to 1.
- Scores [1000, 999, 990]: e^1000 is too big for a 64-bit float, becomes
inf(RuntimeWarning: overflow encountered in exp), and inf / inf =nan(invalid value encountered in divide). - Fix: subtract the largest score first. Softmax only depends on the differences between scores, so the answer is unchanged, and the biggest exponent becomes e^0 = 1. The small scores give exactly the same answer as before.
Multinomial logistic regression and categorical cross-entropy
- One weight vector and bias per class: z_k = w_k · x + b_k. scikit-learn stores them in
coef_with shape (K classes, number of features). - Softmax turns the K scores into probabilities: this is multinomial logistic regression (also softmax regression). "Multinomial" = many categories.
- With K = 2, softmax reduces to the sigmoid of the score difference: the same model as Episode 22.
- Loss: categorical cross-entropy (multiclass log loss) = the average of −ln(probability given to the true class). If the customer ordered coffee, only the coffee probability counts. It is binary cross-entropy extended to K classes.
LogisticRegressionfits the multinomial model automatically when y has 3 or more classes.
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600 # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)] # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P]) # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]] # 30 °C at 3 pm
from sklearn.metrics import log_loss
m = LogisticRegression().fit(X, y) # 3 classes -> softmax (multinomial)
p = m.predict_proba(cust)[0]
print("coef_ shape", m.coef_.shape) # one row of weights per class
print("30 °C, 3 pm:", " ".join(f"{k} {v:.3f}" for k, v in zip(names, p)), f" sum {p.sum():.3f}")
P = m.predict_proba(X)
cce = -np.mean(np.log(P[np.arange(n), y])) # -log(probability of the TRUE class)
print(f"categorical cross-entropy by hand {cce:.4f} | log_loss {log_loss(y, P):.4f}")
print(f"accuracy {m.score(X, y):.3f}")
Output:
coef_ shape (3, 2)
30 °C, 3 pm: chai 0.106 coffee 0.459 cold drink 0.435 sum 1.000
categorical cross-entropy by hand 0.5815 | log_loss 0.5815
accuracy 0.745
P[np.arange(n), y] picks, for each row, the probability in the column of its true class (NumPy "fancy indexing"). Our hand-computed loss equals sklearn.metrics.log_loss.
Class weights for rare classes
A factory checks 4,000 pressure cookers (made up) with two sensor readings; only 4% are faulty. When one class is much rarer than another the data is imbalanced. 96% of the loss comes from good cookers, so the model learns mostly about them.
stratify=y keeps 4% faulty in both halves of the split (Episode 15).
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(3); n = 4000 # made-up factory: 2 sensors per cooker, 1 = faulty
X = rng.normal(0, 1, (n, 2))
y = (rng.random(n) < 1 / (1 + np.exp(4.5 - 1.5 * X[:, 0] - X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
print(f"faulty: {y.mean():.1%} of cookers")
for cw in [None, "balanced"]:
m = LogisticRegression(class_weight=cw).fit(X_tr, y_tr); pred = m.predict(X_te)
caught, alarms = np.sum(pred[y_te == 1] == 1), np.sum(pred[y_te == 0] == 1)
print(f"{cw!s:<8} caught {caught}/{y_te.sum()} false alarms {alarms:3} accuracy {m.score(X_te, y_te):.3f}")
Output:
faulty: 4.0% of cookers
None caught 6/80 false alarms 5 accuracy 0.961
balanced caught 64/80 false alarms 402 accuracy 0.791
- Plain model: accuracy 0.961, but only 6 of 80 faulty cookers caught.
class_weight="balanced": 64 of 80 caught, at the price of 402 false alarms (good cookers sent for re-checks); accuracy falls to 0.791 although the model is far more useful here. Accuracy hides the problem; Episode 24 introduces better metrics.
How class weights work:
loss = Σ weight[class of example i] × loss_i
balanced: weight_c = n / (K × n_c) n = examples, K = classes, n_c = examples in class c
For 4% faulty: faulty ≈ 4000 / (2 × 160) = 12.5, good ≈ 4000 / (2 × 3840) ≈ 0.52 (computed on the training half by scikit-learn). You can also pass your own: class_weight={0: 1, 1: 10}.
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| reading a weight as a probability change | "+0.748 per complaint" would pass 100% | weights are in log-odds | odds ratio e^w |
| treating C like α | C = 100 barely changes anything; C = 0.0001 predicts one class | C = 1 / strength | small C = strong penalty |
| adding up one-vs-rest probabilities | 0.072 + 0.308 + 0.383 = 0.763 | models trained separately | multinomial (softmax), or OneVsRestClassifier which normalises |
| softmax without subtracting the max | RuntimeWarning: overflow encountered in exp, result [nan nan nan] |
e^1000 = inf | np.exp(z - z.max()) |
| trusting accuracy on a rare class | accuracy 0.961 while catching 6 of 80 | the majority class dominates | class weights, better metrics (Episode 24) |
| comparing raw odds ratios across features | months 0.953 vs complaints 2.113 | different units | standardise, then compare (mini project) |
Mini project: which feature matters most?
Odds ratios per one standard deviation (standardised features) put every feature on the same scale:
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000 # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
"data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
m = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
sd = X.std(ddof=0)
print("feature 1 SD odds ratio per 1 SD")
for name, w in sorted(zip(X.columns, m[-1].coef_[0]), key=lambda t: -abs(t[1])):
print(f"{name:>10} {sd[name]:7.2f} {np.exp(w):.3f}")
Output:
feature 1 SD odds ratio per 1 SD
months 17.05 0.444
complaints 1.08 2.237
data_gb 16.28 0.735
Per typical change, a customer's tenure (17 months ↔ odds × 0.444) matters about as much as complaints (1 more ↔ × 2.237, in the other direction), and data usage matters least.
Practice
- Explain the odds ratio of
data_gbin your own words, per 1 GB and per 10 GB. - Print the validation accuracy for 20 values of C between 0.0001 and 100.
- Compare
OneVsRestClassifier(LogisticRegression())with the multinomial model on the café data. - Try
class_weight={0: 1, 1: 5}on the cookers. How many are caught, and how many false alarms?
Solution 1
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000 # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
"data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
m = LogisticRegression(max_iter=1000).fit(X, y)
w = m.coef_[0][2] # data_gb
print(f"per 1 GB: odds x {np.exp(w):.3f} ({(np.exp(w) - 1):+.1%})")
print(f"per 10 GB: odds x {np.exp(10 * w):.3f} ({(np.exp(10 * w) - 1):+.1%})")
Output:
per 1 GB: odds x 0.981 (-1.9%)
per 10 GB: odds x 0.827 (-17.3%)
Each extra GB of data per month multiplies the odds of leaving by 0.981 (1.9% lower); 10 more GB multiply them by 0.981¹⁰ ≈ 0.827 (17.3% lower). Heavy users are a bit less likely to leave, holding months and complaints fixed.
Solution 2
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000 # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
"data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int) # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.3, random_state=0)
for C in np.logspace(-4, 2, 20):
m = make_pipeline(StandardScaler(), LogisticRegression(C=C)).fit(X_tr, y_tr)
print(f"C {C:9.4f} accuracy {m.score(X_va, y_va):.3f}")
Output:
C 0.0001 accuracy 0.687
C 0.0002 accuracy 0.687
C 0.0004 accuracy 0.687
C 0.0009 accuracy 0.687
C 0.0018 accuracy 0.700
C 0.0038 accuracy 0.727
C 0.0078 accuracy 0.740
C 0.0162 accuracy 0.733
C 0.0336 accuracy 0.730
C 0.0695 accuracy 0.733
C 0.1438 accuracy 0.727
C 0.2976 accuracy 0.727
C 0.6158 accuracy 0.723
C 1.2743 accuracy 0.723
C 2.6367 accuracy 0.723
C 5.4556 accuracy 0.723
C 11.2884 accuracy 0.723
C 23.3572 accuracy 0.723
C 48.3293 accuracy 0.723
C 100.0000 accuracy 0.723
Very small C predicts one class (0.687); the best region is around C ≈ 0.008; above C ≈ 1 nothing changes.
Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600 # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)] # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P]) # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]] # 30 °C at 3 pm
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import log_loss
ovr = OneVsRestClassifier(LogisticRegression()).fit(X, y)
multi = LogisticRegression().fit(X, y)
for name, m in [("one-vs-rest", ovr), ("multinomial", multi)]:
p = m.predict_proba(cust)[0]
print(f"{name}: {p.round(3)} sum {p.sum():.3f} accuracy {m.score(X, y):.3f} log loss {log_loss(y, m.predict_proba(X)):.4f}")
Output:
one-vs-rest: [0.095 0.403 0.502] sum 1.000 accuracy 0.730 log loss 0.6014
multinomial: [0.106 0.459 0.435] sum 1.000 accuracy 0.745 log loss 0.5815
OneVsRestClassifier normalises its probabilities to sum to 1, but they still come from three separate models: for this customer it even picks a different drink (cold drink, 0.502) than the multinomial model (coffee, 0.459). Multinomial has the better accuracy and log loss here.
Solution 4
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(3); n = 4000 # made-up factory: 2 sensors per cooker, 1 = faulty
X = rng.normal(0, 1, (n, 2))
y = (rng.random(n) < 1 / (1 + np.exp(4.5 - 1.5 * X[:, 0] - X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
print(f"faulty: {y.mean():.1%} of cookers")
for cw in [None, {0: 1, 1: 5}, "balanced"]:
m = LogisticRegression(class_weight=cw).fit(X_tr, y_tr); pred = m.predict(X_te)
caught, alarms = np.sum(pred[y_te == 1] == 1), np.sum(pred[y_te == 0] == 1)
print(f"{str(cw):<16} caught {caught}/{y_te.sum()} false alarms {alarms:3} accuracy {m.score(X_te, y_te):.3f}")
Output:
faulty: 4.0% of cookers
None caught 6/80 false alarms 5 accuracy 0.961
{0: 1, 1: 5} caught 30/80 false alarms 83 accuracy 0.933
balanced caught 64/80 false alarms 402 accuracy 0.791
Weight 5 sits in between: 30 caught with 83 false alarms. The weight is a dial between misses and false alarms.
Quick quiz
churn model, feature "late_payments": weight = 0.693 (e^0.693 ≈ 2.0)
one more late payment ...
A) doubles the odds of leaving B) adds 0.693 to the probability C) doubles the probability D) adds 69.3% to the odds
Answer
A. The weight adds 0.693 to the log-odds, so the odds are multiplied by e^0.693 ≈ 2. The probability can't double in general: a customer at 0.9 can't go to 1.8.
Interview questions
- How do you interpret a logistic regression coefficient? A one-unit increase adds w to the log-odds, i.e. multiplies the odds by e^w (the odds ratio), holding other features fixed.
- What is C in scikit-learn's LogisticRegression? The inverse of the regularisation strength; smaller C = stronger penalty. Default 1, L2 penalty.
- One-vs-rest vs multinomial? OvR trains K independent binary models (works with any binary classifier, probabilities not jointly calibrated); multinomial trains one model with softmax and categorical cross-entropy, giving coherent probabilities.
- What is softmax and how do you compute it safely? e^(z_k) / Σ e^(z_j); subtract max(z) first to avoid overflow (log-sum-exp trick).
- Relation between softmax and sigmoid? For two classes, softmax of (z₁, z₂) gives σ(z₁ − z₂) for class 1.
- What is categorical cross-entropy? The mean of −ln(predicted probability of the true class); for K = 2 it equals binary cross-entropy.
- How do you handle imbalanced classes in logistic regression? Class weights (
class_weight="balanced"or a dict), resampling, choosing the threshold, and judging with recall/precision rather than accuracy. - Why standardise before comparing coefficients? Raw coefficients are per unit of each feature; standardised coefficients are per standard deviation and comparable. Standardising also makes the penalty fair (Episode 21).
- What does
coef_look like for 3 classes and 2 features? Shape (3, 2): one row of weights per class.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Churn | Customers leaving a service | left = 1 |
LogisticRegression |
scikit-learn's logistic regression (L2 penalty by default) | LogisticRegression(C=1) |
predict_proba |
Probability of each class, one column per class | column 1 = P(left) |
coef_, intercept_ |
Learned weights and bias | coef_ shape (3, 2) for 3 classes |
max_iter |
Maximum optimiser iterations | max_iter=1000 |
| Odds ratio | e^w: how the odds are multiplied per unit increase | complaints 2.113 |
| Log-odds | ln(p / (1 − p)), linear in the features (Episode 22) | +0.748 per complaint |
| C | Inverse regularisation strength, C = 1 / strength | small C = strong penalty |
| L2 penalty | Sum of squared weights (Episode 21) | default in LogisticRegression |
| Underfitting | Model too simple to capture the pattern (Episode 20) | C = 0.0001: everyone "stays" |
| Multiclass classification | Predicting one of 3 or more classes | chai / coffee / cold drink |
| One-vs-rest (OvR) | K binary models, one per class against the rest | "coffee or not" |
OneVsRestClassifier |
scikit-learn wrapper for OvR | OneVsRestClassifier(LogisticRegression()) |
| Score (logit) per class | z_k = w_k · x + b_k | chai 2, coffee 1, cold drink −1 |
| Softmax | e^(z_k) / Σ e^(z_j): scores → probabilities summing to 1 | [2, 1, −1] → [0.705, 0.259, 0.035] |
| Overflow | A number too large for the float type → inf | e^1000 |
| Multinomial logistic regression | Logistic regression with softmax for K classes (softmax regression) | default for 3+ classes |
| Categorical cross-entropy | Mean of −ln(p of the true class) | 0.5815 on the café data |
log_loss |
scikit-learn function for (categorical) cross-entropy | log_loss(y, P) |
| Imbalanced data | One class much rarer than another | 4% faulty |
| Class weight | Multiplier on each example's loss by class | class_weight="balanced" |
| Balanced weights | n / (K × n_c) | faulty ≈ 12.5, good ≈ 0.52 |
| Stratified split | Split keeping class proportions (Episode 15) | stratify=y |
| False alarm | Predicting the rare class when it isn't (a false positive, Episode 24) | 402 good cookers flagged |
Summary
LogisticRegression= Episode 22's model + an L2 penalty; read weights as odds ratios e^w (complaints: × 2.113 odds per complaint), not probability changes.- C = 1 / strength: C = 0.0001 predicted "stays" for everyone (0.687); C = 0.01 was best.
- One-vs-rest probabilities didn't add up (0.763); softmax fixes that; subtract the max to avoid overflow.
- Multinomial logistic regression gives coherent probabilities (sum 1.000) and is trained with categorical cross-entropy (0.5815 by hand =
log_loss). - Class weights caught 64 of 80 faulty cookers instead of 6, while accuracy fell from 0.961 to 0.791: accuracy can lie.
Next episode: classification metrics: the confusion matrix, precision, recall, specificity, F1, F-beta, and macro, micro and weighted averages.