SyntaxPilot
Courses/Machine Learning/Lesson 23

Machine Learning · Lesson 23 of 90 · 13 min

Logistic regression in practice and multiclass

LogisticRegression in scikit-learnC (inverse regularisation)coefficients as odds ratiosmulticlass classificationone-vs-restsoftmaxmultinomial logistic regressioncategorical cross-entropyclass weights

Machine Learning Full Course · Episode 23. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. All data is made up for teaching.

The big idea: scikit-learn's LogisticRegression is Episode 22's model with an L2 penalty switched on. Its weights live in log-odds, so read them as odds ratios (e^w), not as changes in probability. C is the inverse of the penalty strength. For more than two classes, one-vs-rest trains one yes/no model per class, but their probabilities don't agree; softmax turns one score per class into probabilities that add up to 1 (multinomial logistic regression, trained with categorical cross-entropy). When one class is rare, class weights make its mistakes cost more.

The problem: who will leave the mobile plan?

A mobile network has 1,000 (made-up) customers: months with the network, number of complaints, and monthly data_gb. Column left is 1 if the customer switched to another network. Customers leaving is called churn; predicting it is one of the most common classification jobs in business. The manager wants predictions and reasons: how much does one complaint matter?

LogisticRegression and its weights

LogisticRegression follows the scikit-learn API (Episode 17): fit(X, y), predict(X) (classes), predict_proba(X) (one probability column per class, in the order of classes_; column 1 is P(left)), coef_ (weights) and intercept_ (bias). max_iter is the number of optimiser iterations allowed.

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000            # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
                   "data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)   # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
m = LogisticRegression(max_iter=1000).fit(X, y)
for name, w in zip(X.columns, m.coef_[0]):
    print(f"{name:>10}: weight {w:+.3f}   odds ratio {np.exp(w):.3f}")
same = pd.DataFrame({"months": 24, "complaints": range(5), "data_gb": 20})  # one customer, 0-4 complaints
p = m.predict_proba(same)[:, 1]                     # column 1 = P(left)
o = p / (1 - p)                                      # odds
for c in range(5): print(f"{c} complaints: p {p[c]:.3f}  odds {o[c]:.3f}  x{o[c] / o[c - 1] if c else 1:.3f}")

Output:

    months: weight -0.048   odds ratio 0.953
complaints: weight +0.748   odds ratio 2.113
   data_gb: weight -0.019   odds ratio 0.981
0 complaints: p 0.241  odds 0.318  x1.000
1 complaints: p 0.402  odds 0.672  x2.113
2 complaints: p 0.587  odds 1.419  x2.113
3 complaints: p 0.750  odds 2.999  x2.113
4 complaints: p 0.864  odds 6.336  x2.113

The obvious (wrong) reading: "each complaint adds 0.748 to the probability of leaving". That would push a customer with two complaints past 100%. The output shows the probability jumps are uneven (+0.161, +0.185, +0.163, +0.114), but the odds are multiplied by exactly 2.113 every time, the number printed as the odds ratio.

Coefficients as odds ratios

Logistic regression is a straight line in log-odds (Episode 22):

ln(odds) = w1·months + w2·complaints + w3·data_gb + b
  • +1 complaint → log-odds + 0.748 → odds × e^0.748 = × 2.113.
  • Odds ratio = e^w, the factor by which the odds change for a one-unit increase of the feature (others held fixed).
  • odds ratio > 1: raises the odds (complaints 2.113: each complaint more than doubles the odds of leaving)
  • odds ratio < 1: lowers them (months 0.953: each month with the network cuts the odds by 4.7%)
  • odds ratio = 1 (w = 0): no effect
  • The change in probability depends on where you start, because the S-curve is steep in the middle and flat at the ends.
  • Odds ratios are per unit (per complaint, per month, per GB). To compare features with each other, standardise first (mini project below).

C: the inverse regularisation strength

Real-life example: a strict teacher keeps every answer short and simple (small C: strong penalty, small weights); a lenient teacher lets anything through (large C: weak penalty).

  • LogisticRegression adds an L2 penalty (ridge's penalty, Episode 21) by default.
  • C = 1 / regularisation strength: the opposite of Episode 21's α. Small C = more regularisation. Default C = 1.
  • C=np.inf means no penalty (we used it in Episode 22). l1_ratio=1 gives an L1 (lasso-style) penalty instead (with a solver that supports it, e.g. solver="saga").
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000            # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
                   "data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)   # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
for C in [0.0001, 0.01, 1, 100]:                     # C = 1 / regularisation strength
    m = make_pipeline(StandardScaler(), LogisticRegression(C=C)).fit(X_tr, y_tr)
    print(f"C {C:<6}  weights {m[-1].coef_[0].round(3)}  test accuracy {m.score(X_te, y_te):.3f}")

Output:

C 0.0001  weights [-0.01   0.011 -0.004]  test accuracy 0.687
C 0.01    weights [-0.417  0.445 -0.151]  test accuracy 0.730
C 1       weights [-0.827  0.867 -0.313]  test accuracy 0.723
C 100     weights [-0.837  0.878 -0.317]  test accuracy 0.723
  • C = 0.0001 crushes every weight to about 0 and the test accuracy, 0.687, equals the share of test customers who stayed: the model predicts "stays" for everyone (underfitting).
  • C = 0.01 is best here (0.730); above C = 1 there is barely any penalty left and the weights stop changing.
  • Choose C on validation data, like α (cross-validation, Episode 26).

More than two classes: one-vs-rest

A café (made up) has 600 orders: temperature (°C), hour of the day, and the drink: 0 chai, 1 coffee, 2 cold drink. Predicting one of more than two classes is multiclass classification.

The obvious idea: three yes/no models, "chai or not", "coffee or not", "cold drink or not". y == k is True for drink k and False for the rest:

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600              # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)]          # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])  # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]]       # 30 °C at 3 pm
total = 0
for k in range(3):                                   # one yes/no model per drink
    pk = LogisticRegression().fit(X, y == k).predict_proba(cust)[0, 1]
    print(f"model '{names[k]} or not': p = {pk:.3f}"); total += pk
print(f"sum of the three = {total:.3f}")

Output:

model 'chai or not': p = 0.072
model 'coffee or not': p = 0.308
model 'cold drink or not': p = 0.383
sum of the three = 0.763

The customer will certainly order one of the three drinks, so the probabilities must add up to 1, but they sum to 0.763. Each model was trained alone and doesn't know the others exist.

This scheme is one-vs-rest (OvR), also called one-vs-all:

  • K classes → K binary models; predict the class whose model is most confident.
  • It works with any binary classifier. scikit-learn's wrapper OneVsRestClassifier rescales the K outputs so they sum to 1 (practice 3).
  • The raw probabilities are not coordinated. Better: one model that scores all classes together.

Softmax

Real-life example: friends sharing one pizza. Each shouts how hungry they are (a score for each class, any number, even negative). Hungrier friends get much bigger slices and nobody gets a negative slice (exponentiate: e^z is always positive and grows fast). The slices must make exactly one pizza (divide by the sum).

softmax(z)_k = e^(z_k) / (e^(z_1) + e^(z_2) + ... + e^(z_K))
import numpy as np
def softmax(z): return np.exp(z) / np.exp(z).sum()
z = np.array([2.0, 1.0, -1.0])                       # scores for chai, coffee, cold drink
print("scores", z, "-> softmax", softmax(z).round(3), "sum", softmax(z).sum())
big = np.array([1000.0, 999.0, 990.0])
print("big scores ->", softmax(big))
def softmax_safe(z): e = np.exp(z - z.max()); return e / e.sum()   # subtract the max first
print("safe       ->", softmax_safe(big).round(3), "| small:", softmax_safe(z).round(3))

Output:

scores [ 2.  1. -1.] -> softmax [0.705 0.259 0.035] sum 1.0
big scores -> [nan nan nan]
safe       -> [0.731 0.269 0.   ] | small: [0.705 0.259 0.035]
main.py:2: RuntimeWarning: overflow encountered in exp
  def softmax(z): return np.exp(z) / np.exp(z).sum()
main.py:2: RuntimeWarning: invalid value encountered in divide
  def softmax(z): return np.exp(z) / np.exp(z).sum()
  • Scores [2, 1, −1] → [0.705, 0.259, 0.035], summing to 1.
  • Scores [1000, 999, 990]: e^1000 is too big for a 64-bit float, becomes inf (RuntimeWarning: overflow encountered in exp), and inf / inf = nan (invalid value encountered in divide).
  • Fix: subtract the largest score first. Softmax only depends on the differences between scores, so the answer is unchanged, and the biggest exponent becomes e^0 = 1. The small scores give exactly the same answer as before.

Multinomial logistic regression and categorical cross-entropy

  • One weight vector and bias per class: z_k = w_k · x + b_k. scikit-learn stores them in coef_ with shape (K classes, number of features).
  • Softmax turns the K scores into probabilities: this is multinomial logistic regression (also softmax regression). "Multinomial" = many categories.
  • With K = 2, softmax reduces to the sigmoid of the score difference: the same model as Episode 22.
  • Loss: categorical cross-entropy (multiclass log loss) = the average of −ln(probability given to the true class). If the customer ordered coffee, only the coffee probability counts. It is binary cross-entropy extended to K classes.
  • LogisticRegression fits the multinomial model automatically when y has 3 or more classes.
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600              # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)]          # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])  # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]]       # 30 °C at 3 pm
from sklearn.metrics import log_loss
m = LogisticRegression().fit(X, y)                   # 3 classes -> softmax (multinomial)
p = m.predict_proba(cust)[0]
print("coef_ shape", m.coef_.shape)                 # one row of weights per class
print("30 °C, 3 pm:", "  ".join(f"{k} {v:.3f}" for k, v in zip(names, p)), f"  sum {p.sum():.3f}")
P = m.predict_proba(X)
cce = -np.mean(np.log(P[np.arange(n), y]))          # -log(probability of the TRUE class)
print(f"categorical cross-entropy by hand {cce:.4f} | log_loss {log_loss(y, P):.4f}")
print(f"accuracy {m.score(X, y):.3f}")

Output:

coef_ shape (3, 2)
30 °C, 3 pm: chai 0.106  coffee 0.459  cold drink 0.435   sum 1.000
categorical cross-entropy by hand 0.5815 | log_loss 0.5815
accuracy 0.745

P[np.arange(n), y] picks, for each row, the probability in the column of its true class (NumPy "fancy indexing"). Our hand-computed loss equals sklearn.metrics.log_loss.

Class weights for rare classes

A factory checks 4,000 pressure cookers (made up) with two sensor readings; only 4% are faulty. When one class is much rarer than another the data is imbalanced. 96% of the loss comes from good cookers, so the model learns mostly about them.

stratify=y keeps 4% faulty in both halves of the split (Episode 15).

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(3); n = 4000             # made-up factory: 2 sensors per cooker, 1 = faulty
X = rng.normal(0, 1, (n, 2))
y = (rng.random(n) < 1 / (1 + np.exp(4.5 - 1.5 * X[:, 0] - X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
print(f"faulty: {y.mean():.1%} of cookers")
for cw in [None, "balanced"]:
    m = LogisticRegression(class_weight=cw).fit(X_tr, y_tr); pred = m.predict(X_te)
    caught, alarms = np.sum(pred[y_te == 1] == 1), np.sum(pred[y_te == 0] == 1)
    print(f"{cw!s:<8}  caught {caught}/{y_te.sum()}  false alarms {alarms:3}  accuracy {m.score(X_te, y_te):.3f}")

Output:

faulty: 4.0% of cookers
None      caught 6/80  false alarms   5  accuracy 0.961
balanced  caught 64/80  false alarms 402  accuracy 0.791
  • Plain model: accuracy 0.961, but only 6 of 80 faulty cookers caught.
  • class_weight="balanced": 64 of 80 caught, at the price of 402 false alarms (good cookers sent for re-checks); accuracy falls to 0.791 although the model is far more useful here. Accuracy hides the problem; Episode 24 introduces better metrics.

How class weights work:

loss = Σ weight[class of example i] × loss_i
balanced:  weight_c = n / (K × n_c)        n = examples, K = classes, n_c = examples in class c

For 4% faulty: faulty ≈ 4000 / (2 × 160) = 12.5, good ≈ 4000 / (2 × 3840) ≈ 0.52 (computed on the training half by scikit-learn). You can also pass your own: class_weight={0: 1, 1: 10}.

Common mistakes

Mistake Real symptom Why Fix
reading a weight as a probability change "+0.748 per complaint" would pass 100% weights are in log-odds odds ratio e^w
treating C like α C = 100 barely changes anything; C = 0.0001 predicts one class C = 1 / strength small C = strong penalty
adding up one-vs-rest probabilities 0.072 + 0.308 + 0.383 = 0.763 models trained separately multinomial (softmax), or OneVsRestClassifier which normalises
softmax without subtracting the max RuntimeWarning: overflow encountered in exp, result [nan nan nan] e^1000 = inf np.exp(z - z.max())
trusting accuracy on a rare class accuracy 0.961 while catching 6 of 80 the majority class dominates class weights, better metrics (Episode 24)
comparing raw odds ratios across features months 0.953 vs complaints 2.113 different units standardise, then compare (mini project)

Mini project: which feature matters most?

Odds ratios per one standard deviation (standardised features) put every feature on the same scale:

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000            # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
                   "data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)   # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
m = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
sd = X.std(ddof=0)
print("feature      1 SD      odds ratio per 1 SD")
for name, w in sorted(zip(X.columns, m[-1].coef_[0]), key=lambda t: -abs(t[1])):
    print(f"{name:>10}  {sd[name]:7.2f}   {np.exp(w):.3f}")

Output:

feature      1 SD      odds ratio per 1 SD
    months    17.05   0.444
complaints     1.08   2.237
   data_gb    16.28   0.735

Per typical change, a customer's tenure (17 months ↔ odds × 0.444) matters about as much as complaints (1 more ↔ × 2.237, in the other direction), and data usage matters least.

Practice

  1. Explain the odds ratio of data_gb in your own words, per 1 GB and per 10 GB.
  2. Print the validation accuracy for 20 values of C between 0.0001 and 100.
  3. Compare OneVsRestClassifier(LogisticRegression()) with the multinomial model on the café data.
  4. Try class_weight={0: 1, 1: 5} on the cookers. How many are caught, and how many false alarms?
Solution 1
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000            # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
                   "data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)   # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
m = LogisticRegression(max_iter=1000).fit(X, y)
w = m.coef_[0][2]                                     # data_gb
print(f"per 1 GB:  odds x {np.exp(w):.3f}  ({(np.exp(w) - 1):+.1%})")
print(f"per 10 GB: odds x {np.exp(10 * w):.3f}  ({(np.exp(10 * w) - 1):+.1%})")

Output:

per 1 GB:  odds x 0.981  (-1.9%)
per 10 GB: odds x 0.827  (-17.3%)

Each extra GB of data per month multiplies the odds of leaving by 0.981 (1.9% lower); 10 more GB multiply them by 0.981¹⁰ ≈ 0.827 (17.3% lower). Heavy users are a bit less likely to leave, holding months and complaints fixed.

Solution 2
import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(23); n = 1000            # made-up mobile-plan customers
df = pd.DataFrame({"months": rng.integers(1, 60, n), "complaints": rng.poisson(1.2, n),
                   "data_gb": rng.uniform(1, 60, n).round(1)})
z = 0.8 * df.complaints - 0.05 * df.months - 0.02 * df.data_gb + 0.3
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)   # 1 = customer left
X, y = df[["months", "complaints", "data_gb"]], df.left
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.3, random_state=0)
for C in np.logspace(-4, 2, 20):
    m = make_pipeline(StandardScaler(), LogisticRegression(C=C)).fit(X_tr, y_tr)
    print(f"C {C:9.4f}  accuracy {m.score(X_va, y_va):.3f}")

Output:

C    0.0001  accuracy 0.687
C    0.0002  accuracy 0.687
C    0.0004  accuracy 0.687
C    0.0009  accuracy 0.687
C    0.0018  accuracy 0.700
C    0.0038  accuracy 0.727
C    0.0078  accuracy 0.740
C    0.0162  accuracy 0.733
C    0.0336  accuracy 0.730
C    0.0695  accuracy 0.733
C    0.1438  accuracy 0.727
C    0.2976  accuracy 0.727
C    0.6158  accuracy 0.723
C    1.2743  accuracy 0.723
C    2.6367  accuracy 0.723
C    5.4556  accuracy 0.723
C   11.2884  accuracy 0.723
C   23.3572  accuracy 0.723
C   48.3293  accuracy 0.723
C  100.0000  accuracy 0.723

Very small C predicts one class (0.687); the best region is around C ≈ 0.008; above C ≈ 1 nothing changes.

Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5); n = 600              # made-up café orders
X = np.c_[rng.uniform(10, 42, n), rng.uniform(7, 23, n)]          # temperature °C, hour of day
Z = np.c_[-0.15 * X[:, 0] - 0.1 * X[:, 1] + 6, 0.2 * X[:, 1] - 0.05 * X[:, 0], 0.25 * X[:, 0] - 6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])  # 0 chai, 1 coffee, 2 cold drink
names = ["chai", "coffee", "cold drink"]; cust = [[30, 15]]       # 30 °C at 3 pm
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import log_loss
ovr = OneVsRestClassifier(LogisticRegression()).fit(X, y)
multi = LogisticRegression().fit(X, y)
for name, m in [("one-vs-rest", ovr), ("multinomial", multi)]:
    p = m.predict_proba(cust)[0]
    print(f"{name}: {p.round(3)} sum {p.sum():.3f}  accuracy {m.score(X, y):.3f}  log loss {log_loss(y, m.predict_proba(X)):.4f}")

Output:

one-vs-rest: [0.095 0.403 0.502] sum 1.000  accuracy 0.730  log loss 0.6014
multinomial: [0.106 0.459 0.435] sum 1.000  accuracy 0.745  log loss 0.5815

OneVsRestClassifier normalises its probabilities to sum to 1, but they still come from three separate models: for this customer it even picks a different drink (cold drink, 0.502) than the multinomial model (coffee, 0.459). Multinomial has the better accuracy and log loss here.

Solution 4
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(3); n = 4000             # made-up factory: 2 sensors per cooker, 1 = faulty
X = rng.normal(0, 1, (n, 2))
y = (rng.random(n) < 1 / (1 + np.exp(4.5 - 1.5 * X[:, 0] - X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
print(f"faulty: {y.mean():.1%} of cookers")
for cw in [None, {0: 1, 1: 5}, "balanced"]:
    m = LogisticRegression(class_weight=cw).fit(X_tr, y_tr); pred = m.predict(X_te)
    caught, alarms = np.sum(pred[y_te == 1] == 1), np.sum(pred[y_te == 0] == 1)
    print(f"{str(cw):<16} caught {caught}/{y_te.sum()}  false alarms {alarms:3}  accuracy {m.score(X_te, y_te):.3f}")

Output:

faulty: 4.0% of cookers
None             caught 6/80  false alarms   5  accuracy 0.961
{0: 1, 1: 5}     caught 30/80  false alarms  83  accuracy 0.933
balanced         caught 64/80  false alarms 402  accuracy 0.791

Weight 5 sits in between: 30 caught with 83 false alarms. The weight is a dial between misses and false alarms.

Quick quiz

churn model, feature "late_payments": weight = 0.693   (e^0.693 ≈ 2.0)
one more late payment ...

A) doubles the odds of leaving B) adds 0.693 to the probability C) doubles the probability D) adds 69.3% to the odds

Answer

A. The weight adds 0.693 to the log-odds, so the odds are multiplied by e^0.693 ≈ 2. The probability can't double in general: a customer at 0.9 can't go to 1.8.

Interview questions

  1. How do you interpret a logistic regression coefficient? A one-unit increase adds w to the log-odds, i.e. multiplies the odds by e^w (the odds ratio), holding other features fixed.
  2. What is C in scikit-learn's LogisticRegression? The inverse of the regularisation strength; smaller C = stronger penalty. Default 1, L2 penalty.
  3. One-vs-rest vs multinomial? OvR trains K independent binary models (works with any binary classifier, probabilities not jointly calibrated); multinomial trains one model with softmax and categorical cross-entropy, giving coherent probabilities.
  4. What is softmax and how do you compute it safely? e^(z_k) / Σ e^(z_j); subtract max(z) first to avoid overflow (log-sum-exp trick).
  5. Relation between softmax and sigmoid? For two classes, softmax of (z₁, z₂) gives σ(z₁ − z₂) for class 1.
  6. What is categorical cross-entropy? The mean of −ln(predicted probability of the true class); for K = 2 it equals binary cross-entropy.
  7. How do you handle imbalanced classes in logistic regression? Class weights (class_weight="balanced" or a dict), resampling, choosing the threshold, and judging with recall/precision rather than accuracy.
  8. Why standardise before comparing coefficients? Raw coefficients are per unit of each feature; standardised coefficients are per standard deviation and comparable. Standardising also makes the penalty fair (Episode 21).
  9. What does coef_ look like for 3 classes and 2 features? Shape (3, 2): one row of weights per class.

Glossary

Term Meaning Example
Churn Customers leaving a service left = 1
LogisticRegression scikit-learn's logistic regression (L2 penalty by default) LogisticRegression(C=1)
predict_proba Probability of each class, one column per class column 1 = P(left)
coef_, intercept_ Learned weights and bias coef_ shape (3, 2) for 3 classes
max_iter Maximum optimiser iterations max_iter=1000
Odds ratio e^w: how the odds are multiplied per unit increase complaints 2.113
Log-odds ln(p / (1 − p)), linear in the features (Episode 22) +0.748 per complaint
C Inverse regularisation strength, C = 1 / strength small C = strong penalty
L2 penalty Sum of squared weights (Episode 21) default in LogisticRegression
Underfitting Model too simple to capture the pattern (Episode 20) C = 0.0001: everyone "stays"
Multiclass classification Predicting one of 3 or more classes chai / coffee / cold drink
One-vs-rest (OvR) K binary models, one per class against the rest "coffee or not"
OneVsRestClassifier scikit-learn wrapper for OvR OneVsRestClassifier(LogisticRegression())
Score (logit) per class z_k = w_k · x + b_k chai 2, coffee 1, cold drink −1
Softmax e^(z_k) / Σ e^(z_j): scores → probabilities summing to 1 [2, 1, −1] → [0.705, 0.259, 0.035]
Overflow A number too large for the float type → inf e^1000
Multinomial logistic regression Logistic regression with softmax for K classes (softmax regression) default for 3+ classes
Categorical cross-entropy Mean of −ln(p of the true class) 0.5815 on the café data
log_loss scikit-learn function for (categorical) cross-entropy log_loss(y, P)
Imbalanced data One class much rarer than another 4% faulty
Class weight Multiplier on each example's loss by class class_weight="balanced"
Balanced weights n / (K × n_c) faulty ≈ 12.5, good ≈ 0.52
Stratified split Split keeping class proportions (Episode 15) stratify=y
False alarm Predicting the rare class when it isn't (a false positive, Episode 24) 402 good cookers flagged

Summary

  • LogisticRegression = Episode 22's model + an L2 penalty; read weights as odds ratios e^w (complaints: × 2.113 odds per complaint), not probability changes.
  • C = 1 / strength: C = 0.0001 predicted "stays" for everyone (0.687); C = 0.01 was best.
  • One-vs-rest probabilities didn't add up (0.763); softmax fixes that; subtract the max to avoid overflow.
  • Multinomial logistic regression gives coherent probabilities (sum 1.000) and is trained with categorical cross-entropy (0.5815 by hand = log_loss).
  • Class weights caught 64 of 80 faulty cookers instead of 6, while accuracy fell from 0.961 to 0.791: accuracy can lie.

Next episode: classification metrics: the confusion matrix, precision, recall, specificity, F1, F-beta, and macro, micro and weighted averages.