Machine Learning Full Course · Episode 21. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching, and the auction purse is a made-up illustration, not a real league's rule.
The big idea: overfit models hide huge weights. Add a penalty term to the loss that makes big weights expensive, and the model keeps its capacity but loses its wildness. Ridge (L2) shrinks every weight; lasso (L1) sets useless weights to exactly zero (sparsity, free feature selection); elastic net mixes both. Always scale first, and choose the regularisation strength α on validation data.
Overfit models have huge weights
Episode 20's chai stall: 20 days of cups sold against temperature, a degree-15 polynomial. The report function fits degree 15 with whatever model we pass and prints the biggest weight:
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5) # episode 20's chai
rng = np.random.default_rng(21)
def sample(n):
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_tr, y_tr = sample(20); X_va, y_va = sample(200)
def report(name, model):
m = make_pipeline(PolynomialFeatures(15), model).fit(X_tr, y_tr)
w = np.abs(m[-1].coef_).max() # biggest weight
tr, va = mse(y_tr, m.predict(X_tr)), mse(y_va, m.predict(X_va))
print(f"{name:<12} max|w| {w:8.1f} train {tr:6.1f} val {va:9.1f}")
report("no penalty", LinearRegression())
for a in [0.0001, 0.01, 0.1, 1, 100]:
report(f"ridge {a}", Ridge(alpha=a))
Output:
no penalty max|w| 76208.2 train 8.6 val 4280828.2
ridge 0.0001 max|w| 182.7 train 21.2 val 2852.9
ridge 0.01 max|w| 102.9 train 24.5 val 349.4
ridge 0.1 max|w| 78.7 train 43.3 val 83.9
ridge 1 max|w| 60.7 train 149.2 val 271.7
ridge 100 max|w| 4.9 train 1995.3 val 1977.1
With no penalty the biggest weight is 76,208 for data that never exceeds 200 cups. That's how a curve wiggles through every dot: giant positive and negative weights that almost cancel, and stop cancelling between the dots.
The penalty term
A cricket team at an auction with unlimited money buys everyone at any price. Give it a fixed purse and every rupee must earn its place: big money on stars, little on fringe players. For a model, the purse is a penalty term added to the loss:
ridge loss = MSE + α · (w1² + w2² + ... + wd²)
- Penalty term: α · Σw², an extra cost for big weights. The bias
bis not penalised (shifting the line up or down isn't complexity). - L2 penalty: the sum of squared weights. Linear regression with it is ridge regression.
- Regularisation strength α: how expensive weights are. scikit-learn calls it
alpha; many books call it λ (lambda). It is a hyperparameter (Episode 8): you choose it. - α = 0: plain linear regression. α → ∞: every weight → 0, the model predicts the mean.
- The normal equation (Episode 18) changes by one term: θ = (XᵀX + αI)⁻¹Xᵀy (with the bias left out of the penalty).
- Regularisation in general: anything added to training that discourages a complicated model.
(Exact scikit-learn objectives: Ridge minimises ‖y − Xw‖² + α‖w‖² (a sum of squares, not a mean); Lasso minimises (1/2n)‖y − Xw‖² + α‖w‖₁; ElasticNet minimises (1/2n)‖y − Xw‖² + α·r‖w‖₁ + ½α(1 − r)‖w‖². So the same α means different strengths in different estimators.)
From the run above:
| α | biggest weight | train MSE | validation MSE | |
|---|---|---|---|---|
| 0 (no penalty) | 76,208.2 | 8.6 | 4,280,828.2 | overfits |
| 0.0001 | 182.7 | 21.2 | 2,852.9 | |
| 0.01 | 102.9 | 24.5 | 349.4 | |
| 0.1 | 78.7 | 43.3 | 83.9 | best here |
| 1 | 60.7 | 149.2 | 271.7 | |
| 100 | 4.9 | 1,995.3 | 1,977.1 | underfits |
Too little α overfits, too much underfits: α is a capacity knob, chosen with a validation curve (Episode 20) or cross-validation (Episode 26).
Ridge from scratch
The ridge normal equation in NumPy matches scikit-learn's Ridge (the features are centred so the bias can be left unpenalised):
import numpy as np
from sklearn.linear_model import Ridge
rng = np.random.default_rng(0)
X = rng.normal(size=(50, 4)); y = X @ [3, -2, 0, 1] + 5 + rng.normal(0, 1, 50)
alpha = 10
Xc, yc = X - X.mean(axis=0), y - y.mean() # centre: bias handled separately
w = np.linalg.solve(Xc.T @ Xc + alpha * np.eye(4), Xc.T @ yc)
b = y.mean() - X.mean(axis=0) @ w
print("NumPy :", w.round(4), round(b, 4))
m = Ridge(alpha=alpha).fit(X, y)
print("Ridge :", m.coef_.round(4), round(m.intercept_, 4))
Output:
NumPy : [ 2.3664 -1.4904 0.1092 0.8902] 4.888
Ridge : [ 2.3664 -1.4904 0.1092 0.8902] 4.888
Lasso, sparsity and feature selection
Episode 19's flats (area, metro distance, age) plus 40 columns of pure random noise. Every model is scaled first:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
for name, reg in [("linear", LinearRegression()), ("ridge", Ridge(alpha=30)), ("lasso", Lasso(alpha=0.3))]:
m = make_pipeline(StandardScaler(), reg).fit(X_tr, y_tr); w = m[-1].coef_
print(f"{name:6} real {np.round(w[:3], 2)} junk used {np.sum(w[3:] != 0):2} R2 {m.score(X_te, y_te):.3f}")
Output:
linear real [ 4.41 -2.58 -1.21] junk used 40 R2 0.845
ridge real [ 3.83 -2.2 -1.04] junk used 40 R2 0.833
lasso real [ 4.1 -2.24 -0.84] junk used 0 R2 0.866
- Linear: all 40 junk weights in use, test R² 0.845.
- Ridge (α = 30): still 40 junk weights (small, never zero), 0.833.
- Lasso (α = 0.3): 0 junk weights, best test R² 0.866.
lasso loss = MSE + α · (|w1| + |w2| + ... + |wd|)
- L1 penalty: the sum of absolute values. Lasso = least absolute shrinkage and selection operator.
- Why exact zeros? A weight of 0.01 costs 0.0001 under L2 (ridge barely cares, so tiny weights survive) but 0.01 under L1 (still hurts). Under lasso a weight keeps paying all the way to 0, so unless a feature cuts the error by more than its cost, its weight becomes exactly 0.
- Sparsity: many weights exactly zero (a sparse model: mostly empty).
- Feature selection: choosing which features to keep. A zero weight means the feature is unused, so lasso selects features for free.
- Geometric picture: the L1 constraint region is a diamond with corners on the axes, and the best point under that constraint often lands on a corner, where some weights are 0. The L2 region is a circle with no corners.
The regularisation path
Fit lasso for many α values and record each weight:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
cols = ["area", "metro_km", "age", "junk1", "junk2", "junk3"]; alphas = np.logspace(-2, 1, 40) # 0.01..10
W = np.array([make_pipeline(StandardScaler(), Lasso(alpha=a)).fit(X_tr[cols], y_tr)[-1].coef_ for a in alphas])
for j, c in enumerate(cols): # first alpha where the weight is 0
print(f"{c:>8} becomes 0 at alpha {alphas[np.argmax(W[:, j] == 0)]:.2f}")
plt.figure(figsize=(10, 4.8)); plt.xscale("log"); plt.plot(alphas, W, label=cols); plt.axhline(0, color="k")
plt.xlabel("alpha"); plt.ylabel("weight"); plt.legend(); plt.savefig("lasso_path.png", dpi=110)
Output:
area becomes 0 at alpha 4.92
metro_km becomes 0 at alpha 2.89
age becomes 0 at alpha 1.19
junk1 becomes 0 at alpha 0.01
junk2 becomes 0 at alpha 0.08
junk3 becomes 0 at alpha 0.10

A plot of every weight against α is the regularisation path. Junk goes first (junk1 is already 0 at α = 0.01), then age (≈1.19), metro distance (≈2.89) and area last (≈4.92): the most useful feature survives longest. scikit-learn also has lasso_path and LassoCV, which compute the path efficiently and pick α by cross-validation.
Scale before regularising
A tax of ₹1 per digit on your bill would be silly: same money, different tax, just from how it's written. An unscaled penalty is the same mistake. Area in sq ft needs a weight of 0.019; in thousands of sq ft the same information needs 19, so a penalty on weights punishes it 1,000× more purely because of units:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
cols = ["area", "metro_km", "age"]; A_tr, A_te = X_tr[cols], X_te[cols]
K_tr, K_te = (A.assign(area=A.area / 1000) for A in (A_tr, A_te)) # area in 1000s of sq ft
for name, a, b in [("sq ft", A_tr, A_te), ("1000 sq ft", K_tr, K_te)]:
raw = Lasso(alpha=0.5).fit(a, y_tr)
scaled = make_pipeline(StandardScaler(), Lasso(alpha=0.5)).fit(a, y_tr)
print(f"{name:>10}: raw test R2 {raw.score(b, y_te):.3f} scaled {scaled.score(b, y_te):.3f}")
Output:
sq ft: raw test R2 0.875 scaled 0.847
1000 sq ft: raw test R2 0.728 scaled 0.847
- Raw: 0.875 with sq ft, 0.728 with thousands of sq ft. Same flats, a worse model, only because of units.
- Scaled: 0.847 both times. With standardisation (Episode 16: mean 0, standard deviation 1) the penalty is fair.
- Put the scaler inside the pipeline, so it is fitted on training data only (no leakage, Episode 15).
Elastic net
Twin features (Episode 19's area and area in m²): lasso keeps one and zeroes the other, an almost arbitrary choice; ridge shares the weight but never drops junk. Mix them:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import ElasticNet
T_tr = X_tr.assign(area_sqm=(X_tr.area / 10.764).round(1)) # area's twin, in sq m
for name, reg in [("lasso", Lasso(alpha=0.5)), ("elastic net", ElasticNet(alpha=0.5, l1_ratio=0.9))]:
w = dict(zip(T_tr.columns, make_pipeline(StandardScaler(), reg).fit(T_tr, y_tr)[-1].coef_))
junk = sum(w[f"junk{i}"] != 0 for i in range(1, 41))
print(f"{name:>11}: area {w['area']:.2f} area_sqm {w['area_sqm']:.2f} junk used {junk}")
Output:
lasso: area 3.91 area_sqm 0.00 junk used 0
elastic net: area 1.93 area_sqm 1.93 junk used 0
- Lasso: all of area's weight on one twin, 0 on the other.
- Elastic net (α = 0.5,
l1_ratio= 0.9): 1.93 and 1.93, shared evenly, and still 0 junk weights.
elastic net penalty = α · ( r · Σ|w| + (1 − r)/2 · Σw² ) r = l1_ratio
l1_ratio= 1: pure lasso; 0: pure ridge; in between: a blend.- L1 part: sparsity. L2 part: shares weight between correlated features (multicollinearity).
- Two hyperparameters (α and
l1_ratio);ElasticNetCVtunes both by cross-validation.
Ridge vs lasso vs elastic net
| Ridge (L2) | Lasso (L1) | Elastic net | |
|---|---|---|---|
| Penalty | α·Σw² | α·Σ|w| | mix |
| Exact zeros | no | yes | yes |
| Feature selection | no | yes | yes |
| Correlated twins | shares weight | picks one | shares weight |
| Good when | most features useful | few features matter | many correlated features, some junk |
| Needs scaling | yes | yes | yes |
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| regularising unscaled features | test R² 0.875 → 0.728 just from changing units | the penalty depends on units | StandardScaler in the pipeline |
| α too big | α = 100: train 1,995.3, validation 1,977.1 | underfitting | validation curve / cross-validation for α |
| expecting ridge to drop features | 40 junk weights not 0 | L2 never gives exact zeros | lasso or elastic net |
| reading meaning into which twin lasso kept | area kept, area_sqm = 0 | arbitrary between twins | elastic net, or drop a twin |
| scaling all the data before splitting | optimistic score | leakage (Episode 15) | fit the scaler on train only (pipeline) |
| comparing α across estimators | same α, very different effect | Ridge uses a sum, Lasso a mean (1/2n) |
tune α per estimator |
| lasso with too few iterations | ConvergenceWarning: Objective did not converge |
coordinate descent stopped early | scale features, raise max_iter |
Mini project: choose α automatically
RidgeCV and LassoCV try many α values with cross-validation (Episode 26) and keep the best:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import RidgeCV, LassoCV
alphas = np.logspace(-3, 3, 30)
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas)).fit(X_tr, y_tr)
lasso = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0)).fit(X_tr, y_tr)
print("RidgeCV alpha", round(ridge[-1].alpha_, 3), " test R2", round(ridge.score(X_te, y_te), 3))
print("LassoCV alpha", round(lasso[-1].alpha_, 3), " test R2", round(lasso.score(X_te, y_te), 3),
" features kept:", int(np.sum(lasso[-1].coef_ != 0)), "of", X.shape[1])
print("kept:", list(X.columns[lasso[-1].coef_ != 0]))
Output:
RidgeCV alpha 8.532 test R2 0.845
LassoCV alpha 0.221 test R2 0.871 features kept: 7 of 43
kept: ['area', 'metro_km', 'age', 'junk4', 'junk23', 'junk28', 'junk35']
Cross-validation picks the α that predicts best, not the sparsest model: LassoCV chose a slightly smaller α than our 0.3, so a few junk columns survive with tiny weights, and its test R² is a touch higher. If you want fewer features, choose a larger α on purpose (or the largest α within one standard error of the best).
Practice
- Draw a validation curve for ridge's α on the chai data. Which α is best?
- Add 100 junk columns instead of 40. How many does lasso (α = 0.3) keep?
- Implement ridge with the normal equation for the degree-15 chai model and compare with
Ridge. - Run elastic net on the twins with
l1_ratio0.1, 0.5 and 0.9. How many junk weights survive?
Solution 1
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n):
t = rng.uniform(5, 40, n)
return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_tr, y_tr = sample(20); X_va, y_va = sample(200)
best = None
for a in np.logspace(-4, 2, 13):
m = make_pipeline(PolynomialFeatures(15), Ridge(alpha=a)).fit(X_tr, y_tr)
va = mse(y_va, m.predict(X_va)); best = min(best or (va, a), (va, a))
print(f"alpha {a:9.4f}: train {mse(y_tr, m.predict(X_tr)):7.1f} validation {va:8.1f}")
print("best alpha", round(best[1], 4), "validation", round(best[0], 1))
Output:
alpha 0.0001: train 21.2 validation 2852.9
alpha 0.0003: train 22.1 validation 1944.2
alpha 0.0010: train 22.8 validation 1093.5
alpha 0.0032: train 23.3 validation 611.6
alpha 0.0100: train 24.5 validation 349.4
alpha 0.0316: train 29.2 validation 142.5
alpha 0.1000: train 43.3 validation 83.9
alpha 0.3162: train 75.1 validation 211.0
alpha 1.0000: train 149.2 validation 271.7
alpha 3.1623: train 374.4 validation 285.4
alpha 10.0000: train 876.9 validation 700.1
alpha 31.6228: train 1497.2 validation 1411.7
alpha 100.0000: train 1995.3 validation 1977.1
best alpha 0.1 validation 83.9
A U again: the validation error falls, bottoms out, then rises as α crushes the weights.
Solution 2
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018 * df.area - 1.2 * df.metro_km - 0.15 * df.age + 6 + rng.normal(0, 2.5, n)).round(1)
junk = pd.DataFrame(rng.normal(0, 1, (n, 100)).round(2), columns=[f"junk{i}" for i in range(1, 101)])
df = pd.concat([df, junk], axis=1)
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
for name, reg in [("linear", LinearRegression()), ("lasso", Lasso(alpha=0.3))]:
m = make_pipeline(StandardScaler(), reg).fit(X_tr, y_tr); w = m[-1].coef_
print(f"{name}: real kept {np.sum(w[:3] != 0)}/3, junk kept {np.sum(w[3:] != 0)}/100, test R2 {m.score(X_te, y_te):.3f}")
Output:
linear: real kept 3/3, junk kept 100/100, test R2 0.789
lasso: real kept 3/3, junk kept 9/100, test R2 0.862
With 103 features and only 225 training rows, plain linear regression overfits badly; lasso keeps the real features and almost no junk.
Solution 3
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 20); x = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 20)
P = PolynomialFeatures(15, include_bias=False).fit_transform(x) # 15 columns: x .. x^15
alpha = 0.1
Pc, yc = P - P.mean(axis=0), y - y.mean()
w = np.linalg.solve(Pc.T @ Pc + alpha * np.eye(15), Pc.T @ yc)
b = y.mean() - P.mean(axis=0) @ w
m = Ridge(alpha=alpha).fit(P, y)
print("max |difference| in weights:", float(np.abs(w - m.coef_).max()))
print("intercepts:", round(b, 4), round(m.intercept_, 4))
Output:
max |difference| in weights: 1.865174681370263e-13
intercepts: 43.6457 43.6457
The one-term change to the normal equation, + αI, is exactly what Ridge solves.
Solution 4
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300 # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2) # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import ElasticNet
T_tr = X_tr.assign(area_sqm=(X_tr.area / 10.764).round(1))
for r in [0.1, 0.5, 0.9]:
m = make_pipeline(StandardScaler(), ElasticNet(alpha=0.5, l1_ratio=r)).fit(T_tr, y_tr)
w = dict(zip(T_tr.columns, m[-1].coef_))
junk = sum(w[f"junk{i}"] != 0 for i in range(1, 41))
print(f"l1_ratio {r}: area {w['area']:.2f} area_sqm {w['area_sqm']:.2f} junk kept {junk}")
Output:
l1_ratio 0.1: area 1.75 area_sqm 1.75 junk kept 29
l1_ratio 0.5: area 1.83 area_sqm 1.83 junk kept 3
l1_ratio 0.9: area 1.93 area_sqm 1.93 junk kept 0
More L1 (higher l1_ratio) means more junk dropped; the L2 part keeps the twins sharing the weight evenly.
Quick quiz
weights after fitting, 6 scaled features:
[ 2.41, 0.0, -1.10, 0.0, 0.0, 0.37 ]
alpha = 0.3
Ridge, lasso, or plain linear?
A) Lasso B) Ridge C) Plain linear regression D) Can't tell
Answer
A) Lasso. Exact zeros are the signature of an L1 penalty; ridge and plain linear regression essentially never produce weights of exactly 0.
Interview questions
- What is regularisation and why does it help? Adding a penalty (or constraint) that discourages complex models, trading a little bias for a large cut in variance.
- Ridge vs lasso? L2 (Σw²) shrinks all weights smoothly, never to 0, handles correlated features by sharing. L1 (Σ|w|) produces exact zeros: sparse models and feature selection; with correlated features it picks one.
- Why does L1 give exact zeros but L2 doesn't? L1's cost per unit of weight stays constant near 0 (non-smooth at 0; the diamond has corners on the axes); L2's cost vanishes near 0, so tiny weights are cheap.
- Why scale before regularising? The penalty acts on raw weights, whose size depends on feature units; without scaling, features are penalised unequally.
- How do you choose α? Validation curve or cross-validation (
RidgeCV,LassoCV,GridSearchCV). - What is elastic net and when would you use it? A mix of L1 and L2 (
l1_ratio); many correlated features with some irrelevant ones, or more features than rows. - Is the intercept regularised? No; penalising it would make predictions depend on where the target's zero is.
- How does ridge relate to the normal equation? θ = (XᵀX + αI)⁻¹Xᵀy; adding αI also makes the matrix invertible when features are collinear.
- Bayesian view? Ridge = MAP with a Gaussian prior on weights; lasso = MAP with a Laplace prior (Episode 12).
Glossary
| Term | Meaning | Example |
|---|---|---|
| Regularisation | Anything added to training that discourages complexity | a penalty on weights |
| Penalty term | Extra loss for big weights | α·Σw² |
| Regularisation strength (α, λ) | How expensive weights are | alpha=0.1 |
| Hyperparameter | A setting you choose, not learned | α, l1_ratio |
| L2 penalty | Sum of squared weights | 0.01² = 0.0001 |
| Ridge regression | Linear regression + L2 penalty | Ridge(alpha=0.1) |
| L1 penalty | Sum of absolute weights | |0.01| = 0.01 |
| Lasso regression | Linear regression + L1 penalty | Lasso(alpha=0.3) |
| Sparsity / sparse model | Many weights exactly 0 | 40 junk weights = 0 |
| Feature selection | Choosing which features to keep | lasso drops junk |
| Regularisation path | Each weight plotted against α | junk → 0 first, area last |
| Elastic net | L1 + L2 mixed | ElasticNet(alpha=0.5, l1_ratio=0.9) |
l1_ratio |
The L1 share in elastic net | 1 = lasso, 0 = ridge |
| Standardisation | Mean 0, standard deviation 1 per feature | StandardScaler |
| Shrinkage | Pulling weights toward 0 | 76,208 → 78.7 |
| Identity matrix (I) | Square matrix with 1s on the diagonal (Episode 4) | αI in the normal equation |
RidgeCV, LassoCV |
Pick α by cross-validation | alpha_ attribute |
| Multicollinearity | Features that copy each other (Episode 19) | area and area_sqm |
Summary
- Overfit models hide huge weights (76,208 for a 200-cup target).
- Penalty term: loss = MSE + α·penalty. Ridge (L2) shrinks every weight; with α = 0.1 validation MSE fell from 4,280,828 to 83.9; α = 100 underfits.
- Lasso (L1) sets useless weights to exactly zero: sparsity and feature selection (40 junk columns → 0, test R² 0.866).
- The regularisation path shows features dropping out in order of usefulness.
- Scale first: unscaled lasso changed from 0.875 to 0.728 just by changing units; scaled stayed 0.847.
- Elastic net shares weight between twins and still drops junk.
Next episode: logistic regression from scratch: predicting yes or no with the sigmoid, odds, log-odds, the decision boundary and log loss.