Machine Learning Full Course · Episode 19. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching (the braking numbers come from a made-up formula, not real braking data).
The big idea: fitting a linear model is one line. Using it well is everything after: read the weights correctly, score the model in units people understand (MAE, RMSE) and against a baseline (R²), and check the shape of its mistakes (residual plots) before you trust it.
Fitting LinearRegression and reading coefficients
A property broker's (made-up) rules of thumb: "every extra square foot adds about ₹19", "every km from the metro takes off about ₹1,100", "same flat, just 1 km further". The last one is the key to reading any linear model.
The data: 300 made-up flats with area (sq ft), distance to the metro (km), age (years) and rent (₹ thousands). As in Episode 15, a quarter is kept aside as a test set.
LinearRegression is scikit-learn's ready-made linear model (Episode 17's estimator API). fit finds the least-squares weights (Episode 18). After fitting:
intercept_holds the bias (intercept): the prediction when every feature is 0.coef_holds the coefficients (weights), one per feature, in column order.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
model = LinearRegression().fit(X_tr, y_tr)
print("intercept:", round(model.intercept_, 2))
for name, w in zip(X.columns, model.coef_):
print(f"{name:>9}: {w:+.4f}")
new = pd.DataFrame({"area": [1000], "metro_km": [2], "age": [5]})
print("1000 sq ft, 2 km, 5 yrs ->", model.predict(new).round(2))
Output:
intercept: 4.54
area: +0.0190
metro_km: -1.1386
age: -0.1374
1000 sq ft, 2 km, 5 yrs -> [20.6]
How to interpret coefficients
| Feature | Weight | In words (rent is in ₹ thousands) |
|---|---|---|
| area | +0.0190 | each extra sq ft adds about ₹19, holding metro distance and age fixed |
| metro_km | −1.1386 | each extra km from the metro takes off about ₹1,139 |
| age | −0.1374 | each extra year of age takes off about ₹137 |
| intercept | 4.54 | the rent of a 0 sq ft, brand-new flat at the metro gate: no such flat, so no meaning here |
Rules:
- A coefficient is the change in the prediction when its feature goes up by one unit, holding all other features fixed.
- Units matter. Area's weight looks tiny only because a square foot is a tiny unit. To compare which feature matters most, standardise first (Episode 16); then each weight is "per standard deviation".
- The intercept is often meaningless on its own (no real row has every feature at 0).
- A weight shows an association in this data, not proof of cause (correlation vs causation, Episode 11).
- Weights are only trustworthy when features don't copy each other (see multicollinearity below).
MAE and RMSE: "how wrong, in rupees?"
The landlord asks "how wrong is it, usually?" Last episode's answer, MSE 5.43, is in squared thousands of rupees. We need scores in the target's own units.
- Absolute value |x|: the size of a number without its sign: |−3| = 3.
- MAE (mean absolute error) = mean(|y − ŷ|): the typical miss.
- RMSE (root mean squared error) = √MSE = √mean((y − ŷ)²): squares first, so big misses count extra, then the square root brings it back to normal units.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.metrics import mean_squared_error, mean_absolute_error
model = LinearRegression().fit(X_tr, y_tr)
pred = model.predict(X_te)
mse = mean_squared_error(y_te, pred)
print("MSE :", round(mse, 2), " (thousands of rupees, squared)")
print("MAE :", round(mean_absolute_error(y_te, pred), 2))
print("RMSE:", round(np.sqrt(mse), 2))
print("biggest miss:", round(np.abs(y_te - pred).max(), 2))
Output:
MSE : 5.43 (thousands of rupees, squared)
MAE : 1.84
RMSE: 2.33
biggest miss: 7.91
- MAE 1.84: on a typical test flat the model is off by about ₹1,840.
- RMSE 2.33 ≥ MAE 1.84. RMSE is always at least MAE; the gap grows when some misses are large (the biggest here is ₹7,910).
| MAE | RMSE | |
|---|---|---|
| Units | target's units | target's units |
| One miss of 10 vs ten misses of 1 | count the same | the single big miss counts much more |
| Use when | you want a plain "typical miss" | big misses are especially costly |
| scikit-learn | mean_absolute_error |
np.sqrt(mean_squared_error(...)) (or root_mean_squared_error) |
R²: compared with what?
Is a typical miss of ₹1,840 good? For flats at ₹20,000, maybe; for a hostel bed at ₹4,000, terrible. We need a relative score.
The lazy friend who guesses "everyone gets the class average" is the baseline (Episode 17): always predict the mean. R² (R-squared, coefficient of determination) is the share of the baseline's squared error your model removes:
SS_res = Σ (y − ŷ)² the model's squared misses (residual sum of squares)
SS_tot = Σ (y − ȳ)² the mean-guesser's squared misses (total sum of squares)
R² = 1 − SS_res / SS_tot
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.metrics import r2_score
from sklearn.dummy import DummyRegressor
model = LinearRegression().fit(X_tr, y_tr)
pred = model.predict(X_te)
ss_res = np.sum((y_te - pred) ** 2) # model's squared misses
ss_tot = np.sum((y_te - y_te.mean()) ** 2) # 'always the mean' misses
print("R2 by hand:", round(1 - ss_res / ss_tot, 3))
print("r2_score :", round(r2_score(y_te, pred), 3))
base = DummyRegressor(strategy="mean").fit(X_tr, y_tr)
print("baseline R2:", round(r2_score(y_te, base.predict(X_te)), 3))
Output:
R2 by hand: 0.881
r2_score : 0.881
baseline R2: -0.039
- R² = 0.881: the model removes 88% of the mean-guesser's squared error ("explains 88% of the variance").
- R² = 1: perfect. R² = 0: no better than the mean. R² < 0: worse than the mean. The baseline is slightly negative on the test set because the mean it learned on the training flats isn't exactly the test flats' mean.
Adjusted R²: junk can't buy a better score
On the training data, R² can only go up when you add a column: the model can always use it to fit a few rows slightly better by luck. So we add 0, 30 and 90 columns of pure random noise:
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.metrics import r2_score
def adj_r2(r2, n, p): return 1 - (1 - r2) * (n - 1) / (n - p - 1)
noise = np.random.default_rng(0)
for k in [0, 30, 90]: # add k columns of pure noise
Xa, Xb = (np.c_[A, noise.normal(size=(len(A), k))] for A in (X_tr, X_te))
m = LinearRegression().fit(Xa, y_tr); r2 = r2_score(y_tr, m.predict(Xa))
print(f"{3 + k:3} features: train R2 {r2:.3f} adj {adj_r2(r2, len(X_tr), 3 + k):.3f}"
f" test {r2_score(y_te, m.predict(Xb)):.3f}")
Output:
3 features: train R2 0.824 adj 0.821 test 0.881
33 features: train R2 0.848 adj 0.821 test 0.860
93 features: train R2 0.881 adj 0.797 test 0.809
- Training R² climbs 0.824 → 0.848 → 0.881 on pure junk.
- Test R² falls 0.881 → 0.860 → 0.809: the model learned noise (overfitting, Episode 20).
- Adjusted R² = 1 − (1 − R²)·(n − 1)/(n − p − 1), with n rows and p features, charges for every feature: it stays at 0.821, then drops to 0.797.
Adjusted R² is a quick check on training data; the real check is always a held-out test set.
Multicollinearity: two flatmates, one bill
Ravi and Aman always pay one bill together: ₹600. Who paid what? 300 + 300? 900 − 300? Every split gives the same total. Add area in square metres next to area in square feet and the model faces the same puzzle:
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
X2 = X.assign(area_sqm=(X.area / 10.764).round(1)) # same size, in sq m
cols = ["area", "area_sqm", "metro_km", "age"]
for half in [X_tr.index[:110], X_tr.index[110:]]: # two halves of train
m = LinearRegression().fit(X2.loc[half, cols], y[half])
w, r2 = m.coef_, m.score(X2.loc[X_te.index, cols], y_te)
print(f"area {w[0]:+.3f} area_sqm {w[1]:+.3f} test R2 {r2:.3f}")
print("corr(area, area_sqm) =", round(X2.area.corr(X2.area_sqm), 5))
Output:
area +1.339 area_sqm -14.193 test R2 0.885
area -1.383 area_sqm +15.078 test R2 0.861
corr(area, area_sqm) = 1.0
Fitted on two halves of the training data, the twin weights flip sign (+1.339 / −14.193 vs −1.383 / +15.078), yet both test R² values are fine. The total is right; the split is random.
- Multicollinearity: two or more features carry (almost) the same information. Multi = many; collinear = lying on one line.
- Symptoms: big weights with opposite signs that cancel, flipping with small data changes, large uncertainty.
- Predictions usually still OK; reading the weights is what breaks.
- Detect: a correlation (Episode 11: how closely two numbers move together, −1 to +1) near ±1 between features; for groups of features, the variance inflation factor (VIF).
- Fix: drop one twin, combine them, or regularise (ridge, Episode 21).
Residual plots and the linearity assumption
Braking data (made-up formula): stopping distance grows faster than speed. A straight line scores R² 0.955, which looks great. But averaging the residuals by speed band shows a pattern, and the program saves a residual plot:
import numpy as np, matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(7); speed = rng.uniform(10, 100, 120).round() # km/h
stop = 0.2 * speed + 0.006 * speed ** 2 + rng.normal(0, 3, 120) # metres
S = speed.reshape(-1, 1)
line = LinearRegression().fit(S, stop); res = stop - line.predict(S)
print("line R2:", round(line.score(S, stop), 3))
for lo, hi in [(10, 30), (30, 70), (70, 101)]:
band = (speed >= lo) & (speed < hi)
print(f" speed {lo}-{hi}: mean residual {res[band].mean():+.2f} m")
P = PolynomialFeatures(degree=2) # speed -> 1, speed, speed²
poly = make_pipeline(P, LinearRegression()).fit(S, stop)
print("degree 2 R2:", round(poly.score(S, stop), 3), "| 50 ->", P.fit_transform([[50]]).tolist())
fig, ax = plt.subplots(1, 2, figsize=(13, 4.8), sharey=True)
for a, m, t in [(ax[0], line, "straight line"), (ax[1], poly, "degree 2")]:
a.scatter(speed, stop - m.predict(S), s=18); a.axhline(0, color="k"); a.set(title="residuals: " + t)
plt.savefig("residuals.png", dpi=110, bbox_inches="tight")
Output:
line R2: 0.955
speed 10-30: mean residual +4.11 m
speed 30-70: mean residual -3.60 m
speed 70-101: mean residual +2.30 m
degree 2 R2: 0.986 | 50 -> [[1.0, 50.0, 2500.0]]

- A residual plot has one dot per example: the input (or the prediction) across, the residual (y − ŷ) up, and a line at 0. A good model leaves a shapeless cloud around 0.
- The straight line's residuals make a U: positive at both ends (+4.11, +2.30), negative in the middle (−3.60). The data curves and the line can't.
- Linearity assumption: linear regression assumes the target is a straight-line (weighted-sum) function of the features. A curved residual plot says it is broken.
Polynomial features and polynomial regression
The model can only add weighted features, so hand it a feature that already curves: speed².
PolynomialFeatures(degree=2)turns each input x into[1, x, x²](all powers up to the degree). 50 km/h →[1.0, 50.0, 2500.0].- The prediction ŷ = w₁·speed + w₂·speed² + b is a curve, but it is still a weighted sum (linear in the weights), so plain
LinearRegressionfits it. - Polynomial regression = linear regression on polynomial features. Here R² goes 0.955 → 0.986 and the residuals become a random cloud (right panel).
- With several features, degree 2 also adds interaction terms like area·metro_km.
- Higher degree = more bendy; too high overfits (Episode 20).
Homoscedasticity: residuals that fan out
Made-up flat prices where bigger flats vary more:
import numpy as np, matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(3)
area = rng.uniform(300, 3000, 300).round() # sq ft
price = 0.08 * area * np.exp(rng.normal(0, 0.25, 300)) # ₹ lakhs
A = area.reshape(-1, 1)
bands = [(300, 1200), (1200, 2100), (2100, 3001)]
def spreads(t): # residual std in each size band
r = t - LinearRegression().fit(A, t).predict(A)
return r, [round(float(r[(area >= lo) & (area < hi)].std()), 2) for lo, hi in bands]
res, s = spreads(price)
print("price : residual spread", s)
print("log(price): residual spread", spreads(np.log(price))[1])
plt.figure(figsize=(9, 4.8)); plt.scatter(area, res, s=14); plt.axhline(0, color="k")
plt.xlabel("area (sq ft)"); plt.ylabel("residual (lakh)"); plt.title("residuals fan out")
plt.savefig("funnel.png", dpi=110, bbox_inches="tight")
Output:
price : residual spread [15.45, 36.19, 50.64]
log(price): residual spread [0.28, 0.25, 0.27]

- Homoscedasticity (homo = same, scedastic = scatter): the residuals have about the same spread everywhere. Linear regression's error estimates assume it.
- Heteroscedasticity: the spread changes, here 15.45 → 36.19 → 50.64 lakh, a funnel in the residual plot. A single "typical miss" then misleads: the model is much better on small flats than big ones.
- Common fix: model log(target). The spreads become 0.28, 0.25, 0.27: even. (Predict in log units, then take
np.expto get lakhs back.)
The checklist after every linear fit
- Score on the test set: MAE, RMSE, R².
- Residual plot: a curve or U → linearity broken → polynomial features (or a non-linear model).
- Residual plot: a funnel → homoscedasticity broken → try log(target).
- Twin features → multicollinearity → drop / combine / regularise.
- Only then read the coefficients.
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| reading a weight without its units | "area's 0.019 is tiny, it doesn't matter" | per sq ft is a tiny unit | standardise to compare, read "holding others fixed" |
| reporting MSE to people | "5.43 ... squared thousands of rupees?" | squared units | MAE or RMSE |
| trusting training R² | 0.824 → 0.881 by adding pure noise | R² can only rise on train | test R², adjusted R² |
| keeping twin features | weights +1.339 / −14.193 flip to −1.383 / +15.078 | multicollinearity | drop one, combine, or ridge |
| trusting a high R² without plotting | R² 0.955 but residuals +4.11 / −3.60 / +2.30 | wrong shape | residual plot; polynomial features |
| ignoring a funnel | spread 15 → 51 lakh | heteroscedasticity | log(target) |
mean_squared_error(...).round(2) |
AttributeError: 'float' object has no attribute 'round' |
the metric returns a Python float | round(x, 2) |
Mini project: a regression report card
A function that does the whole checklist for any fitted regressor and test set:
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
def report_card(model, X_te, y_te, feature_names):
pred = model.predict(X_te); res = y_te - pred
n, p = X_te.shape
r2 = r2_score(y_te, pred)
print(f"MAE {mean_absolute_error(y_te, pred):.3f}")
print(f"RMSE {np.sqrt(mean_squared_error(y_te, pred)):.3f}")
print(f"R2 {r2:.3f} adjusted {1 - (1 - r2) * (n - 1) / (n - p - 1):.3f}")
thirds = np.array_split(np.argsort(pred), 3) # low / mid / high predictions
print("mean residual by prediction third:", [float(res[i].mean().round(2)) for i in thirds])
print("residual spread by prediction third:", [float(res[i].std().round(2)) for i in thirds])
corr = np.corrcoef(np.asarray(X_te, float).T)
np.fill_diagonal(corr, 0)
i, j = np.unravel_index(np.abs(corr).argmax(), corr.shape)
print(f"most correlated pair: {feature_names[i]} & {feature_names[j]} ({corr[i, j]:.2f})")
rng = np.random.default_rng(19); n = 300
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(),
"metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018 * df.area - 1.2 * df.metro_km - 0.15 * df.age + 6 + rng.normal(0, 2.5, n)).round(1)
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
m = LinearRegression().fit(X_tr.to_numpy(), y_tr.to_numpy())
report_card(m, X_te.to_numpy(), y_te.to_numpy(), list(X.columns))
Output:
MAE 1.835
RMSE 2.331
R2 0.881 adjusted 0.876
mean residual by prediction third: [-0.67, -0.08, -0.39]
residual spread by prediction third: [2.2, 2.28, 2.38]
most correlated pair: area & metro_km (-0.19)
Mean residuals near 0 in every third (no curve), similar spreads (no funnel), and no feature pair close to ±1: this model passes the checklist.
Practice
- Standardise the three features and compare the weights. Which feature matters most per standard deviation?
- Drop
area_sqmagain: are the weights stable across the two halves? - Fit degree 1, 2, 3 and 10 polynomials to the braking data and compare training and test R².
- Compute the MAE of the log-price model in lakhs.
Solution 1
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
m = make_pipeline(StandardScaler(), LinearRegression()).fit(X_tr, y_tr)
for name, w in zip(X.columns, m[-1].coef_):
print(f"{name:>9}: {w:+.3f} per standard deviation")
Output:
area: +4.388 per standard deviation
metro_km: -2.542 per standard deviation
age: -1.132 per standard deviation
Per standard deviation, area matters most (about +4.4), then metro distance (about −2.5), then age (about −1.1). On the raw scale, area's weight looked the smallest.
Solution 2
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(19); n = 300 # 300 made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
"age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1) # ₹ thousands
X, y = df[["area", "metro_km", "age"]], df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
cols = ["area", "metro_km", "age"]
for half in [X_tr.index[:110], X_tr.index[110:]]:
m = LinearRegression().fit(X.loc[half, cols], y[half])
print(" ".join(f"{c} {w:+.4f}" for c, w in zip(cols, m.coef_)))
Output:
area +0.0203 metro_km -1.0482 age -0.1157
area +0.0177 metro_km -1.2142 age -0.1503
Without the twin, both halves give similar weights: no sign flips.
Solution 3
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(7)
speed = rng.uniform(10, 100, 120).round()
stop = 0.2 * speed + 0.006 * speed ** 2 + rng.normal(0, 3, 120)
S_tr, S_te, y_tr, y_te = train_test_split(speed.reshape(-1, 1), stop, test_size=0.3, random_state=0)
for d in [1, 2, 3, 10]:
m = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).fit(S_tr, y_tr)
print(f"degree {d:2}: train R2 {m.score(S_tr, y_tr):.4f} test R2 {m.score(S_te, y_te):.4f}")
Output:
degree 1: train R2 0.9554 test R2 0.9501
degree 2: train R2 0.9841 test R2 0.9885
degree 3: train R2 0.9843 test R2 0.9870
degree 10: train R2 0.9855 test R2 0.9854
Degree 2 captures the curve; higher degrees barely improve training R² and don't improve test R². The extra bends only follow noise (Episode 20).
Solution 4
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
rng = np.random.default_rng(3)
area = rng.uniform(300, 3000, 300).round()
price = 0.08 * area * np.exp(rng.normal(0, 0.25, 300))
A, logA = area.reshape(-1, 1), np.log(area).reshape(-1, 1)
plain = LinearRegression().fit(A, price)
loglog = LinearRegression().fit(logA, np.log(price)) # log(price) from log(area)
back = np.exp(loglog.predict(logA)) # back to lakhs
print("MAE plain model :", round(mean_absolute_error(price, plain.predict(A)), 2), "lakh")
print("MAE log model :", round(mean_absolute_error(price, back), 2), "lakh")
for lo, hi in [(300, 1200), (1200, 2100), (2100, 3001)]:
s = (area >= lo) & (area < hi)
print(f"area {lo}-{hi}: plain {mean_absolute_error(price[s], plain.predict(A[s])):6.2f}"
f" log {mean_absolute_error(price[s], back[s]):6.2f}")
Output:
MAE plain model : 27.79 lakh
MAE log model : 27.47 lakh
area 300-1200: plain 13.41 log 11.95
area 1200-2100: plain 29.72 log 30.36
area 2100-3001: plain 37.87 log 37.44
Predict in log units, then np.exp to return to lakhs. Here we model log(price) from log(area), which matches how this made-up data was generated (price proportional to area, times a random percentage). Big flats still have bigger misses in lakhs, because the noise is a percentage of the price; in log units the spread is even.
Quick quiz
from sklearn.metrics import mean_absolute_error, mean_squared_error
y = [10, 10, 10, 10]
pred = [ 9, 11, 9, 18]
print(mean_absolute_error(y, pred))
print(mean_squared_error(y, pred) ** 0.5)
A) 2.75 and 4.09... B) 2.75 and 2.75 C) 1.0 and 4.09... D) 4.09... and 2.75
Answer
A. Real output:
2.75
4.092676385936225
Misses 1, 1, 1, 8 → MAE = 11/4 = 2.75. Squares 1, 1, 1, 64 → mean 16.75 → RMSE = √16.75 ≈ 4.09. One big miss pushes RMSE well above MAE.
Interview questions
- How do you interpret a linear regression coefficient? The expected change in the prediction for a one-unit increase in that feature, holding the other features fixed; in the target's units per the feature's unit.
- MAE vs RMSE? Both in target units. MAE is the average absolute miss and is robust to a few large errors; RMSE squares first, so it penalises large errors more. RMSE ≥ MAE always.
- What does R² mean, and can it be negative? 1 − SS_res/SS_tot: the share of the mean-predictor's squared error removed. Yes, on new data a model worse than predicting the mean has R² < 0.
- Why adjusted R²? Training R² never decreases when you add features; adjusted R² penalises the number of features p, so useless features lower it.
- What is multicollinearity and how do you detect and handle it? Highly correlated features; unstable, uninterpretable coefficients (predictions often fine). Detect with correlations or VIF; drop/combine features or use ridge.
- What do residual plots tell you? Curves → missing non-linearity; funnels → heteroscedasticity; outliers and clusters → data problems or missing features.
- Is polynomial regression a linear model? Yes: linear in the weights; the non-linearity is in the features.
- What are the classic assumptions of linear regression? Linearity, independent errors, homoscedasticity, (for inference) normally distributed errors, and no perfect multicollinearity.
Glossary
| Term | Meaning | Example |
|---|---|---|
LinearRegression |
scikit-learn's least-squares linear model | LinearRegression().fit(X_tr, y_tr) |
coef_ |
The fitted weights, one per feature | area +0.0190 |
intercept_ |
The fitted bias | 4.54 |
| Coefficient (weight) | Change in prediction per +1 unit of a feature, others fixed | −1.1386 per km |
| Holding others fixed | Changing one feature while all others stay the same | same flat, 1 km further |
| Association vs causation | Moving together vs one causing the other | a weight is an association |
| Absolute value | A number's size without its sign | |
| MSE | Mean of squared residuals (squared units) | 5.43 |
| MAE (mean absolute error) | Mean of absolute residuals | 1.84 = ₹1,840 |
| RMSE (root mean squared error) | √MSE, back in target units | 2.33 |
| Baseline | The simplest reference model | always predict the mean |
DummyRegressor |
scikit-learn's baseline regressor | strategy="mean" |
| SS_res | Sum of the model's squared residuals | Σ(y − ŷ)² |
| SS_tot | Sum of squared deviations from the mean | Σ(y − ȳ)² |
| R² (R-squared) | 1 − SS_res/SS_tot: share of baseline error removed | 0.881 |
| Negative R² | Worse than predicting the mean | −0.039 |
| Adjusted R² | R² penalised for the number of features | 0.821 → 0.797 |
| Overfitting | Learning noise: good on train, worse on new data | test R² 0.881 → 0.809 |
| Multicollinearity | Features that (almost) copy each other | area and area_sqm |
| Correlation | How closely two numbers move together (−1 to +1) | 1.0 |
| VIF (variance inflation factor) | How much a weight's variance is inflated by correlated features | VIF > 10 is a warning |
| Residual | True value − prediction | +4.11 m |
| Residual plot | Residuals against an input or the prediction | the U shape |
| Linearity assumption | The target is a weighted sum of the features | broken by braking data |
| Polynomial features | Powers of the inputs added as new features | 50 → [1, 50, 2500] |
| Degree | The highest power used | degree 2 |
| Polynomial regression | Linear regression on polynomial features | R² 0.986 |
| Interaction term | A product of two features | area·metro_km |
| Homoscedasticity | Residuals have the same spread everywhere | log-price spreads 0.25-0.28 |
| Heteroscedasticity | Residual spread changes: a funnel | 15 → 51 lakh |
| Log transform of the target | Model log(y), then exp back |
np.log(price) |
make_pipeline |
Chains steps into one model (Episode 17) | PolynomialFeatures → LinearRegression |
Summary
LinearRegressionstores the bias inintercept_and the weights incoef_; read each weight as "per unit, holding the others fixed".- MAE (typical miss) and RMSE (punishes big misses) are in the target's units; R² compares the model with always predicting the mean; adjusted R² charges for each feature; the test set has the final word.
- Multicollinearity makes weights wild and meaningless while predictions stay fine.
- Residual plots check the model's shape: a U breaks linearity (add polynomial features), a funnel breaks homoscedasticity (try log of the target).
Next episode: bias, variance and overfitting: why not degree 20? Underfitting, overfitting, the bias-variance trade-off, irreducible error, model capacity, learning curves and validation curves.