Machine Learning Full Course · Episode 18. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. All data and fares are made up for teaching.
The big idea: linear regression draws the straight line (or flat surface) that misses the data by the least mean squared error. You can find it exactly with one formula, the normal equation, or by walking downhill with gradient descent, one batch of rows at a time. Both give exactly what scikit-learn's LinearRegression gives.
The linear model: an auto-rickshaw meter
Picture a meter with made-up rates: ₹30 the moment you sit down, plus ₹15 per km. 8 km → 15 × 8 + 30 = ₹150.
| Meter | Linear model | Name |
|---|---|---|
| ₹30 at 0 km | b |
bias (intercept): the output when every input is 0 |
| ₹15 per km | w |
weight (coefficient): how much the output grows per unit of input |
| fare for 8 km | ŷ = w·x + b |
prediction (ŷ, "y hat") from the hypothesis function |
ŷ = w·x + b (one feature)
ŷ = w1·x1 + w2·x2 + ... + wd·xd + b (d features) = X·w + b
- A linear model is any model whose output is a weighted sum of its inputs plus a bias.
- The hypothesis function is the formula a model uses to turn inputs into a prediction; here, the line.
wandbare parameters (Episode 8): learned from data, not chosen by you. Learning = finding thewandbthat fit best.
"Linear" means linear in the weights: you can still feed it area² as a feature (polynomial features, Episode 19).
Residuals, and why we square them
The data: 200 made-up flats, area in sq ft and rent in ₹ thousands, generated as roughly 0.02 × area + 2 plus noise. The model doesn't know that recipe.
A residual is true value − prediction: how far the line missed one row. The obvious score for a line, "add up the residuals", fails:
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round() # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200) # rent in ₹ thousands
def predict(w, b): return w * area + b # the line
for w, b in [(0.03, 0), (0.0, rent.mean()), (0.02, 2)]:
res = rent - predict(w, b) # residuals
print(f"w={w:<5} b={b:5.1f} sum of residuals {res.sum():8.1f}"
f" MSE {np.mean(res ** 2):6.1f}")
Output:
w=0.03 b= 0.0 sum of residuals -1440.1 MSE 68.2
w=0.0 b= 20.8 sum of residuals -0.0 MSE 37.6
w=0.02 b= 2.0 sum of residuals 24.1 MSE 10.9
The flat line at the average rent (w = 0) has a sum of residuals of zero (−0.0 is just a rounded zero), yet it predicts the same rent for every flat. Misses above and below cancel: +10 and −10 add up to 0. A shopkeeper whose scale gives 100 g too much, then 100 g too little, is not accurate "on average".
Squaring fixes it: every miss counts as positive, and big misses count much more (a miss of 10 → 100, a miss of 20 → 400). The mean squared error is
MSE = (1/n) · Σ (yᵢ − ŷᵢ)²
It's the loss function (Episode 8) of linear regression: lower is better. Now the third line wins (10.9 vs 37.6 vs 68.2). MSE is in squared units (₹ thousands²); its square root, RMSE, is back in normal units (Episode 19).
| Residual | Squared | |
|---|---|---|
| +10 | 100 | a miss above |
| −10 | 100 | a miss below counts the same |
| +20 | 400 | twice the miss, four times the penalty |
The normal equation
Plotted against w and b, MSE is a smooth bowl, a convex function (Episode 6), with one lowest point. At the bottom every slope is zero, so we set the derivatives to zero and solve.
- Put a column of 1s in front of the data matrix
X(Episode 4). Thenbis just the weight of that column, and all parameters fit in one vector θ ("theta") = [b, w₁, ..., w_d]. - Setting the gradient of MSE to zero gives the normal equations
XᵀX θ = Xᵀy, so
θ = (XᵀX)⁻¹ Xᵀ y
- In code, don't invert; solve the system:
np.linalg.solve(X.T @ X, X.T @ y)is faster and more accurate.
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
X = np.c_[np.ones(200), area] # bias column + area
theta = np.linalg.solve(X.T @ X, X.T @ rent) # normal equation
print("normal equation: b =", theta[0].round(3), " w =", theta[1].round(5))
m = LinearRegression().fit(area.reshape(-1, 1), rent)
print("scikit-learn: b =", m.intercept_.round(3), " w =", m.coef_[0].round(5))
print("MSE:", np.mean((X @ theta - rent) ** 2).round(2))
Output:
normal equation: b = 1.527 w = 0.02064
scikit-learn: b = 1.527 w = 0.02064
MSE: 10.84
Exactly scikit-learn's numbers: each extra square foot adds about ₹20.6 of rent, and MSE 10.84 is the lowest any line can reach on this data.
Why not always use it?
| Normal equation | Gradient descent | |
|---|---|---|
| Steps | one | many |
| Answer | exact | approximate, converges to the same |
| Cost with d features | about d³ to solve (double the features → about 8× the work) | about n·d per pass over the data |
| Memory | XᵀX is d × d (100,000 features → 10 billion numbers) | small |
| Works for | linear models only | almost any model (logistic regression, neural networks) |
| Needs scaling | no | yes |
| Fails when | columns are linearly dependent (Episode 16's dummy trap) | the learning rate is too big |
(LinearRegression in scikit-learn uses a least-squares solver based on the same idea; it doesn't run gradient descent.)
Batch gradient descent
Gradient descent (Episode 8): stand somewhere on the bowl, feel the slope, take a step downhill, repeat. For a line with MSE:
err = (w·x + b) − y for every row
∂MSE/∂w = 2 · mean(err · x) (chain rule, Episode 7)
∂MSE/∂b = 2 · mean(err)
w ← w − lr · ∂MSE/∂w b ← b − lr · ∂MSE/∂b
The learning rate lr is the stride. Using all rows to compute every single step is batch gradient descent. Repeat until the MSE stops falling: convergence.
On raw square feet it explodes
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round() # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200) # rent in ₹ thousands
w, b, lr = 0.0, 0.0, 0.01 # start anywhere
for step in range(41):
err = (w * area + b) - rent # prediction - truth
dw = 2 * np.mean(err * area) # slope of MSE w.r.t. w
db = 2 * np.mean(err) # slope of MSE w.r.t. b
w, b = w - lr * dw, b - lr * db # step downhill
if step < 3 or step % 10 == 0:
print(step, f" w = {w:.3g} MSE = {np.mean(err ** 2):.3g}")
Output:
0 w = 413 MSE = 469
1 w = -7.69e+06 MSE = 1.59e+11
2 w = 1.43e+11 MSE = 5.52e+19
10 w = 2.08e+45 MSE = 1.16e+88
20 w = 1.05e+88 MSE = 2.96e+173
30 w = 5.3e+130 MSE = 7.53e+258
40 w = 2.67e+173 MSE = inf
RuntimeWarning: overflow encountered in square
After one step w is 413 (the answer is 0.02); then −7.7 million, 143 billion, ... until the numbers overflow (RuntimeWarning: overflow encountered in square) and MSE becomes inf. This is divergence. The slope ∂MSE/∂w multiplies the error by the area (around 1,000), so each step is huge, overshoots the bottom of the bowl and lands higher on the other side, again and again.
Standardise, then it converges
Episode 16's standardisation gives area mean 0 and std 1, so a learning rate of 0.1 is safe:
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round() # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200) # rent in ₹ thousands
x = (area - area.mean()) / area.std() # standardise (episode 16)
w, b, lr = 0.0, 0.0, 0.1
for step in range(101):
err = (w * x + b) - rent
w -= lr * 2 * np.mean(err * x)
b -= lr * 2 * np.mean(err)
if step % 20 == 0: print(step, " MSE", np.mean(err ** 2).round(2))
w_raw = w / area.std(); b_raw = b - w_raw * area.mean() # back to sq ft
print("gradient descent: b =", round(b_raw, 3), " w =", round(w_raw, 5))
Output:
0 MSE 468.68
20 MSE 10.9
40 MSE 10.84
60 MSE 10.84
80 MSE 10.84
100 MSE 10.84
gradient descent: b = 1.527 w = 0.02064
MSE falls 468.68 → 10.9 → 10.84 and stays there: converged to the normal equation's minimum. The weight learned on the standardised feature is "per standard deviation of area", so we convert back: w_raw = w / std, b_raw = b − w_raw · mean. Same b and w as before.
Stochastic and mini-batch gradient descent
A cook improving a recipe can ask all 200 diners and change it once a day (batch), change it after every diner (stochastic), or after each table (mini-batch).
| Method | Rows per step | Steps per epoch (200 rows) | Behaviour |
|---|---|---|---|
| Batch GD | all (200) | 1 | smooth, accurate, slow on big data |
| Stochastic GD (SGD) | 1 random row | 200 | many fast, noisy steps; bounces around the minimum |
| Mini-batch GD | e.g. 32 (usually 32 to 256) | 7 | the usual choice: enough steps, steady enough to trust, fast on GPUs |
An epoch is one full pass over the training data. "Stochastic" means random: each step uses randomly chosen rows, and we shuffle every epoch so batches differ.
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round() # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200) # rent in ₹ thousands
x = (area - area.mean()) / area.std()
def train(batch, lr=0.05, epochs=20):
w = b = 0.0; updates = 0
for epoch in range(epochs):
order = rng.permutation(200) # shuffle every epoch
for i in range(0, 200, batch):
j = order[i:i + batch] # this batch's rows
err = (w * x[j] + b) - rent[j]
w -= lr * 2 * np.mean(err * x[j]); b -= lr * 2 * np.mean(err)
updates += 1
mse = np.mean((w * x + b - rent) ** 2)
print(f"batch {batch:3}: {updates:4} updates, MSE {mse:.2f}, w {w:.3f}")
for batch in [200, 1, 32]: # batch GD, stochastic GD, mini-batch GD
train(batch)
Output:
batch 200: 20 updates, MSE 17.60, w 4.545
batch 1: 4000 updates, MSE 12.47, w 3.927
batch 32: 140 updates, MSE 10.86, w 5.300
With the same 20 epochs:
- Batch: only 20 updates; MSE 17.60, still on its way down.
- SGD: 4,000 updates, but MSE 12.47 and
woff: each step follows one noisy flat, so with a fixed learning rate it never settles. - Mini-batch (32): 140 updates, MSE 10.86, practically the minimum (10.84).
The cure for SGD's bouncing is a learning-rate schedule that shrinks the stride over time (see practice 4).
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| scoring a line by the sum of residuals | the flat line scores 0 | + and − cancel | MSE |
| forgetting the column of 1s | the line is forced through (0, 0) | no bias | np.c_[np.ones(n), X] |
np.linalg.inv(X.T @ X) |
slow, and junk when columns are dependent | explicit inverse | np.linalg.solve, np.linalg.lstsq |
| gradient descent on raw sq ft | RuntimeWarning: overflow encountered in square, MSE inf |
giant steps → divergence | standardise first |
| learning rate too big (even when scaled) | lr 1.1: MSE 2.58e+07 after 30 steps | overshooting | smaller lr |
| fixed-lr SGD | MSE bounces (12.47 instead of 10.84) | noisy steps | mini-batch or a learning-rate schedule |
| reading weights learned on scaled data as per sq ft | weight 5.3 instead of 0.02 | different units | convert back, or keep the scaler in a pipeline |
theta[0] read as the weight |
quiz option B | the column of 1s comes first | θ = [b, w] |
Mini project: your own LinearRegression class
The same API as scikit-learn (Episode 17): settings in the constructor, fit returns self, learned values end in _. Mini-batch gradient descent, standardisation learned inside fit, and a shrinking learning rate:
import numpy as np
from sklearn.linear_model import LinearRegression
class MyLinearRegression:
"""Linear regression with mini-batch gradient descent; standardises X inside."""
def __init__(self, lr=0.05, epochs=200, batch=32, seed=0):
self.lr, self.epochs, self.batch, self.seed = lr, epochs, batch, seed
def fit(self, X, y):
rng = np.random.default_rng(self.seed)
self.mu_, self.sd_ = X.mean(axis=0), X.std(axis=0) # learned on training data
Z = (X - self.mu_) / self.sd_
n, d = Z.shape; w = np.zeros(d); b = 0.0
for epoch in range(self.epochs):
lr = self.lr / (1 + epoch / 10) # smaller steps later on
order = rng.permutation(n)
for i in range(0, n, self.batch):
j = order[i:i + self.batch]
err = Z[j] @ w + b - y[j]
w -= lr * 2 * Z[j].T @ err / len(j); b -= lr * 2 * err.mean()
self.coef_ = w / self.sd_ # back to original units
self.intercept_ = b - self.coef_ @ self.mu_
return self
def predict(self, X):
return X @ self.coef_ + self.intercept_
rng = np.random.default_rng(18); n = 500
X = np.c_[rng.normal(950, 250, n), rng.uniform(0, 10, n), rng.integers(0, 15, n)] # area, km, floor
y = 0.02 * X[:, 0] - 0.8 * X[:, 1] + 0.3 * X[:, 2] + 6 + rng.normal(0, 3, n)
mine = MyLinearRegression().fit(X, y)
sk = LinearRegression().fit(X, y)
print("mine: ", mine.intercept_.round(3), mine.coef_.round(4))
print("sklearn:", sk.intercept_.round(3), sk.coef_.round(4))
print("MSE mine", np.mean((mine.predict(X) - y) ** 2).round(3), " sklearn", np.mean((sk.predict(X) - y) ** 2).round(3))
Output:
mine: 6.395 [ 0.0193 -0.7567 0.3104]
sklearn: 6.394 [ 0.0193 -0.7537 0.3096]
MSE mine 9.348 sklearn 9.348
Three features (area, km to the metro, floor number), 500 rows, and the result matches scikit-learn to three decimal places of MSE.
Practice
- Add a second feature (km to the metro) to the normal equation.
- Run the standardised gradient descent with
lr = 0.1,0.5and1.1. - Plot the MSE after every update for batch, SGD and mini-batch.
- Make SGD settle by shrinking the learning rate every epoch.
Solution 1
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(18); n = 200
area = rng.normal(950, 250, n).round(); km = rng.uniform(0, 10, n).round(1)
rent = 0.02 * area - 0.8 * km + 6 + rng.normal(0, 3, n) # ₹ thousands
X = np.c_[np.ones(n), area, km] # bias, area, km
theta = np.linalg.solve(X.T @ X, X.T @ rent)
print("b, w_area, w_km:", theta.round(4))
m = LinearRegression().fit(np.c_[area, km], rent)
print("scikit-learn: ", m.intercept_.round(4), m.coef_.round(4))
Output:
b, w_area, w_km: [ 4.7003 0.0209 -0.7236]
scikit-learn: 4.7003 [ 0.0209 -0.7236]
Just one more column in X; θ is now [b, w_area, w_km].
Solution 2
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for lr in [0.1, 0.5, 1.1]:
w = b = 0.0
for step in range(30):
err = (w * x + b) - rent
w -= lr * 2 * np.mean(err * x); b -= lr * 2 * np.mean(err)
print(f"lr {lr}: MSE after 30 steps = {np.mean((w * x + b - rent) ** 2):.4g}")
Output:
lr 0.1: MSE after 30 steps = 10.84
lr 0.5: MSE after 30 steps = 10.84
lr 1.1: MSE after 30 steps = 2.58e+07
For a standardised single feature, the MSE bowl's curvature is 2, so any lr above 1 overshoots further every step and diverges; 0.5 lands at the bottom in one step.
Solution 3
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for batch in [200, 1, 32]:
w = b = 0.0; curve = []
for epoch in range(20):
order = rng.permutation(200)
for i in range(0, 200, batch):
j = order[i:i + batch]
err = (w * x[j] + b) - rent[j]
w -= 0.05 * 2 * np.mean(err * x[j]); b -= 0.05 * 2 * np.mean(err)
curve.append(np.mean((w * x + b - rent) ** 2)) # MSE on all rows
plt.plot(curve, label=f"batch {batch}")
print(f"batch {batch:3}: last 5 MSEs", np.round(curve[-5:], 2))
plt.xscale("log"); plt.yscale("log"); plt.xlabel("update"); plt.ylabel("MSE"); plt.legend()
plt.savefig("mse_curves.png")
Output:
batch 200: last 5 MSEs [26.56 23.57 21.15 19.19 17.6 ]
batch 1: last 5 MSEs [13.17 12.75 12.74 12.33 12.47]
batch 32: last 5 MSEs [10.85 10.87 10.86 10.85 10.86]
The plot (mse_curves.png, log scales) shows batch sliding down slowly and smoothly, SGD dropping fast then jittering, and mini-batch dropping fast and settling.
Solution 4
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for decay in [False, True]:
w = b = 0.0
for epoch in range(20):
lr = 0.05 / (1 + epoch) if decay else 0.05 # shrink the stride each epoch
for i in rng.permutation(200):
err = (w * x[i] + b) - rent[i]
w -= lr * 2 * err * x[i]; b -= lr * 2 * err
print(f"decay {decay!s:5}: MSE {np.mean((w * x + b - rent) ** 2):.2f}")
Output:
decay False: MSE 13.48
decay True : MSE 10.84
With lr = 0.05 / (1 + epoch), the steps shrink and SGD settles at the minimum, 10.84.
Quick quiz
import numpy as np
x = np.array([1.0, 2.0, 3.0])
y = np.array([5.0, 7.0, 9.0])
X = np.c_[np.ones(3), x]
theta = np.linalg.solve(X.T @ X, X.T @ y)
print(theta.round(2))
A) [3. 2.] B) [2. 3.] C) [0. 3.] D) [5. 2.]
Answer
A) [3. 2.]. The points lie exactly on y = 2x + 3. θ lists b first because the column of 1s is the first column of X.
Interview questions
- Write the linear regression model and its loss. ŷ = Xw + b; MSE = (1/n)·Σ(y − ŷ)².
- Why squared error and not the plain sum of errors? Signed errors cancel; squaring makes every miss positive, penalises big misses more, and gives a smooth convex bowl with a closed-form minimum. (It also corresponds to Gaussian noise under maximum likelihood, Episode 12.)
- Derive or state the normal equation. Setting ∇MSE = 0 gives XᵀXθ = Xᵀy, so θ = (XᵀX)⁻¹Xᵀy, with a column of 1s for the bias.
- When would you prefer gradient descent over the normal equation? Many features (the solve costs about d³ and XᵀX takes d² memory), streaming or huge data, or models with no closed form.
- Batch vs stochastic vs mini-batch gradient descent? All rows / one row / a small batch per update; trade-off between accurate smooth steps and many cheap noisy ones; mini-batch is the default.
- Why must features be scaled for gradient descent? Features in big units create huge gradients and an elongated bowl; one learning rate can't suit all directions, so it diverges or crawls.
- What happens if XᵀX is singular? No unique solution (e.g. duplicated or dummy-trap columns); use
lstsq/pseudo-inverse, drop columns, or regularise (ridge, Episode 21). - How do you know gradient descent has converged? The loss stops decreasing (changes below a tolerance), or the gradient is near zero; monitor the loss curve.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Linear model | Output = weighted sum of inputs + bias | ŷ = 0.0206·area + 1.53 |
| Hypothesis function | The formula the model uses to predict | ŷ = w·x + b |
| Weight (coefficient) | How much the output changes per unit of a feature | ₹15 per km; coef_ |
| Bias (intercept) | The output when all inputs are 0 | ₹30; intercept_ |
| Prediction (ŷ) | The model's output for a row | ₹150 for 8 km |
| Parameters | Numbers learned from data | w, b |
| Residual | True value − prediction | +10 or −10 |
| Mean squared error (MSE) | Average of squared residuals | 10.84 |
| Loss function | The number training tries to make small | MSE |
| Convex function | A bowl with one lowest point | MSE of a line |
| Column of 1s (bias column) | Lets the bias be learned as a weight | np.ones(n) |
| θ (theta) | All parameters in one vector | [b, w] |
| Normal equation | θ = (XᵀX)⁻¹Xᵀy, the exact least-squares solution | np.linalg.solve |
np.linalg.solve |
Solves A·x = b without inverting A | solve(X.T @ X, X.T @ y) |
| Gradient | The slopes of the loss with respect to each parameter | ∂MSE/∂w |
| Gradient descent | Repeatedly step against the gradient | w ← w − lr·∂MSE/∂w |
| Learning rate (lr) | The size of each step | 0.1 |
| Batch gradient descent | Every step uses all rows | 20 updates in 20 epochs |
| Stochastic gradient descent (SGD) | Every step uses one random row | 4,000 updates |
| Mini-batch gradient descent | Every step uses a small batch of rows | batch 32 |
| Epoch | One full pass over the training data | 200 rows |
| Shuffle | Random reordering of rows each epoch | rng.permutation(200) |
| Convergence | The loss stops falling; parameters settle | MSE 10.84 |
| Divergence | Steps overshoot and the loss grows | MSE → inf |
| Overflow | A number too big for the computer to store | RuntimeWarning: overflow |
| Learning-rate schedule | Shrinking the learning rate over time | 0.05 / (1 + epoch) |
Summary
- A linear model predicts ŷ = w·x + b: weights (rates) and a bias (starting amount).
- A residual is truth − prediction. Summing residuals lets misses cancel (a useless flat line scored 0); MSE averages the squares.
- The normal equation θ = (XᵀX)⁻¹Xᵀy (with a column of 1s, solved with
np.linalg.solve) gives the exact answer: b = 1.527, w = 0.02064, identical to scikit-learn. - Batch gradient descent walks down the MSE bowl; on raw square feet it diverges to
inf, on standardised data it converges to the same answer. - SGD takes many noisy one-row steps; mini-batch (32 to 256 rows) is the usual choice.
Next episode: linear regression in practice: LinearRegression in scikit-learn, interpreting coefficients, MAE, RMSE, R² and adjusted R², residual plots, linearity, homoscedasticity, multicollinearity and polynomial features.