SyntaxPilot
Courses/Machine Learning/Lesson 18

Machine Learning · Lesson 18 of 90 · 12 min

Linear regression from scratch

linear modelhypothesis functionweights (coefficients)bias (intercept)predictionresidualmean squared error (MSE)normal equationbatch gradient descentstochastic gradient descent (SGD)mini-batch gradient descentimplementing it in NumPy

Machine Learning Full Course · Episode 18. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. All data and fares are made up for teaching.

The big idea: linear regression draws the straight line (or flat surface) that misses the data by the least mean squared error. You can find it exactly with one formula, the normal equation, or by walking downhill with gradient descent, one batch of rows at a time. Both give exactly what scikit-learn's LinearRegression gives.

The linear model: an auto-rickshaw meter

Picture a meter with made-up rates: ₹30 the moment you sit down, plus ₹15 per km. 8 km → 15 × 8 + 30 = ₹150.

Meter Linear model Name
₹30 at 0 km b bias (intercept): the output when every input is 0
₹15 per km w weight (coefficient): how much the output grows per unit of input
fare for 8 km ŷ = w·x + b prediction (ŷ, "y hat") from the hypothesis function
ŷ = w·x + b                          (one feature)
ŷ = w1·x1 + w2·x2 + ... + wd·xd + b   (d features)   =   X·w + b
  • A linear model is any model whose output is a weighted sum of its inputs plus a bias.
  • The hypothesis function is the formula a model uses to turn inputs into a prediction; here, the line.
  • w and b are parameters (Episode 8): learned from data, not chosen by you. Learning = finding the w and b that fit best.

"Linear" means linear in the weights: you can still feed it area² as a feature (polynomial features, Episode 19).

Residuals, and why we square them

The data: 200 made-up flats, area in sq ft and rent in ₹ thousands, generated as roughly 0.02 × area + 2 plus noise. The model doesn't know that recipe.

A residual is true value − prediction: how far the line missed one row. The obvious score for a line, "add up the residuals", fails:

import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()                # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)          # rent in ₹ thousands
def predict(w, b): return w * area + b                  # the line
for w, b in [(0.03, 0), (0.0, rent.mean()), (0.02, 2)]:
    res = rent - predict(w, b)                          # residuals
    print(f"w={w:<5} b={b:5.1f}  sum of residuals {res.sum():8.1f}"
          f"  MSE {np.mean(res ** 2):6.1f}")

Output:

w=0.03  b=  0.0  sum of residuals  -1440.1  MSE   68.2
w=0.0   b= 20.8  sum of residuals     -0.0  MSE   37.6
w=0.02  b=  2.0  sum of residuals     24.1  MSE   10.9

The flat line at the average rent (w = 0) has a sum of residuals of zero (−0.0 is just a rounded zero), yet it predicts the same rent for every flat. Misses above and below cancel: +10 and −10 add up to 0. A shopkeeper whose scale gives 100 g too much, then 100 g too little, is not accurate "on average".

Squaring fixes it: every miss counts as positive, and big misses count much more (a miss of 10 → 100, a miss of 20 → 400). The mean squared error is

MSE = (1/n) · Σ (yᵢ − ŷᵢ)²

It's the loss function (Episode 8) of linear regression: lower is better. Now the third line wins (10.9 vs 37.6 vs 68.2). MSE is in squared units (₹ thousands²); its square root, RMSE, is back in normal units (Episode 19).

Residual Squared
+10 100 a miss above
−10 100 a miss below counts the same
+20 400 twice the miss, four times the penalty

The normal equation

Plotted against w and b, MSE is a smooth bowl, a convex function (Episode 6), with one lowest point. At the bottom every slope is zero, so we set the derivatives to zero and solve.

  1. Put a column of 1s in front of the data matrix X (Episode 4). Then b is just the weight of that column, and all parameters fit in one vector θ ("theta") = [b, w₁, ..., w_d].
  2. Setting the gradient of MSE to zero gives the normal equations XᵀX θ = Xᵀy, so
θ = (XᵀX)⁻¹ Xᵀ y
  1. In code, don't invert; solve the system: np.linalg.solve(X.T @ X, X.T @ y) is faster and more accurate.
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
X = np.c_[np.ones(200), area]                 # bias column + area
theta = np.linalg.solve(X.T @ X, X.T @ rent)  # normal equation
print("normal equation: b =", theta[0].round(3), " w =", theta[1].round(5))
m = LinearRegression().fit(area.reshape(-1, 1), rent)
print("scikit-learn:    b =", m.intercept_.round(3), " w =", m.coef_[0].round(5))
print("MSE:", np.mean((X @ theta - rent) ** 2).round(2))

Output:

normal equation: b = 1.527  w = 0.02064
scikit-learn:    b = 1.527  w = 0.02064
MSE: 10.84

Exactly scikit-learn's numbers: each extra square foot adds about ₹20.6 of rent, and MSE 10.84 is the lowest any line can reach on this data.

Why not always use it?

Normal equation Gradient descent
Steps one many
Answer exact approximate, converges to the same
Cost with d features about d³ to solve (double the features → about 8× the work) about n·d per pass over the data
Memory XᵀX is d × d (100,000 features → 10 billion numbers) small
Works for linear models only almost any model (logistic regression, neural networks)
Needs scaling no yes
Fails when columns are linearly dependent (Episode 16's dummy trap) the learning rate is too big

(LinearRegression in scikit-learn uses a least-squares solver based on the same idea; it doesn't run gradient descent.)

Batch gradient descent

Gradient descent (Episode 8): stand somewhere on the bowl, feel the slope, take a step downhill, repeat. For a line with MSE:

err = (w·x + b) − y                     for every row
∂MSE/∂w = 2 · mean(err · x)             (chain rule, Episode 7)
∂MSE/∂b = 2 · mean(err)
w ← w − lr · ∂MSE/∂w      b ← b − lr · ∂MSE/∂b

The learning rate lr is the stride. Using all rows to compute every single step is batch gradient descent. Repeat until the MSE stops falling: convergence.

On raw square feet it explodes

import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()                # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)          # rent in ₹ thousands
w, b, lr = 0.0, 0.0, 0.01                       # start anywhere
for step in range(41):
    err = (w * area + b) - rent                 # prediction - truth
    dw = 2 * np.mean(err * area)                # slope of MSE w.r.t. w
    db = 2 * np.mean(err)                       # slope of MSE w.r.t. b
    w, b = w - lr * dw, b - lr * db             # step downhill
    if step < 3 or step % 10 == 0:
        print(step, f" w = {w:.3g}  MSE = {np.mean(err ** 2):.3g}")

Output:

0  w = 413  MSE = 469
1  w = -7.69e+06  MSE = 1.59e+11
2  w = 1.43e+11  MSE = 5.52e+19
10  w = 2.08e+45  MSE = 1.16e+88
20  w = 1.05e+88  MSE = 2.96e+173
30  w = 5.3e+130  MSE = 7.53e+258
40  w = 2.67e+173  MSE = inf
RuntimeWarning: overflow encountered in square

After one step w is 413 (the answer is 0.02); then −7.7 million, 143 billion, ... until the numbers overflow (RuntimeWarning: overflow encountered in square) and MSE becomes inf. This is divergence. The slope ∂MSE/∂w multiplies the error by the area (around 1,000), so each step is huge, overshoots the bottom of the bowl and lands higher on the other side, again and again.

Standardise, then it converges

Episode 16's standardisation gives area mean 0 and std 1, so a learning rate of 0.1 is safe:

import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()                # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)          # rent in ₹ thousands
x = (area - area.mean()) / area.std()           # standardise (episode 16)
w, b, lr = 0.0, 0.0, 0.1
for step in range(101):
    err = (w * x + b) - rent
    w -= lr * 2 * np.mean(err * x)
    b -= lr * 2 * np.mean(err)
    if step % 20 == 0: print(step, " MSE", np.mean(err ** 2).round(2))
w_raw = w / area.std(); b_raw = b - w_raw * area.mean()   # back to sq ft
print("gradient descent: b =", round(b_raw, 3), " w =", round(w_raw, 5))

Output:

0  MSE 468.68
20  MSE 10.9
40  MSE 10.84
60  MSE 10.84
80  MSE 10.84
100  MSE 10.84
gradient descent: b = 1.527  w = 0.02064

MSE falls 468.68 → 10.9 → 10.84 and stays there: converged to the normal equation's minimum. The weight learned on the standardised feature is "per standard deviation of area", so we convert back: w_raw = w / std, b_raw = b − w_raw · mean. Same b and w as before.

Stochastic and mini-batch gradient descent

A cook improving a recipe can ask all 200 diners and change it once a day (batch), change it after every diner (stochastic), or after each table (mini-batch).

Method Rows per step Steps per epoch (200 rows) Behaviour
Batch GD all (200) 1 smooth, accurate, slow on big data
Stochastic GD (SGD) 1 random row 200 many fast, noisy steps; bounces around the minimum
Mini-batch GD e.g. 32 (usually 32 to 256) 7 the usual choice: enough steps, steady enough to trust, fast on GPUs

An epoch is one full pass over the training data. "Stochastic" means random: each step uses randomly chosen rows, and we shuffle every epoch so batches differ.

import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()                # 200 flats, sq ft
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)          # rent in ₹ thousands
x = (area - area.mean()) / area.std()
def train(batch, lr=0.05, epochs=20):
    w = b = 0.0; updates = 0
    for epoch in range(epochs):
        order = rng.permutation(200)              # shuffle every epoch
        for i in range(0, 200, batch):
            j = order[i:i + batch]                # this batch's rows
            err = (w * x[j] + b) - rent[j]
            w -= lr * 2 * np.mean(err * x[j]); b -= lr * 2 * np.mean(err)
            updates += 1
    mse = np.mean((w * x + b - rent) ** 2)
    print(f"batch {batch:3}: {updates:4} updates, MSE {mse:.2f}, w {w:.3f}")
for batch in [200, 1, 32]:     # batch GD, stochastic GD, mini-batch GD
    train(batch)

Output:

batch 200:   20 updates, MSE 17.60, w 4.545
batch   1: 4000 updates, MSE 12.47, w 3.927
batch  32:  140 updates, MSE 10.86, w 5.300

With the same 20 epochs:

  • Batch: only 20 updates; MSE 17.60, still on its way down.
  • SGD: 4,000 updates, but MSE 12.47 and w off: each step follows one noisy flat, so with a fixed learning rate it never settles.
  • Mini-batch (32): 140 updates, MSE 10.86, practically the minimum (10.84).

The cure for SGD's bouncing is a learning-rate schedule that shrinks the stride over time (see practice 4).

Common mistakes

Mistake Real symptom Why Fix
scoring a line by the sum of residuals the flat line scores 0 + and − cancel MSE
forgetting the column of 1s the line is forced through (0, 0) no bias np.c_[np.ones(n), X]
np.linalg.inv(X.T @ X) slow, and junk when columns are dependent explicit inverse np.linalg.solve, np.linalg.lstsq
gradient descent on raw sq ft RuntimeWarning: overflow encountered in square, MSE inf giant steps → divergence standardise first
learning rate too big (even when scaled) lr 1.1: MSE 2.58e+07 after 30 steps overshooting smaller lr
fixed-lr SGD MSE bounces (12.47 instead of 10.84) noisy steps mini-batch or a learning-rate schedule
reading weights learned on scaled data as per sq ft weight 5.3 instead of 0.02 different units convert back, or keep the scaler in a pipeline
theta[0] read as the weight quiz option B the column of 1s comes first θ = [b, w]

Mini project: your own LinearRegression class

The same API as scikit-learn (Episode 17): settings in the constructor, fit returns self, learned values end in _. Mini-batch gradient descent, standardisation learned inside fit, and a shrinking learning rate:

import numpy as np
from sklearn.linear_model import LinearRegression

class MyLinearRegression:
    """Linear regression with mini-batch gradient descent; standardises X inside."""
    def __init__(self, lr=0.05, epochs=200, batch=32, seed=0):
        self.lr, self.epochs, self.batch, self.seed = lr, epochs, batch, seed
    def fit(self, X, y):
        rng = np.random.default_rng(self.seed)
        self.mu_, self.sd_ = X.mean(axis=0), X.std(axis=0)     # learned on training data
        Z = (X - self.mu_) / self.sd_
        n, d = Z.shape; w = np.zeros(d); b = 0.0
        for epoch in range(self.epochs):
            lr = self.lr / (1 + epoch / 10)                     # smaller steps later on
            order = rng.permutation(n)
            for i in range(0, n, self.batch):
                j = order[i:i + self.batch]
                err = Z[j] @ w + b - y[j]
                w -= lr * 2 * Z[j].T @ err / len(j); b -= lr * 2 * err.mean()
        self.coef_ = w / self.sd_                               # back to original units
        self.intercept_ = b - self.coef_ @ self.mu_
        return self
    def predict(self, X):
        return X @ self.coef_ + self.intercept_

rng = np.random.default_rng(18); n = 500
X = np.c_[rng.normal(950, 250, n), rng.uniform(0, 10, n), rng.integers(0, 15, n)]  # area, km, floor
y = 0.02 * X[:, 0] - 0.8 * X[:, 1] + 0.3 * X[:, 2] + 6 + rng.normal(0, 3, n)
mine = MyLinearRegression().fit(X, y)
sk = LinearRegression().fit(X, y)
print("mine:   ", mine.intercept_.round(3), mine.coef_.round(4))
print("sklearn:", sk.intercept_.round(3), sk.coef_.round(4))
print("MSE mine", np.mean((mine.predict(X) - y) ** 2).round(3), " sklearn", np.mean((sk.predict(X) - y) ** 2).round(3))

Output:

mine:    6.395 [ 0.0193 -0.7567  0.3104]
sklearn: 6.394 [ 0.0193 -0.7537  0.3096]
MSE mine 9.348  sklearn 9.348

Three features (area, km to the metro, floor number), 500 rows, and the result matches scikit-learn to three decimal places of MSE.

Practice

  1. Add a second feature (km to the metro) to the normal equation.
  2. Run the standardised gradient descent with lr = 0.1, 0.5 and 1.1.
  3. Plot the MSE after every update for batch, SGD and mini-batch.
  4. Make SGD settle by shrinking the learning rate every epoch.
Solution 1
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(18); n = 200
area = rng.normal(950, 250, n).round(); km = rng.uniform(0, 10, n).round(1)
rent = 0.02 * area - 0.8 * km + 6 + rng.normal(0, 3, n)       # ₹ thousands
X = np.c_[np.ones(n), area, km]                              # bias, area, km
theta = np.linalg.solve(X.T @ X, X.T @ rent)
print("b, w_area, w_km:", theta.round(4))
m = LinearRegression().fit(np.c_[area, km], rent)
print("scikit-learn:   ", m.intercept_.round(4), m.coef_.round(4))

Output:

b, w_area, w_km: [ 4.7003  0.0209 -0.7236]
scikit-learn:    4.7003 [ 0.0209 -0.7236]

Just one more column in X; θ is now [b, w_area, w_km].

Solution 2
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for lr in [0.1, 0.5, 1.1]:
    w = b = 0.0
    for step in range(30):
        err = (w * x + b) - rent
        w -= lr * 2 * np.mean(err * x); b -= lr * 2 * np.mean(err)
    print(f"lr {lr}: MSE after 30 steps = {np.mean((w * x + b - rent) ** 2):.4g}")

Output:

lr 0.1: MSE after 30 steps = 10.84
lr 0.5: MSE after 30 steps = 10.84
lr 1.1: MSE after 30 steps = 2.58e+07

For a standardised single feature, the MSE bowl's curvature is 2, so any lr above 1 overshoots further every step and diverges; 0.5 lands at the bottom in one step.

Solution 3
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for batch in [200, 1, 32]:
    w = b = 0.0; curve = []
    for epoch in range(20):
        order = rng.permutation(200)
        for i in range(0, 200, batch):
            j = order[i:i + batch]
            err = (w * x[j] + b) - rent[j]
            w -= 0.05 * 2 * np.mean(err * x[j]); b -= 0.05 * 2 * np.mean(err)
            curve.append(np.mean((w * x + b - rent) ** 2))      # MSE on all rows
    plt.plot(curve, label=f"batch {batch}")
    print(f"batch {batch:3}: last 5 MSEs", np.round(curve[-5:], 2))
plt.xscale("log"); plt.yscale("log"); plt.xlabel("update"); plt.ylabel("MSE"); plt.legend()
plt.savefig("mse_curves.png")

Output:

batch 200: last 5 MSEs [26.56 23.57 21.15 19.19 17.6 ]
batch   1: last 5 MSEs [13.17 12.75 12.74 12.33 12.47]
batch  32: last 5 MSEs [10.85 10.87 10.86 10.85 10.86]

The plot (mse_curves.png, log scales) shows batch sliding down slowly and smoothly, SGD dropping fast then jittering, and mini-batch dropping fast and settling.

Solution 4
import numpy as np
rng = np.random.default_rng(18)
area = rng.normal(950, 250, 200).round()
rent = 0.02 * area + 2 + rng.normal(0, 3, 200)
x = (area - area.mean()) / area.std()
for decay in [False, True]:
    w = b = 0.0
    for epoch in range(20):
        lr = 0.05 / (1 + epoch) if decay else 0.05            # shrink the stride each epoch
        for i in rng.permutation(200):
            err = (w * x[i] + b) - rent[i]
            w -= lr * 2 * err * x[i]; b -= lr * 2 * err
    print(f"decay {decay!s:5}: MSE {np.mean((w * x + b - rent) ** 2):.2f}")

Output:

decay False: MSE 13.48
decay True : MSE 10.84

With lr = 0.05 / (1 + epoch), the steps shrink and SGD settles at the minimum, 10.84.

Quick quiz

import numpy as np

x = np.array([1.0, 2.0, 3.0])
y = np.array([5.0, 7.0, 9.0])
X = np.c_[np.ones(3), x]
theta = np.linalg.solve(X.T @ X, X.T @ y)
print(theta.round(2))

A) [3. 2.] B) [2. 3.] C) [0. 3.] D) [5. 2.]

Answer

A) [3. 2.]. The points lie exactly on y = 2x + 3. θ lists b first because the column of 1s is the first column of X.

Interview questions

  1. Write the linear regression model and its loss. ŷ = Xw + b; MSE = (1/n)·Σ(y − ŷ)².
  2. Why squared error and not the plain sum of errors? Signed errors cancel; squaring makes every miss positive, penalises big misses more, and gives a smooth convex bowl with a closed-form minimum. (It also corresponds to Gaussian noise under maximum likelihood, Episode 12.)
  3. Derive or state the normal equation. Setting ∇MSE = 0 gives XᵀXθ = Xᵀy, so θ = (XᵀX)⁻¹Xᵀy, with a column of 1s for the bias.
  4. When would you prefer gradient descent over the normal equation? Many features (the solve costs about d³ and XᵀX takes d² memory), streaming or huge data, or models with no closed form.
  5. Batch vs stochastic vs mini-batch gradient descent? All rows / one row / a small batch per update; trade-off between accurate smooth steps and many cheap noisy ones; mini-batch is the default.
  6. Why must features be scaled for gradient descent? Features in big units create huge gradients and an elongated bowl; one learning rate can't suit all directions, so it diverges or crawls.
  7. What happens if XᵀX is singular? No unique solution (e.g. duplicated or dummy-trap columns); use lstsq/pseudo-inverse, drop columns, or regularise (ridge, Episode 21).
  8. How do you know gradient descent has converged? The loss stops decreasing (changes below a tolerance), or the gradient is near zero; monitor the loss curve.

Glossary

Term Meaning Example
Linear model Output = weighted sum of inputs + bias ŷ = 0.0206·area + 1.53
Hypothesis function The formula the model uses to predict ŷ = w·x + b
Weight (coefficient) How much the output changes per unit of a feature ₹15 per km; coef_
Bias (intercept) The output when all inputs are 0 ₹30; intercept_
Prediction (ŷ) The model's output for a row ₹150 for 8 km
Parameters Numbers learned from data w, b
Residual True value − prediction +10 or −10
Mean squared error (MSE) Average of squared residuals 10.84
Loss function The number training tries to make small MSE
Convex function A bowl with one lowest point MSE of a line
Column of 1s (bias column) Lets the bias be learned as a weight np.ones(n)
θ (theta) All parameters in one vector [b, w]
Normal equation θ = (XᵀX)⁻¹Xᵀy, the exact least-squares solution np.linalg.solve
np.linalg.solve Solves A·x = b without inverting A solve(X.T @ X, X.T @ y)
Gradient The slopes of the loss with respect to each parameter ∂MSE/∂w
Gradient descent Repeatedly step against the gradient w ← w − lr·∂MSE/∂w
Learning rate (lr) The size of each step 0.1
Batch gradient descent Every step uses all rows 20 updates in 20 epochs
Stochastic gradient descent (SGD) Every step uses one random row 4,000 updates
Mini-batch gradient descent Every step uses a small batch of rows batch 32
Epoch One full pass over the training data 200 rows
Shuffle Random reordering of rows each epoch rng.permutation(200)
Convergence The loss stops falling; parameters settle MSE 10.84
Divergence Steps overshoot and the loss grows MSE → inf
Overflow A number too big for the computer to store RuntimeWarning: overflow
Learning-rate schedule Shrinking the learning rate over time 0.05 / (1 + epoch)

Summary

  • A linear model predicts ŷ = w·x + b: weights (rates) and a bias (starting amount).
  • A residual is truth − prediction. Summing residuals lets misses cancel (a useless flat line scored 0); MSE averages the squares.
  • The normal equation θ = (XᵀX)⁻¹Xᵀy (with a column of 1s, solved with np.linalg.solve) gives the exact answer: b = 1.527, w = 0.02064, identical to scikit-learn.
  • Batch gradient descent walks down the MSE bowl; on raw square feet it diverges to inf, on standardised data it converges to the same answer.
  • SGD takes many noisy one-row steps; mini-batch (32 to 256 rows) is the usual choice.

Next episode: linear regression in practice: LinearRegression in scikit-learn, interpreting coefficients, MAE, RMSE, R² and adjusted R², residual plots, linearity, homoscedasticity, multicollinearity and polynomial features.