Machine Learning Full Course · Episode 5. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output. You need episode 3 (vectors, dot product, norms, unit vectors) and episode 4 (matrices, @, transpose, rank).
The problem: how far along the road?
Raju's chai stall starts home delivery. On a map with the stall at the origin (0, 0), east and north in km, the delivery boy only rides along one straight road that goes 3 km east for every 4 km north: the road is the direction (3, 4). Priya lives at (0, 5). How far along the road should he ride to be nearest her gate?
The obvious answer fails
The road starts at the stall and heads roughly east, so use Priya's east coordinate:
import numpy as np
# road runs north-east from the stall; places as (east km, north km)
road = np.array([3, 4])
priya = np.array([0, 5])
print("just x:", priya[0])
u = road / np.linalg.norm(road) # unit vector along the road
print("along road:", priya @ u)
print("shadow point:", (priya @ u) * u)
Output:
just x: 0
along road: 4.0
shadow point: [2.4 3.2]
"Just x" says 0 km, as if Priya lived at the stall, though she's 5 km away. Why: x measures distance along east; the road points another way. We must measure along the road's own direction.
Projection: a shadow
Imagine the sun shining at a right angle to the road. Priya's house casts a shadow on the road: the road point nearest her house. That shadow is the projection of the vector v (her position) onto the direction u.
| Piece | Formula | Here |
|---|---|---|
| unit vector along the road | u = road / ‖road‖ | (3, 4) / 5 = (0.6, 0.8) |
| distance along the road (scalar projection) | v · u | 0×0.6 + 5×0.8 = 4.0 km |
| the shadow point (vector projection) | (v · u) u | 4 × (0.6, 0.8) = (2.4, 3.2) |
u must have length 1. With the raw road vector, priya @ road = 20, five times too big (scaled by ‖road‖ = 5).
Where it's used: a linear model's score is a projection of the data onto a weight direction; PCA (episode 48) picks the directions in which the projections (shadows) of the data are most spread out.
Orthogonal vectors and an orthonormal basis
Raju also wants to know how far Priya is sideways from the road: along the direction across it, at a right angle: (−4, 3) / 5.
| Term | Meaning | Example |
|---|---|---|
| Orthogonal | at right angles; test: dot product = 0 | east·north = 0; along·across = 0 |
| Basis | a set of directions every point can be built from | east and north: every place = a·east + b·north |
| Orthonormal basis | basis directions that are all orthogonal to each other and each of length 1 | along = (0.6, 0.8), across = (−0.8, 0.6) |
| Coordinates in an orthonormal basis | just the dot products with each direction | Priya = (v·along, v·across) |
import numpy as np
east = np.array([1, 0]); north = np.array([0, 1])
along = np.array([3, 4]) / 5 # along the road
across = np.array([-4, 3]) / 5 # across the road
print(east @ north, round(along @ across, 3))
print(np.linalg.norm(along), np.linalg.norm(across))
priya = np.array([0, 5])
print("road coords:", priya @ along, priya @ across)
Output:
0 0.0
1.0 1.0
road coords: 4.0 3.0
Priya is 4 km along and 3 km across. √(4² + 3²) = 5 km, the same distance as before: turning an orthonormal grid changes the coordinates, never the lengths. (Without round, along @ across prints a tiny rounding leftover such as 2.66e-17.)
Special directions: eigenvectors
From episode 4, a matrix is a machine that moves points. Roll chapati dough in one direction: a line drawn along the roll gets longer but keeps its direction; a slanted line gets longer and tilts. A matrix's special directions that are only stretched, never turned, are its eigenvectors ("eigen" is German for "own"); the stretch factor is the eigenvalue λ (lambda).
import numpy as np
A = np.array([[3, 1],
[1, 3]])
for v in [[1, 0], [0, 1], [1, 1], [1, -1]]:
print(v, "->", A @ v)
Output:
[1, 0] -> [3 1]
[0, 1] -> [1 3]
[1, 1] -> [4 4]
[1, -1] -> [ 2 -2]
(1, 0) and (0, 1) are tilted. (1, 1) → (4, 4) = 4 × (1, 1): an eigenvector with λ = 4. (1, −1) → (2, −2) = 2 × (1, −1): an eigenvector with λ = 2.
Definition: A v = λ v, with v ≠ 0. On its eigenvectors, the matrix acts like the plain number λ.
| λ | Effect on the eigenvector |
|---|---|
| λ > 1 | stretched |
| 0 < λ < 1 | shrunk |
| λ = 0 | squashed to zero (the matrix is singular, episode 4) |
| λ < 0 | flipped to point the opposite way (and scaled by |λ|) |
Any multiple of an eigenvector (like (2, 2)) is the same direction, so libraries return the one of length 1.
Symmetric matrices and np.linalg.eigh
A matrix that equals its own transpose (A = Aᵀ, a mirror image across the main diagonal) is symmetric. The most important ML matrices are symmetric (the covariance matrix in episode 11, the matrix behind PCA in episode 48). Symmetric matrices guarantee real eigenvalues and orthogonal eigenvectors. np.linalg.eigh ("H" for Hermitian, which for real numbers means symmetric) is the routine for them.
import numpy as np
A = np.array([[3, 1],
[1, 3]])
print((A == A.T).all()) # symmetric: A equals its transpose
vals, vecs = np.linalg.eigh(A) # eigenvalues and eigenvectors
print(vals)
print(vecs.round(3))
v = vecs[:, 1] # column 1 = eigenvector for 4
print(A @ v, vals[1] * v)
Output:
True
[2. 4.]
[[-0.707 0.707]
[ 0.707 0.707]]
[2.82842712 2.82842712] [2.82842712 2.82842712]
- Eigenvalues come out smallest first: 2, 4.
- Eigenvectors are the columns of
vecs: (−0.707, 0.707) is (1, −1)/√2 flipped (same line), (0.707, 0.707) is (1, 1)/√2. - The check
A @ v == 4 * vholds.
For a general (non-symmetric) square matrix use np.linalg.eig; its eigenvalues can be complex numbers (for example, a rotation turns every direction, so it has no real eigenvector).
Eigendecomposition: turn, stretch, turn back
Any symmetric matrix splits into three simple moves:
- Qᵀ: turn so the eigenvectors line up with the axes;
- Λ (capital lambda): stretch each axis by its eigenvalue, a diagonal matrix (numbers only on the main diagonal);
- Q: turn back.
A = Q Λ Qᵀ is the eigendecomposition (Q holds the eigenvectors as columns).
import numpy as np
A = np.array([[3, 1],
[1, 3]])
vals, Q = np.linalg.eigh(A)
print(round(Q[:, 0] @ Q[:, 1], 3)) # eigenvectors are orthogonal
L = np.diag(vals) # eigenvalues on a diagonal
print(L)
print(Q @ L @ Q.T) # eigendecomposition rebuilds A
Output:
0.0
[[2. 0.]
[0. 4.]]
[[3. 1.]
[1. 3.]]
Bonus facts worth knowing: the determinant equals the product of the eigenvalues (2 × 4 = 8 = 3·3 − 1·1), and a matrix is singular exactly when some eigenvalue is 0.
Real data isn't square
Raju's notebook: 6 customers × 4 items (chai, samosa, cake, pastry); some are chai people, some cake people.
import numpy as np
# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
[6, 5, 1, 0],
[4, 4, 0, 0],
[0, 1, 5, 4],
[1, 0, 6, 5],
[0, 0, 4, 4]])
print(orders.shape)
vals, vecs = np.linalg.eigh(orders)
Output:
(6, 4)
Traceback (most recent call last):
...
numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square
Eigenvectors exist only for square matrices: a 6 × 4 matrix takes 4 numbers in and gives 6 out, so "the same direction" makes no sense. Almost no dataset is square.
Singular value decomposition (SVD)
Any matrix, any shape, can be written X = U Σ Vᵀ:
| Part | Shape here (compact) | Meaning |
|---|---|---|
| Vᵀ | 4 × 4 | orthonormal directions in item space (rows = item patterns) |
| Σ (sigma) | 4 values | a diagonal of singular values: stretch factors, never negative, sorted biggest first |
| U | 6 × 4 | orthonormal directions in customer space (how much each customer has of each pattern) |
It is "turn, stretch, turn" for rectangles. A big singular value = an important pattern; a tiny one = mostly noise.
import numpy as np
# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
[6, 5, 1, 0],
[4, 4, 0, 0],
[0, 1, 5, 4],
[1, 0, 6, 5],
[0, 0, 4, 4]])
U, s, Vt = np.linalg.svd(orders, full_matrices=False)
print(U.shape, s.shape, Vt.shape)
print(s.round(2)) # singular values, biggest first
print(Vt[:2].round(2)) # top 2 patterns
print(np.allclose(U @ np.diag(s) @ Vt, orders))
Output:
(6, 4) (4,) (4, 4)
[12.45 10.7 1.2 1.03]
[[-0.54 -0.46 -0.54 -0.46]
[-0.53 -0.46 0.53 0.46]]
True
- Two big singular values (12.45, 10.7) and two tiny ones (1.2, 1.03): the table really has two patterns.
- Pattern 1 has the same sign for every item: overall appetite. Pattern 2 is − for chai/samosa and + for cake/pastry: chai side vs cake side. Nobody told SVD about tastes; it found them.
- Signs can come out flipped (a pattern and its negative are the same direction);
full_matrices=Falsegives the compact shapes. np.allcloseconfirms the three parts rebuild the table.
Connection: the singular values of X are the square roots of the eigenvalues of the symmetric matrix XᵀX, and V holds its eigenvectors. That's why SVD and eigenvectors appear together in PCA.
Low-rank approximation
Keeping only the top k patterns (first k columns of U, k singular values, first k rows of Vᵀ) gives a rank-k approximation: the closest possible rank-k matrix to the original.
import numpy as np
# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
[6, 5, 1, 0],
[4, 4, 0, 0],
[0, 1, 5, 4],
[1, 0, 6, 5],
[0, 0, 4, 4]])
# obvious way to shrink: keep only 2 columns
keep = orders.copy(); keep[:, 2:] = 0
print("drop columns error:", np.linalg.norm(orders - keep).round(2))
U, s, Vt = np.linalg.svd(orders, full_matrices=False)
k = 2
approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
print("rank-2 SVD error: ", np.linalg.norm(orders - approx).round(2))
print(approx.round(1))
Output:
drop columns error: 11.66
rank-2 SVD error: 1.58
[[ 4.8 4.2 0.5 0.4]
[ 5.9 5.1 0.6 0.5]
[ 4.3 3.7 0. -0. ]
[ 0.5 0.4 4.8 4.2]
[ 0.6 0.5 5.9 5.1]
[ 0. -0. 4.3 3.7]]
Dropping columns wipes out the cake lovers (error 11.66); rank-2 SVD keeps both tastes (error 1.58, about 7× smaller) with the same budget of two patterns. The error is the norm of the difference (episode 3; for matrices this is called the Frobenius norm). The -0. entries are tiny negatives rounded.
Storage arithmetic: a 1000 × 1000 table (say a greyscale photo) is 1,000,000 numbers. Rank 50 stores 50 × 1000 (U) + 50 (Σ) + 50 × 1000 (Vᵀ) = 100,050 numbers: about a tenth. This is the idea behind SVD-based image compression and filling gaps in ratings tables for recommendations (episode 54).
Common mistakes (and the real error messages)
| Mistake | Error / result | Why | Fix |
|---|---|---|---|
| distance along a road = x coordinate | Priya: 0 km | the road isn't the x axis | project: v @ (road / np.linalg.norm(road)) |
| projecting onto a non-unit vector | priya @ road = 20, not 4 |
scaled by ‖road‖ = 5 | divide by the norm first |
np.linalg.eigh(orders) on a 6 × 4 matrix |
numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square |
eigenvectors need a square matrix | np.linalg.svd(orders) |
| reading eigenvectors as rows | wrong vectors | NumPy returns them as columns | vecs[:, i] |
expecting eigh values biggest first |
[2. 4.] |
eigh sorts ascending; svd sorts descending |
vals[::-1] if needed |
eigh on a non-symmetric matrix |
silently wrong answer | eigh only reads one triangle |
check (A == A.T).all(); else np.linalg.eig |
| shrinking data by deleting columns | error 11.66 | throws away whole patterns | rank-k SVD: error 1.58 |
Mini project: how many patterns do you need?
import numpy as np
orders = np.array([[5, 4, 0, 1],
[6, 5, 1, 0],
[4, 4, 0, 0],
[0, 1, 5, 4],
[1, 0, 6, 5],
[0, 0, 4, 4]])
U, s, Vt = np.linalg.svd(orders, full_matrices=False)
for k in [1, 2, 3, 4]:
approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
err = np.linalg.norm(orders - approx)
kept = (s[:k] ** 2).sum() / (s ** 2).sum()
print(k, round(err, 2), f"{kept:.1%}")
Output:
1 10.82 57.0%
2 1.58 99.1%
3 1.03 99.6%
4 0.0 100.0%
kept is the share of the table's total "energy" (sum of squared singular values) the top k patterns hold. Going from 1 to 2 patterns jumps from 57% to 99.1%; after that, extra patterns add almost nothing. Also note: the error equals √(sum of the dropped s²), e.g. √(1.2² + 1.03²) ≈ 1.58. Choosing k by "how much is kept" is exactly how the number of components is chosen in PCA.
Practice
- Project (6, 8) onto (1, 0) and onto (0, 1).
Solution
v @ np.array([1, 0]) → 6, v @ np.array([0, 1]) → 8. Projecting onto the standard axes just reads off the coordinates; (1, 0) and (0, 1) are an orthonormal basis.
- Eigenvalues of
[[4, 0], [0, 1]]without NumPy.
Solution
A diagonal matrix stretches the x axis by 4 and the y axis by 1, so the eigenvectors are (1, 0) and (0, 1) with eigenvalues 4 and 1. np.linalg.eigh prints [1. 4.] (ascending).
- Add a 7th customer
[3, 3, 3, 3](likes everything) and re-run SVD.
Solution
np.linalg.svd(orders7, compute_uv=False).round(2) → [13.81 10.7 1.2 1.11]. Only the first value (appetite) grew: an all-round eater adds to "how much", not to "which side".
- Is (−4, 3) orthogonal to (3, 4)? Is (2, 1) orthogonal to (1, 2)?
Solution
(−4)(3) + (3)(4) = 0: yes. 2·1 + 1·2 = 4 ≠ 0: no.
Quick quiz
import numpy as np
A = np.array([[2, 0],
[0, 5]])
v = np.array([0, 1])
print(A @ v)
A) [0 5]: v is an eigenvector, λ = 5 B) [2 5]: not an eigenvector C) [0 2]: λ = 2 D) [5 0]: v was turned
Answer
A. A @ (0, 1) = (0, 5) = 5 × (0, 1): same direction, stretched 5×. For a diagonal matrix the axes are eigenvectors and the diagonal entries are the eigenvalues.
Interview questions
- What is an eigenvector? A non-zero vector v with A v = λ v: the matrix only scales it; λ is the eigenvalue.
- Why are symmetric matrices special? Real eigenvalues, orthogonal eigenvectors, and the decomposition A = Q Λ Qᵀ always exists.
- What is the difference between eigendecomposition and SVD? Eigendecomposition needs a square (diagonalisable) matrix; SVD works for any m × n matrix and always gives orthonormal U, V and non-negative singular values.
- How are SVD and PCA related? PCA's directions are the right singular vectors (V) of the centred data matrix, equivalently the eigenvectors of its covariance matrix; variances are s²/(n − 1).
- What does a low-rank approximation give you? The best rank-k approximation in the least-squares (Frobenius) sense (Eckart–Young theorem): compression and noise removal.
- What is a projection? The closest point on a line (or subspace); onto a unit vector u it is (v · u) u.
- What does orthogonality mean in data? Directions with zero dot product: independent axes that don't overlap, so coordinates along them can be read off separately.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Origin | The zero point (0, 0) of a map or space | Raju's stall |
| Direction | Which way a vector points | the road (3, 4) |
| Projection | The shadow of a vector on a direction: the closest point on it | (2.4, 3.2) |
| Scalar projection | How far along the direction: v · u | 4.0 km |
| Orthogonal | At right angles; dot product 0 | along · across = 0 |
| Basis | Directions every point is built from | east, north |
| Orthonormal basis | Orthogonal basis directions of length 1 | along, across |
| Coordinates | A point's amounts along the basis directions | Priya = (4, 3) in road coords |
| Eigenvector | A non-zero vector a matrix only stretches (A v = λ v) | (1, 1) for [[3, 1], [1, 3]] |
| Eigenvalue (λ) | The stretch factor of an eigenvector | 4 and 2 |
| Symmetric matrix | Equals its transpose (A = Aᵀ) | [[3, 1], [1, 3]] |
| Hermitian | The complex-number version of symmetric; for real matrices, the same thing | the H in eigh |
np.linalg.eigh |
Eigenvalues (ascending) and eigenvectors (columns) of a symmetric matrix | vals, vecs = eigh(A) |
| Diagonal matrix | Non-zero numbers only on the main diagonal | np.diag([2, 4]) |
| Eigendecomposition | A = Q Λ Qᵀ: turn, stretch, turn back | rebuilds [[3, 1], [1, 3]] |
| Square matrix | Same number of rows and columns | 2 × 2 |
| SVD | Any matrix = U Σ Vᵀ | np.linalg.svd(orders) |
| Singular values | The non-negative stretch factors in Σ, biggest first; the importance of each pattern | 12.45, 10.7, 1.2, 1.03 |
| Singular vectors | The columns of U (left) and rows of Vᵀ (right): the patterns | appetite; chai vs cake |
| Rank-k / low-rank approximation | Keep only the top k patterns of the SVD | rank 2, error 1.58 |
| Frobenius norm | The L2 norm of all a matrix's entries, used as the error | np.linalg.norm(orders - approx) |
| Energy kept | Share of Σ s² held by the top k singular values | 99.1% at k = 2 |
np.allclose |
Equality test that tolerates tiny rounding | rebuild check → True |
Summary
- A projection is a shadow: v · u measures how far v goes along the unit direction u.
- Orthogonal = dot product 0; an orthonormal basis is a turned grid of unit directions, and coordinates in it are dot products.
- Eigenvectors are directions a matrix only stretches; eigenvalues are the stretch factors: A v = λ v.
- Symmetric matrices have real eigenvalues and orthogonal eigenvectors: A = Q Λ Qᵀ (turn, stretch, turn back).
- SVD, X = U Σ Vᵀ, works for any shape; singular values rank the patterns hidden in the data.
- A low-rank approximation keeps the top k patterns: far better than deleting columns, and the basis of compression, recommendations and PCA.
Next episode: Maths for ML 4, derivatives: functions, slope, rate of change, the derivative rules, the chain rule and finding the best value.