SyntaxPilot
Courses/Machine Learning/Lesson 5

Machine Learning · Lesson 5 of 90 · 14 min

Maths 3: eigenvectors and SVD

orthogonal vectorsorthonormal basisprojectioneigenvalueeigenvectoreigendecompositionsymmetric matricessingular value decomposition (SVD)singular valueslow-rank approximation

Machine Learning Full Course · Episode 5. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output. You need episode 3 (vectors, dot product, norms, unit vectors) and episode 4 (matrices, @, transpose, rank).

The problem: how far along the road?

Raju's chai stall starts home delivery. On a map with the stall at the origin (0, 0), east and north in km, the delivery boy only rides along one straight road that goes 3 km east for every 4 km north: the road is the direction (3, 4). Priya lives at (0, 5). How far along the road should he ride to be nearest her gate?

The obvious answer fails

The road starts at the stall and heads roughly east, so use Priya's east coordinate:

import numpy as np

# road runs north-east from the stall; places as (east km, north km)
road = np.array([3, 4])
priya = np.array([0, 5])

print("just x:", priya[0])
u = road / np.linalg.norm(road)    # unit vector along the road
print("along road:", priya @ u)
print("shadow point:", (priya @ u) * u)

Output:

just x: 0
along road: 4.0
shadow point: [2.4 3.2]

"Just x" says 0 km, as if Priya lived at the stall, though she's 5 km away. Why: x measures distance along east; the road points another way. We must measure along the road's own direction.

Projection: a shadow

Imagine the sun shining at a right angle to the road. Priya's house casts a shadow on the road: the road point nearest her house. That shadow is the projection of the vector v (her position) onto the direction u.

Piece Formula Here
unit vector along the road u = road / ‖road‖ (3, 4) / 5 = (0.6, 0.8)
distance along the road (scalar projection) v · u 0×0.6 + 5×0.8 = 4.0 km
the shadow point (vector projection) (v · u) u 4 × (0.6, 0.8) = (2.4, 3.2)

u must have length 1. With the raw road vector, priya @ road = 20, five times too big (scaled by ‖road‖ = 5).

Where it's used: a linear model's score is a projection of the data onto a weight direction; PCA (episode 48) picks the directions in which the projections (shadows) of the data are most spread out.

Orthogonal vectors and an orthonormal basis

Raju also wants to know how far Priya is sideways from the road: along the direction across it, at a right angle: (−4, 3) / 5.

Term Meaning Example
Orthogonal at right angles; test: dot product = 0 east·north = 0; along·across = 0
Basis a set of directions every point can be built from east and north: every place = a·east + b·north
Orthonormal basis basis directions that are all orthogonal to each other and each of length 1 along = (0.6, 0.8), across = (−0.8, 0.6)
Coordinates in an orthonormal basis just the dot products with each direction Priya = (v·along, v·across)
import numpy as np

east = np.array([1, 0]); north = np.array([0, 1])
along = np.array([3, 4]) / 5      # along the road
across = np.array([-4, 3]) / 5    # across the road

print(east @ north, round(along @ across, 3))
print(np.linalg.norm(along), np.linalg.norm(across))

priya = np.array([0, 5])
print("road coords:", priya @ along, priya @ across)

Output:

0 0.0
1.0 1.0
road coords: 4.0 3.0

Priya is 4 km along and 3 km across. √(4² + 3²) = 5 km, the same distance as before: turning an orthonormal grid changes the coordinates, never the lengths. (Without round, along @ across prints a tiny rounding leftover such as 2.66e-17.)

Special directions: eigenvectors

From episode 4, a matrix is a machine that moves points. Roll chapati dough in one direction: a line drawn along the roll gets longer but keeps its direction; a slanted line gets longer and tilts. A matrix's special directions that are only stretched, never turned, are its eigenvectors ("eigen" is German for "own"); the stretch factor is the eigenvalue λ (lambda).

import numpy as np

A = np.array([[3, 1],
              [1, 3]])

for v in [[1, 0], [0, 1], [1, 1], [1, -1]]:
    print(v, "->", A @ v)

Output:

[1, 0] -> [3 1]
[0, 1] -> [1 3]
[1, 1] -> [4 4]
[1, -1] -> [ 2 -2]

(1, 0) and (0, 1) are tilted. (1, 1) → (4, 4) = 4 × (1, 1): an eigenvector with λ = 4. (1, −1) → (2, −2) = 2 × (1, −1): an eigenvector with λ = 2.

Definition: A v = λ v, with v ≠ 0. On its eigenvectors, the matrix acts like the plain number λ.

λ Effect on the eigenvector
λ > 1 stretched
0 < λ < 1 shrunk
λ = 0 squashed to zero (the matrix is singular, episode 4)
λ < 0 flipped to point the opposite way (and scaled by |λ|)

Any multiple of an eigenvector (like (2, 2)) is the same direction, so libraries return the one of length 1.

Symmetric matrices and np.linalg.eigh

A matrix that equals its own transpose (A = Aᵀ, a mirror image across the main diagonal) is symmetric. The most important ML matrices are symmetric (the covariance matrix in episode 11, the matrix behind PCA in episode 48). Symmetric matrices guarantee real eigenvalues and orthogonal eigenvectors. np.linalg.eigh ("H" for Hermitian, which for real numbers means symmetric) is the routine for them.

import numpy as np

A = np.array([[3, 1],
              [1, 3]])
print((A == A.T).all())             # symmetric: A equals its transpose

vals, vecs = np.linalg.eigh(A)      # eigenvalues and eigenvectors
print(vals)
print(vecs.round(3))

v = vecs[:, 1]                      # column 1 = eigenvector for 4
print(A @ v, vals[1] * v)

Output:

True
[2. 4.]
[[-0.707  0.707]
 [ 0.707  0.707]]
[2.82842712 2.82842712] [2.82842712 2.82842712]
  • Eigenvalues come out smallest first: 2, 4.
  • Eigenvectors are the columns of vecs: (−0.707, 0.707) is (1, −1)/√2 flipped (same line), (0.707, 0.707) is (1, 1)/√2.
  • The check A @ v == 4 * v holds.

For a general (non-symmetric) square matrix use np.linalg.eig; its eigenvalues can be complex numbers (for example, a rotation turns every direction, so it has no real eigenvector).

Eigendecomposition: turn, stretch, turn back

Any symmetric matrix splits into three simple moves:

  1. Qᵀ: turn so the eigenvectors line up with the axes;
  2. Λ (capital lambda): stretch each axis by its eigenvalue, a diagonal matrix (numbers only on the main diagonal);
  3. Q: turn back.

A = Q Λ Qᵀ is the eigendecomposition (Q holds the eigenvectors as columns).

import numpy as np

A = np.array([[3, 1],
              [1, 3]])
vals, Q = np.linalg.eigh(A)
print(round(Q[:, 0] @ Q[:, 1], 3))  # eigenvectors are orthogonal

L = np.diag(vals)                   # eigenvalues on a diagonal
print(L)
print(Q @ L @ Q.T)                  # eigendecomposition rebuilds A

Output:

0.0
[[2. 0.]
 [0. 4.]]
[[3. 1.]
 [1. 3.]]

Bonus facts worth knowing: the determinant equals the product of the eigenvalues (2 × 4 = 8 = 3·3 − 1·1), and a matrix is singular exactly when some eigenvalue is 0.

Real data isn't square

Raju's notebook: 6 customers × 4 items (chai, samosa, cake, pastry); some are chai people, some cake people.

import numpy as np

# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
                   [6, 5, 1, 0],
                   [4, 4, 0, 0],
                   [0, 1, 5, 4],
                   [1, 0, 6, 5],
                   [0, 0, 4, 4]])

print(orders.shape)
vals, vecs = np.linalg.eigh(orders)

Output:

(6, 4)
Traceback (most recent call last):
  ...
numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square

Eigenvectors exist only for square matrices: a 6 × 4 matrix takes 4 numbers in and gives 6 out, so "the same direction" makes no sense. Almost no dataset is square.

Singular value decomposition (SVD)

Any matrix, any shape, can be written X = U Σ Vᵀ:

Part Shape here (compact) Meaning
Vᵀ 4 × 4 orthonormal directions in item space (rows = item patterns)
Σ (sigma) 4 values a diagonal of singular values: stretch factors, never negative, sorted biggest first
U 6 × 4 orthonormal directions in customer space (how much each customer has of each pattern)

It is "turn, stretch, turn" for rectangles. A big singular value = an important pattern; a tiny one = mostly noise.

import numpy as np

# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
                   [6, 5, 1, 0],
                   [4, 4, 0, 0],
                   [0, 1, 5, 4],
                   [1, 0, 6, 5],
                   [0, 0, 4, 4]])

U, s, Vt = np.linalg.svd(orders, full_matrices=False)
print(U.shape, s.shape, Vt.shape)
print(s.round(2))                   # singular values, biggest first
print(Vt[:2].round(2))              # top 2 patterns
print(np.allclose(U @ np.diag(s) @ Vt, orders))

Output:

(6, 4) (4,) (4, 4)
[12.45 10.7   1.2   1.03]
[[-0.54 -0.46 -0.54 -0.46]
 [-0.53 -0.46  0.53  0.46]]
True
  • Two big singular values (12.45, 10.7) and two tiny ones (1.2, 1.03): the table really has two patterns.
  • Pattern 1 has the same sign for every item: overall appetite. Pattern 2 is − for chai/samosa and + for cake/pastry: chai side vs cake side. Nobody told SVD about tastes; it found them.
  • Signs can come out flipped (a pattern and its negative are the same direction); full_matrices=False gives the compact shapes.
  • np.allclose confirms the three parts rebuild the table.

Connection: the singular values of X are the square roots of the eigenvalues of the symmetric matrix XᵀX, and V holds its eigenvectors. That's why SVD and eigenvectors appear together in PCA.

Low-rank approximation

Keeping only the top k patterns (first k columns of U, k singular values, first k rows of Vᵀ) gives a rank-k approximation: the closest possible rank-k matrix to the original.

import numpy as np

# rows = 6 customers, columns = chai, samosa, cake, pastry
orders = np.array([[5, 4, 0, 1],
                   [6, 5, 1, 0],
                   [4, 4, 0, 0],
                   [0, 1, 5, 4],
                   [1, 0, 6, 5],
                   [0, 0, 4, 4]])

# obvious way to shrink: keep only 2 columns
keep = orders.copy(); keep[:, 2:] = 0
print("drop columns error:", np.linalg.norm(orders - keep).round(2))

U, s, Vt = np.linalg.svd(orders, full_matrices=False)
k = 2
approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
print("rank-2 SVD error:  ", np.linalg.norm(orders - approx).round(2))
print(approx.round(1))

Output:

drop columns error: 11.66
rank-2 SVD error:   1.58
[[ 4.8  4.2  0.5  0.4]
 [ 5.9  5.1  0.6  0.5]
 [ 4.3  3.7  0.  -0. ]
 [ 0.5  0.4  4.8  4.2]
 [ 0.6  0.5  5.9  5.1]
 [ 0.  -0.   4.3  3.7]]

Dropping columns wipes out the cake lovers (error 11.66); rank-2 SVD keeps both tastes (error 1.58, about 7× smaller) with the same budget of two patterns. The error is the norm of the difference (episode 3; for matrices this is called the Frobenius norm). The -0. entries are tiny negatives rounded.

Storage arithmetic: a 1000 × 1000 table (say a greyscale photo) is 1,000,000 numbers. Rank 50 stores 50 × 1000 (U) + 50 (Σ) + 50 × 1000 (Vᵀ) = 100,050 numbers: about a tenth. This is the idea behind SVD-based image compression and filling gaps in ratings tables for recommendations (episode 54).

Common mistakes (and the real error messages)

Mistake Error / result Why Fix
distance along a road = x coordinate Priya: 0 km the road isn't the x axis project: v @ (road / np.linalg.norm(road))
projecting onto a non-unit vector priya @ road = 20, not 4 scaled by ‖road‖ = 5 divide by the norm first
np.linalg.eigh(orders) on a 6 × 4 matrix numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square eigenvectors need a square matrix np.linalg.svd(orders)
reading eigenvectors as rows wrong vectors NumPy returns them as columns vecs[:, i]
expecting eigh values biggest first [2. 4.] eigh sorts ascending; svd sorts descending vals[::-1] if needed
eigh on a non-symmetric matrix silently wrong answer eigh only reads one triangle check (A == A.T).all(); else np.linalg.eig
shrinking data by deleting columns error 11.66 throws away whole patterns rank-k SVD: error 1.58

Mini project: how many patterns do you need?

import numpy as np

orders = np.array([[5, 4, 0, 1],
                   [6, 5, 1, 0],
                   [4, 4, 0, 0],
                   [0, 1, 5, 4],
                   [1, 0, 6, 5],
                   [0, 0, 4, 4]])
U, s, Vt = np.linalg.svd(orders, full_matrices=False)

for k in [1, 2, 3, 4]:
    approx = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
    err = np.linalg.norm(orders - approx)
    kept = (s[:k] ** 2).sum() / (s ** 2).sum()
    print(k, round(err, 2), f"{kept:.1%}")

Output:

1 10.82 57.0%
2 1.58 99.1%
3 1.03 99.6%
4 0.0 100.0%

kept is the share of the table's total "energy" (sum of squared singular values) the top k patterns hold. Going from 1 to 2 patterns jumps from 57% to 99.1%; after that, extra patterns add almost nothing. Also note: the error equals √(sum of the dropped s²), e.g. √(1.2² + 1.03²) ≈ 1.58. Choosing k by "how much is kept" is exactly how the number of components is chosen in PCA.

Practice

  1. Project (6, 8) onto (1, 0) and onto (0, 1).
Solution

v @ np.array([1, 0]) → 6, v @ np.array([0, 1]) → 8. Projecting onto the standard axes just reads off the coordinates; (1, 0) and (0, 1) are an orthonormal basis.

  1. Eigenvalues of [[4, 0], [0, 1]] without NumPy.
Solution

A diagonal matrix stretches the x axis by 4 and the y axis by 1, so the eigenvectors are (1, 0) and (0, 1) with eigenvalues 4 and 1. np.linalg.eigh prints [1. 4.] (ascending).

  1. Add a 7th customer [3, 3, 3, 3] (likes everything) and re-run SVD.
Solution

np.linalg.svd(orders7, compute_uv=False).round(2) → [13.81 10.7 1.2 1.11]. Only the first value (appetite) grew: an all-round eater adds to "how much", not to "which side".

  1. Is (−4, 3) orthogonal to (3, 4)? Is (2, 1) orthogonal to (1, 2)?
Solution

(−4)(3) + (3)(4) = 0: yes. 2·1 + 1·2 = 4 ≠ 0: no.

Quick quiz

import numpy as np

A = np.array([[2, 0],
              [0, 5]])
v = np.array([0, 1])
print(A @ v)

A) [0 5]: v is an eigenvector, λ = 5 B) [2 5]: not an eigenvector C) [0 2]: λ = 2 D) [5 0]: v was turned

Answer

A. A @ (0, 1) = (0, 5) = 5 × (0, 1): same direction, stretched 5×. For a diagonal matrix the axes are eigenvectors and the diagonal entries are the eigenvalues.

Interview questions

  1. What is an eigenvector? A non-zero vector v with A v = λ v: the matrix only scales it; λ is the eigenvalue.
  2. Why are symmetric matrices special? Real eigenvalues, orthogonal eigenvectors, and the decomposition A = Q Λ Qᵀ always exists.
  3. What is the difference between eigendecomposition and SVD? Eigendecomposition needs a square (diagonalisable) matrix; SVD works for any m × n matrix and always gives orthonormal U, V and non-negative singular values.
  4. How are SVD and PCA related? PCA's directions are the right singular vectors (V) of the centred data matrix, equivalently the eigenvectors of its covariance matrix; variances are s²/(n − 1).
  5. What does a low-rank approximation give you? The best rank-k approximation in the least-squares (Frobenius) sense (Eckart–Young theorem): compression and noise removal.
  6. What is a projection? The closest point on a line (or subspace); onto a unit vector u it is (v · u) u.
  7. What does orthogonality mean in data? Directions with zero dot product: independent axes that don't overlap, so coordinates along them can be read off separately.

Glossary

Term Meaning Example
Origin The zero point (0, 0) of a map or space Raju's stall
Direction Which way a vector points the road (3, 4)
Projection The shadow of a vector on a direction: the closest point on it (2.4, 3.2)
Scalar projection How far along the direction: v · u 4.0 km
Orthogonal At right angles; dot product 0 along · across = 0
Basis Directions every point is built from east, north
Orthonormal basis Orthogonal basis directions of length 1 along, across
Coordinates A point's amounts along the basis directions Priya = (4, 3) in road coords
Eigenvector A non-zero vector a matrix only stretches (A v = λ v) (1, 1) for [[3, 1], [1, 3]]
Eigenvalue (λ) The stretch factor of an eigenvector 4 and 2
Symmetric matrix Equals its transpose (A = Aᵀ) [[3, 1], [1, 3]]
Hermitian The complex-number version of symmetric; for real matrices, the same thing the H in eigh
np.linalg.eigh Eigenvalues (ascending) and eigenvectors (columns) of a symmetric matrix vals, vecs = eigh(A)
Diagonal matrix Non-zero numbers only on the main diagonal np.diag([2, 4])
Eigendecomposition A = Q Λ Qᵀ: turn, stretch, turn back rebuilds [[3, 1], [1, 3]]
Square matrix Same number of rows and columns 2 × 2
SVD Any matrix = U Σ Vᵀ np.linalg.svd(orders)
Singular values The non-negative stretch factors in Σ, biggest first; the importance of each pattern 12.45, 10.7, 1.2, 1.03
Singular vectors The columns of U (left) and rows of Vᵀ (right): the patterns appetite; chai vs cake
Rank-k / low-rank approximation Keep only the top k patterns of the SVD rank 2, error 1.58
Frobenius norm The L2 norm of all a matrix's entries, used as the error np.linalg.norm(orders - approx)
Energy kept Share of Σ s² held by the top k singular values 99.1% at k = 2
np.allclose Equality test that tolerates tiny rounding rebuild check → True

Summary

  • A projection is a shadow: v · u measures how far v goes along the unit direction u.
  • Orthogonal = dot product 0; an orthonormal basis is a turned grid of unit directions, and coordinates in it are dot products.
  • Eigenvectors are directions a matrix only stretches; eigenvalues are the stretch factors: A v = λ v.
  • Symmetric matrices have real eigenvalues and orthogonal eigenvectors: A = Q Λ Qᵀ (turn, stretch, turn back).
  • SVD, X = U Σ Vᵀ, works for any shape; singular values rank the patterns hidden in the data.
  • A low-rank approximation keeps the top k patterns: far better than deleting columns, and the basis of compression, recommendations and PCA.

Next episode: Maths for ML 4, derivatives: functions, slope, rate of change, the derivative rules, the chain rule and finding the best value.