SyntaxPilot
Courses/Machine Learning/Lesson 48
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 48 of 90

PCA from scratch

dimensionality reductionfeature extraction vs feature selectioncovariance matrixprincipal componentsexplained varianceexplained variance ratioscree plotPCA via eigendecomposition and SVDwhiteningreconstruction error

Machine Learning Full Course · Episode 48. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Riya's 1,000 customers are made up for teaching. Because we made the data ourselves, we secretly know the two hidden habits that drive it (family size and a love of snacks), which lets us check whether PCA finds them.

The big idea: a table with many columns is hard to draw and slow to learn from. Dimensionality reduction gives you fewer columns. The obvious way, keeping a few of the original columns (feature selection), wastes information when the columns move together: on Riya's 10 spending columns the 2 widest keep only 23% of the variance. PCA (principal component analysis) instead builds new columns (feature extraction): it turns the axes so that the first new axis runs along the longest direction of the data cloud, the second along the next longest at a right angle, and so on. Recipe: centre the data, compute the covariance matrix, take its eigenvectors (the principal components) and eigenvalues (the variance along each), and project. Two components keep 82.5% of the variance, and they turn out to be the two hidden habits: PC1 = staples (veg, milk, rice), PC2 = snacks (chips, soda, chocolate). The SVD of the centred data gives the same numbers (variance = s² / (n − 1)), and so does scikit-learn's PCA, up to flipped signs. To choose how many components to keep, read the explained variance ratio, the scree plot and the reconstruction error: 90% needs 6 components, but after 2 the extra ones mostly rebuild noise (rebuild error Rs 132.7 with 2 components, below the Rs 150 of noise we added). Whitening makes every new column have standard deviation 1. Three mistakes: forgetting to centre (a fake 95%), mixing units without scaling (metres take over PC1), and asking for more components than columns (a ValueError).

Sources: K. Pearson, "LIII. On lines and planes of closest fit to systems of points in space", The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901 (record at bibbase.org); the docstrings of the installed scikit-learn 1.9.1: PCA (it centres the input data but does not scale it; a float n_components between 0 and 1 means "keep enough components for this share of the variance"; attributes explained_variance_, explained_variance_ratio_, components_; option whiten; method inverse_transform).

Words you need first

  • Row, column, feature: a table has one row per customer and one column per measured thing. In machine learning a column used as input is called a feature. Riya's table has 1,000 rows and 10 features.
  • Mean (Episode 10): the average. Example: the mean of 2, 4, 6 is 4.
  • Variance and standard deviation (Episode 10): how spread out a column is. Variance = the average squared distance from the mean (divided by n − 1 for a sample); standard deviation = its square root, in the column's own units. Example: if every customer spends exactly Rs 1,000 on milk, the variance is 0.
  • Covariance and correlation (Episode 11): whether two columns rise together. Positive covariance between milk and rice: customers who spend more on milk also spend more on rice. Correlation is covariance rescaled to lie between −1 and +1; 0 means "no straight-line relation".
  • Vector, matrix, matrix multiplication @ (Episode 5): a vector is a list of numbers (one arrow); a matrix is a table of numbers. A @ B multiplies matrices: a (1000 × 10) table times a (10 × 2) matrix gives a (1000 × 2) table.
  • Transpose (ᵀ, .T): flip a matrix so rows become columns. A (1000 × 10) matrix becomes (10 × 1000).
  • Dot product: multiply two vectors element by element and add: (1, 2) · (3, 4) = 3 + 8 = 11. A dot product of 0 means the arrows are at a right angle.
  • Unit vector: an arrow of length 1, e.g. (0.707, 0.707), because 0.707² + 0.707² ≈ 1.
  • Feature selection (Episode 40): keeping some original columns and dropping the rest.
  • Standardising, StandardScaler (Episode 16): rescale every column to mean 0 and standard deviation 1.
  • Train/test split and data leakage (Episode 15): learn on training rows, judge on test rows the model never saw; leakage is when test rows sneak into training.
  • Unsupervised learning (Episodes 1 and 43): learning from rows without an answer column (no label). PCA is unsupervised: it never looks at a label.
  • Random generator: np.random.default_rng(8) makes the same "random" numbers every run, so you get exactly the outputs printed here.

The real-life hook: ten columns per customer

Riya runs a grocery shop. For each of her 1,000 customers she has 10 columns: how many rupees they spend per month on milk, bread, eggs, vegetables, fruit, rice, dal, chips, soda and chocolate. She wants a map of her customers, but a screen has two axes, not ten. And for the models in later episodes, fewer columns means faster training and often less noise.

We made the data, driven by two hidden habits that nobody recorded: family size (big families buy more staples) and a love of snacks. These six lines create it; every program below starts with them (we call them HDR):

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
  • family and snacks: one number per customer for each habit, drawn from a bell curve with mean 0 and standard deviation 1.
  • load: how strongly each habit drives each column. Row 1 (family): milk 1, bread 0.9, ..., rice 1, but chips only 0.1 and soda 0. Row 2 (snacks): chips 1, soda 0.9, chocolate 0.8, almost nothing on staples.
  • np.c_[family, snacks] @ load: a (1000 × 2) table times a (2 × 10) matrix = (1000 × 10): every column is a mix of the two habits. Then * 300 + 1000 turns it into rupees around Rs 1,000, and r.normal(0, 150, (n, 10)) adds random noise of Rs 150 to every value. .round() makes whole rupees.

Dimension and dimensionality reduction

  • A dimension is one column, one axis of the data. Riya's data is 10-dimensional: each customer is a point in a space with 10 axes. Two columns can be drawn on paper (x and y); ten cannot.
  • Dimensionality reduction means going from many columns to fewer while losing as little as possible. Example: from 10 spending columns to 2 numbers per customer, which you can draw as a map.

Two ways to get fewer columns:

Way What it does Example
Feature selection (Episode 40) Keep some original columns, drop the rest Keep chips and veg, drop the other 8
Feature extraction Build new columns, each a mix of the old ones Summarise a cricketer by batting average and strike rate instead of listing every innings

Variance as information

How do we measure what we keep? With variance. A column where every customer has the same value tells us nothing about who is who; spread is information. The total variance is the sum of all columns' variances. Goal: keep as much of the total variance as possible, in as few columns as possible.

The obvious attempt: keep the two widest columns

Look at each column's share of the total variance and keep the two with the largest:

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
print(X.iloc[:3, :6].astype(int).to_string(), "  ..."); print("shape:", X.shape, "  monthly Rs per category")
v = X.var(); print("var share:", *(v / v.sum()).round(2))
keep = v.nlargest(2).index.tolist()                                      # obvious: keep the 2 widest columns
print("keep", keep, "-> share of total variance kept:", round(v[keep].sum() / v.sum(), 2))

Output:

   milk  bread  eggs  veg  fruit  rice
0   517    678   889  678    481   551
1   561    756   744  629    659   773
2   411    647   693  725    830   613   ...
shape: (1000, 10)   monthly Rs per category
var share: 0.11 0.1 0.08 0.12 0.07 0.11 0.1 0.12 0.1 0.08
keep ['chips', 'veg'] -> share of total variance kept: 0.23
  • X.var() is each column's variance; v / v.sum() its share of the total.
  • Every column holds about a tenth of the variance (between 7% and 12%). No column stands out.
  • v.nlargest(2) picks chips and veg: they keep only 23% of the variance. 77% is thrown away.

Why did it fail? The columns move together. Big families spend more on milk and rice and dal; snack lovers spend more on chips and soda and chocolate. The same information is spread across many columns, so any two columns catch only a small part of it.

New axes along the spread

Look at just two of the ten columns, milk and rice. Plotted against each other, the 1,000 customers form a long tilted oval: when milk spending is high, rice spending is high too.

Instead of keeping milk or rice, turn the axes:

  • Put the first new axis along the length of the oval. A customer's position along it says "how much of a staples buyer" they are.
  • Put the second new axis at a right angle to it, across the oval's width.

These new axes are the principal components, PC1 and PC2. On milk and rice alone, PC1 keeps 90% of the two columns' variance (computed in the worked example below). The idea goes back to Karl Pearson in 1901: finding the lines and planes that fit a cloud of points most closely. A line that runs along the oval is the line closest to all the points, and it is also the direction of the largest spread.

Orthogonal

Orthogonal means at right angles. Principal components are always orthogonal to each other: their dot product is 0. That is why the new columns do not overlap: each one carries information the others do not.

Projection

To project a point onto an axis is to drop it straight down onto that axis (at a right angle) and read off where it lands. Like the shadow of a pole on the ground when the sun is right overhead. For a unit vector w, the position of point x along w is the dot product x · w. Projecting all customers onto PC1 and PC2 turns each customer from 10 numbers into 2.

Finding the axes: the covariance matrix

Step 1: centring

Centring means subtracting each column's mean, so every column has mean 0. Example: milk spending 517, 561, 411 with a column mean of 991.4 becomes −474.4, −430.4, −580.4. The cloud moves so its centre sits at the origin (0, 0, ...). PCA is about the spread around the centre, so the centre must be at zero; the "Common mistakes" section shows what happens if you forget.

Step 2: the covariance matrix

The covariance matrix C is a square table of how every pair of columns moves together. For 10 columns it is 10 × 10:

  • The diagonal (top-left to bottom-right) holds each column's variance.
  • Every other cell (i, j) holds the covariance of column i with column j. It is symmetric: cell (i, j) = cell (j, i).
  • With centred data A (n rows), C = Aᵀ A / (n − 1).

Example: for milk and rice, C is 2 × 2 (from the program below): milk variance 115,607, rice variance 112,130, and covariance 92,205 in both off-diagonal cells. A large positive covariance: the tilted oval.

Step 3: eigenvectors and eigenvalues (recap of Episode 5)

For a square matrix C, an eigenvector v is a direction that C only stretches, without turning it: C v = λ v. The stretch factor λ (lambda) is the eigenvalue. Example: for C = [[5, 4], [4, 5]], v = (1, 1) gives C v = (5 + 4, 4 + 5) = (9, 9) = 9 × (1, 1). So (1, 1) is an eigenvector with eigenvalue 9.

The key fact of PCA:

  • The eigenvectors of the covariance matrix are the principal components.
  • Each eigenvalue is the variance of the data along its component.
  • A covariance matrix is symmetric, so its eigenvectors are orthogonal and its eigenvalues are never negative.
  • Sort them biggest eigenvalue first: PC1 has the most variance, PC2 the next, and so on. The eigenvalues add up to the total variance.

Worked example by hand

Take the tiny covariance matrix C = [[5, 4], [4, 5]] (two columns, each with variance 5, covariance 4).

  1. Eigenvalues solve (5 − λ)² − 4² = 0, so 5 − λ = ±4: λ = 9 and λ = 1.
  2. Total variance = 5 + 5 = 10 = 9 + 1 (the eigenvalues add up to the diagonal).
  3. Share of variance: PC1 keeps 9 / 10 = 90%, PC2 keeps 1 / 10 = 10%.
  4. Eigenvectors: (1, 1) for 9 and (−1, 1) for 1. As unit vectors, divide by √2: (0.707, 0.707) and (−0.707, 0.707). Their dot product is −0.5 + 0.5 = 0: orthogonal.

PC1 = (0.707, 0.707) means "the two columns added up, equally weighted": the length of the oval. NumPy agrees, and the same check on the real milk and rice columns:

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
C = np.array([[5.0, 4.0], [4.0, 5.0]])                                 # a tiny covariance matrix, by hand
vals, vecs = np.linalg.eigh(C); vals, vecs = vals[::-1], vecs[:, ::-1]   # biggest first
print("eigenvalues:", vals, "  share:", (vals / vals.sum()).round(2))
print("PC1:", vecs[:, 0].round(3), "  PC2:", vecs[:, 1].round(3), "  dot product:", round(vecs[:, 0] @ vecs[:, 1], 3))
m = X[["milk", "rice"]]; C2 = np.cov(m.values.T)                         # the real milk and rice columns
print("milk/rice covariance:", C2.round(0).tolist())
v2 = np.sort(np.linalg.eigvalsh(C2))[::-1]; print("eigenvalues:", v2.round(0), "  PC1 share:", round(v2[0] / v2.sum(), 2))
print("column variance shares:", (X.var() / X.var().sum()).min().round(3), "to", (X.var() / X.var().sum()).max().round(3))

Output:

eigenvalues: [9. 1.]   share: [0.9 0.1]
PC1: [0.707 0.707]   PC2: [-0.707  0.707]   dot product: 0.0
milk/rice covariance: [[115607.0, 92205.0], [92205.0, 112130.0]]
eigenvalues: [206090.  21647.]   PC1 share: 0.9
column variance shares: 0.072 to 0.118
  • np.linalg.eigh is NumPy's eigen-solver for symmetric matrices (the "h" stands for Hermitian, the general name). It returns eigenvalues smallest first, so [::-1] reverses them; vecs[:, ::-1] reverses the eigenvector columns to match. eigvalsh returns only the eigenvalues.
  • The hand answer 9 and 1, 90% / 10%, is confirmed, and the two components have dot product 0.
  • Real milk and rice: eigenvalues 206,090 and 21,647, so PC1 keeps 90% of these two columns' variance: the long axis of the oval.
  • The last line confirms the earlier claim: each of the 10 columns holds between 7.2% and 11.8% of the total.

Step 4: project

New data = centred X @ top-k eigenvectors. With the 10 × 2 matrix of the top two eigenvectors, a (1000 × 10) table becomes (1000 × 2).

PCA from scratch

All four steps on the 10 columns:

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
A = X.values - X.values.mean(axis=0)                                    # 1) centre every column
C = A.T @ A / (n - 1)                                                    # 2) covariance matrix, 10 x 10
vals, vecs = np.linalg.eigh(C)                                           # 3) eigenvalues and eigenvectors (ep 5)
order = np.argsort(vals)[::-1]; vals, vecs = vals[order], vecs[:, order] # biggest first
print("covariance matrix:", C.shape, "  same as np.cov:", np.allclose(C, np.cov(X.values.T)))
print("variance ratio, PC1..PC5:", *(vals / vals.sum()).round(3)[:5], "...")
print("first 2 components keep:", round(vals[:2].sum() / vals.sum(), 3))
for k in (0, 1):                                                         # what does each component mix?
    top = sorted(zip(cols, vecs[:, k]), key=lambda t: -abs(t[1]))[:3]
    print(f"PC{k + 1} biggest weights:", ", ".join(f"{c} {w:+.2f}" for c, w in top))
Z = A @ vecs[:, :2]; print("new data:", Z.shape)                          # 4) project: 10 columns -> 2

Output:

covariance matrix: (10, 10)   same as np.cov: True
variance ratio, PC1..PC5: 0.562 0.263 0.025 0.024 0.023 ...
first 2 components keep: 0.825
PC1 biggest weights: veg -0.42, milk -0.41, rice -0.41
PC2 biggest weights: chips -0.63, soda -0.57, chocolate -0.49
new data: (1000, 2)
  • X.values.mean(axis=0): the mean of every column (axis=0 = down the rows). Subtracting it centres the data.
  • A.T @ A / (n - 1) is our covariance matrix; np.allclose confirms it equals NumPy's np.cov (which wants columns as rows, hence .T).
  • np.argsort(vals)[::-1]: the positions of the eigenvalues from biggest to smallest. Each eigenvector is a column of vecs.
  • Variance ratio: PC1 has 56.2%, PC2 26.3%, then every other component about 2%.
  • Two components keep 82.5%, compared with 23% for two original columns.
  • The weights (the numbers inside each eigenvector, also called loadings) say how much each old column contributes to the new one. PC1 is mostly veg, milk and rice: the family-size habit. PC2 is mostly chips, soda and chocolate: snacks. PCA found the two hidden habits without being told.
  • The minus signs don't matter: an axis pointing the other way is the same axis. If v is an eigenvector, so is −v.
  • Z = A @ vecs[:, :2]: the projection, 1,000 customers × 2 new columns. That is Riya's map.

PCA via the SVD

A second route, also from Episode 5. The singular value decomposition (SVD) splits any matrix into three pieces:

A = U · diag(s) · Vᵀ
  • A: the centred data, n × d (1000 × 10).
  • U: n × d; its columns are unit vectors, one per component (the customers' positions along each component, before scaling).
  • s: the singular values, d non-negative numbers, biggest first. diag(s) is the square matrix with s on the diagonal and zeros elsewhere.
  • Vᵀ ("V transposed"): d × d; its rows are the principal components.
  • Variance along component i = sᵢ² / (n − 1).

Why? Aᵀ A = V · diag(s²) · Vᵀ, so C = Aᵀ A / (n − 1) has eigenvectors V and eigenvalues s² / (n − 1). You never build the covariance matrix, which saves memory and avoids rounding trouble when there are many columns. scikit-learn's PCA offers SVD-based solvers, chosen by its svd_solver option (default "auto"). Careful: on this tall, narrow table (1,000 rows, 10 columns) we checked the solver "auto" actually picked, and it was "covariance_eigh", the eigen-decomposition of the covariance matrix, our first route. The video says the same: scikit-learn picks the route from the table's shape, and either way the numbers are identical.

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
A = X.values - X.values.mean(axis=0)
U, s, Vt = np.linalg.svd(A, full_matrices=False)                         # SVD of the centred data (ep 5)
print("variance from singular values:", (s ** 2 / (n - 1))[:3].round(0))
vals = np.sort(np.linalg.eigvalsh(np.cov(A.T)))[::-1]
print("eigenvalues of covariance:   ", vals[:3].round(0))
pca = PCA(n_components=2).fit(X)                                         # sklearn picks a solver itself
print("sklearn explained_variance_:  ", pca.explained_variance_.round(0))
print("ratio:", pca.explained_variance_ratio_.round(3), "  total", pca.explained_variance_ratio_.sum().round(3))
print("PC1 from SVD vs sklearn:", np.allclose(np.abs(Vt[0]), np.abs(pca.components_[0])), "(signs may flip)")

Output:

variance from singular values: [565468. 264621.  25601.]
eigenvalues of covariance:    [565468. 264621.  25601.]
sklearn explained_variance_:   [565468. 264621.]
ratio: [0.562 0.263]   total 0.825
PC1 from SVD vs sklearn: True (signs may flip)
  • np.linalg.svd(A, full_matrices=False): the "economy" SVD, U is 1000 × 10 instead of 1000 × 1000.
  • Three routes, identical numbers: 565,468 and 264,621 (and 25,601 for PC3). These are variances, so they are in rupees squared.
  • PCA(n_components=2).fit(X): we passed the raw X; scikit-learn centres it for us (it stores the column means in mean_). It does not scale it.
  • explained_variance_ratio_: 0.562 and 0.263, total 0.825.
  • components_: one row per component (like Vᵀ). It matches the SVD's first row after taking absolute values (np.abs).

Sign flips

An eigenvector or a row of Vᵀ is only defined up to its sign: v and −v describe the same axis. Different solvers (or versions) may return either. So PC1's weights can come out as all-negative in one program and all-positive in another (our scratch PCA printed veg −0.42, milk −0.41; the mini project below, which also scales the columns and so gets slightly different sizes, prints milk +0.39, veg +0.39). The new column then has its values mirrored (multiplied by −1). Nothing is wrong; just don't compare raw signs between tools.

How many components?

Vocabulary:

  • Explained variance: the variance along a component (its eigenvalue): 565,468 for PC1.
  • Explained variance ratio: its share of the total variance: 0.562 for PC1.
  • Cumulative ratio: the ratios added up from PC1: 0.562 + 0.263 = 0.825 for the first two.
  • Scree plot: the ratios drawn as bars, PC1 first. Scree is the pile of loose rocks at the bottom of a cliff: keep the components before the cliff and drop the rubble.
  • n_components as a float: PCA(n_components=0.90) keeps the fewest components that reach 90% of the variance. An integer (2) means "exactly this many".
  • Reconstruction error: squeeze to k columns, rebuild 10 columns with inverse_transform, and measure how far the rebuild is from the original. We measure with the RMSE (root mean squared error): the square root of the average squared difference, in rupees. Example: differences of 3 and −4 give √((9 + 16) / 2) ≈ 3.5.
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
full = PCA().fit(X)                                                      # all 10 components
print("cumulative ratio:", *np.cumsum(full.explained_variance_ratio_).round(2)[:6], "...")
print("components for 90%:", PCA(n_components=0.90).fit(X).n_components_)   # a float = keep this share
for k in (1, 2, 3, 5, 10):                                               # squeeze to k columns, then rebuild 10
    p = PCA(n_components=k).fit(X); back = p.inverse_transform(p.transform(X))
    print(f"k={k:2}: rebuild error (RMSE) Rs {np.sqrt(((X.values - back) ** 2).mean()):6.1f}")

Output:

cumulative ratio: 0.56 0.82 0.85 0.87 0.9 0.92 ...
components for 90%: 6
k= 1: rebuild error (RMSE) Rs  209.9
k= 2: rebuild error (RMSE) Rs  132.7
k= 3: rebuild error (RMSE) Rs  122.7
k= 5: rebuild error (RMSE) Rs  101.7
k=10: rebuild error (RMSE) Rs    0.0
  • PCA() with no n_components keeps all 10. np.cumsum adds the ratios up.
  • 90% needs 6 components, because after the first two each component adds only about 2%. (The cumulative "0.9" at 5 components is rounded; the run below shows it is 0.897, just short.)
  • n_components_ (with a trailing underscore) is the number scikit-learn actually chose.
  • transform projects to k columns; inverse_transform maps them back to 10 columns (and adds the means back).
  • Rebuild error: Rs 209.9 with one component, Rs 132.7 with two, then it shrinks slowly (122.7, 101.7), and with all 10 it is 0: nothing was thrown away.

The cumulative ratio with more decimals, and the size of the rubble:

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
cum = np.cumsum(PCA().fit(X).explained_variance_ratio_)
print("cumulative ratio, 4 decimals:", cum.round(4))
print("ratio of PC3..PC10:", PCA().fit(X).explained_variance_ratio_[2:].round(3))

Output:

cumulative ratio, 4 decimals: [0.5619 0.8248 0.8502 0.874  0.897  0.9194 0.9404 0.9611 0.9812 1.    ]
ratio of PC3..PC10: [0.025 0.024 0.023 0.022 0.021 0.021 0.02  0.019]

Reading the scree plot

Draw the ratios as bars: 0.562, 0.263, then 0.025, 0.024, 0.023, 0.022, 0.021, 0.021, 0.020, 0.019. There is a clear cliff after two components; everything after it is rubble of about 2% each, nearly flat.

Why flat? We added random noise of Rs 150 to every column. Noise has variance too, and it is spread evenly over all directions, so each "noise component" gets roughly the same small share. With two components the rebuild error (Rs 132.7) is already below the noise level (Rs 150): the extra components mostly rebuild noise. So here two is the honest answer, even though a 90% target would ask for six. A fixed target like 90% is a rule of thumb, not a law.

Whitening

The PCA columns are uncorrelated (correlation 0), but they still have different spreads: PC1 is much wider than PC3. Whitening divides each new column by its standard deviation, so every new column has standard deviation 1 and they stay uncorrelated. The name comes from white noise, which has equal strength everywhere.

  • Helps: models that measure distances, like k-means (Episode 43) and k-nearest neighbours, where a wide column would dominate.
  • Costs: you can no longer see from the numbers that PC1 mattered most.
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
for w in (False, True):
    Z = PCA(n_components=3, whiten=w).fit_transform(X)
    print(f"whiten={w!s:5}  std of each new column:", Z.std(axis=0, ddof=1).round(2))
    print("   correlation of PC1 and PC2:", f"{abs(np.corrcoef(Z.T)[0, 1]):.3f}")

Output:

whiten=False  std of each new column: [751.98 514.41 160.  ]
   correlation of PC1 and PC2: 0.000
whiten=True   std of each new column: [1. 1. 1.]
   correlation of PC1 and PC2: 0.000
  • fit_transform(X) = fit(X) then transform(X) in one call.
  • Z.std(axis=0, ddof=1): the standard deviation of each column (ddof=1 divides by n − 1, like the variance above). np.corrcoef(Z.T)[0, 1]: the correlation of column 0 with column 1.
  • Without whitening: standard deviations 751.98, 514.41, 160.00 (rupees). With whitening: 1, 1, 1. The correlation of PC1 and PC2 is 0.000 both times.

Three mistakes in code

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA; from sklearn.preprocessing import StandardScaler; import textwrap
_, s, Vt = np.linalg.svd(X.values, full_matrices=False)                  # MISTAKE: scratch PCA, forgot to centre
print("no centring: 'PC1' share", (s[0] ** 2 / (s ** 2).sum()).round(2), " weights", *Vt[0, :3].round(2), "...")
X["distance_m"] = r.normal(3000, 1500, n).round()                        # distance from the store, in METRES
for name, data in [("raw", X), ("scaled", StandardScaler().fit_transform(X))]:
    p = PCA(n_components=2).fit(data); w = pd.Series(p.components_[0], X.columns).abs()
    print(f"{name:6} PC1 explains {p.explained_variance_ratio_[0]:.2f}, biggest weight: {w.idxmax()} {w.max():.2f}")
try: PCA(n_components=12).fit(X)                                         # MISTAKE: more components than columns
except ValueError as e: print(*textwrap.wrap("ValueError: " + str(e), 62), sep="\n")

Output:

no centring: 'PC1' share 0.95  weights -0.32 -0.32 -0.32 ...
raw    PC1 explains 0.69, biggest weight: distance_m 1.00
scaled PC1 explains 0.51, biggest weight: veg 0.39
ValueError: n_components=12 must be between 0 and
min(n_samples, n_features)=11 with
svd_solver='covariance_eigh'

1. Forgetting to centre. A scratch SVD on the raw X (no mean subtracted) says the first direction holds 95% of the "variance", with equal weights of −0.32 on every column (1 / √10 ≈ 0.32). That direction just points from zero to the average customer (everyone spends about Rs 1,000 on everything). It measures the mean, not the spread. Fix: subtract the column means first. scikit-learn's PCA does this for you; scratch code does not.

2. Mixed units without scaling. We add an 11th column, distance_m: distance from the store in metres (around 3,000, spread 1,500). On the raw data PC1 explains 69% and is entirely distance (weight 1.00), only because metres make big numbers: a spread of 1,500 metres has a variance far larger than any spending column. After StandardScaler (every column to standard deviation 1), PC1 explains 51% and its biggest weight is veg (0.39): the family habit again. When columns use different units, scale first. (When all columns share one unit, like Riya's rupees, raw PCA is fine and keeps the rupee meaning.)

3. More components than columns. PCA(n_components=12) on 11 columns:

ValueError: n_components=12 must be between 0 and min(n_samples, n_features)=11 with svd_solver='covariance_eigh'

You cannot make more new axes than there are columns (or rows): at most min(rows, columns) = min(1000, 11) = 11. svd_solver is the method scikit-learn picked to compute the components; here it chose 'covariance_eigh', the eigen-decomposition of the covariance matrix, our first route. (textwrap.wrap(text, 62) only splits the long message into lines of at most 62 characters so it fits the screen.)

Fit PCA on training rows only

PCA learns from data: the means, and the axes. So the rule from Episode 15 holds: fit it on the training rows only, ideally inside a pipeline (a chain of steps fitted together, make_pipeline), then transform the test rows with what was learned. If you fit on all rows, the test rows leak into the axes and your test score is too optimistic. Practice 3 shows the pattern.

When PCA helps, and when it hurts

Helps Hurts
Many columns that move together (spending categories, readings from many sensors) New columns are mixes: "PC2 went up" is harder to explain to a shop owner than "chips spending went up"
A 2-D map to look at the data PCA never looks at the label: the direction with the most variance isn't always the one that predicts best
Faster models, less noise, compression Only straight-line structure: curved shapes need the methods of the next episode

Common mistakes

Mistake Exact message / symptom Why Fix
Keep the widest original columns 2 columns keep 23% The information is shared across columns PCA: 2 components keep 82.5%
Scratch PCA without centring no centring: 'PC1' share 0.95, equal weights −0.32 It finds the mean, not the spread Subtract the column means
Mixed units, no scaling raw PC1 explains 0.69, biggest weight: distance_m 1.00 Big numbers win StandardScaler first
Chasing 90% blindly components for 90%: 6, mostly noise Noise has variance too Scree plot and rebuild error: 2 here
More components than columns ValueError: n_components=12 must be between 0 and min(n_samples, n_features)=11 with svd_solver='covariance_eigh' Only 11 columns k ≤ min(rows, columns)
Comparing raw signs between tools PC1 weights all negative in one run (veg −0.42), all positive in another (veg +0.39) Sign flips: v and −v are the same axis Compare absolute values
Fitting PCA on all rows before splitting Test rows shape the axes Leakage (Episode 15) Pipeline, fit on train only

Mini project: Riya's customer map

Split first, choose k from the scree cliff on the training rows, name each component by its biggest weights, map the test customers, and rebuild them in rupees to find the customer the map explains worst. make_pipeline(StandardScaler(), PCA(...)) scales then reduces; the pipeline's inverse_transform undoes both steps.

import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA; from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler; from sklearn.model_selection import train_test_split
X_tr, X_te = train_test_split(X, test_size=0.2, random_state=1)          # 1) split first
scout = make_pipeline(StandardScaler(), PCA()).fit(X_tr)                 # 2) scree on train rows
ratio = scout[-1].explained_variance_ratio_; print("scree:", ratio.round(3))
k = int(np.argmax(ratio[:-1] / ratio[1:])) + 1                            # 3) the biggest drop = the cliff
print("cliff after component", k)
pipe = make_pipeline(StandardScaler(), PCA(n_components=k)).fit(X_tr)
for i in range(k):                                                       # 4) name each component
    w = pd.Series(pipe[-1].components_[i], X.columns); top = w.abs().nlargest(3).index
    print(f"PC{i + 1}:", ", ".join(f"{c} {w[c]:+.2f}" for c in top))
Z = pd.DataFrame(pipe.transform(X_te), columns=["PC1", "PC2"], index=X_te.index)   # 5) the map for test rows
print("map:", Z.shape, "  first row:", Z.iloc[0].round(2).tolist())
back = pipe.inverse_transform(Z.values)                                  # 6) rebuild test rows in rupees
err = np.sqrt(((X_te.values - back) ** 2).mean(axis=1))
print(f"test RMSE Rs {np.sqrt(((X_te.values - back) ** 2).mean()):.1f}   worst customer {X_te.index[err.argmax()]}: Rs {err.max():.1f}")

Output:

scree: [0.553 0.26  0.032 0.027 0.026 0.024 0.022 0.019 0.019 0.018]
cliff after component 2
PC1: milk +0.39, veg +0.39, rice +0.38
PC2: chips +0.57, soda +0.57, chocolate +0.54
map: (200, 2)   first row: [2.64, -2.84]
test RMSE Rs 130.4   worst customer 437: Rs 232.8
  • scout[-1] is the last step of the pipeline (the PCA). On the 800 scaled training rows the scree is 0.553, 0.260, then 0.032 and below.
  • ratio[:-1] / ratio[1:] divides each ratio by the next one; the biggest drop is after component 2: the cliff, found automatically.
  • PC1 = milk, veg, rice (staples); PC2 = chips, soda, chocolate (snacks). The signs are positive this time: a sign flip, same axes.
  • The map has 200 test customers × 2 columns, ready to plot. The first test customer sits at (2.64, −2.84) in standardised units.
  • Rebuilding the test customers from 2 numbers each misses by Rs 130.4 on average, close to the training result. Customer 437 is explained worst (Rs 232.8): worth a look by hand.

Practice

1. Scratch PCA: rebuild X from 2 components by hand (no scikit-learn). Do you get the same error as inverse_transform?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
A = X.values - X.values.mean(axis=0)                                    # scratch PCA, then rebuild
vals, vecs = np.linalg.eigh(np.cov(A.T)); W = vecs[:, ::-1][:, :2]        # top-2 eigenvectors, 10 x 2
Z = A @ W                                                                # squeeze: 1000 x 2
back = Z @ W.T + X.values.mean(axis=0)                                   # rebuild: 1000 x 10, add the means back
print("Z:", Z.shape, "  back:", back.shape)
print("customer 0 real:   ", X.values[0, :5].astype(int))
print("customer 0 rebuilt:", back[0, :5].round(0).astype(int))
print("RMSE Rs", round(np.sqrt(((X.values - back) ** 2).mean()), 1))

Output:

Z: (1000, 2)   back: (1000, 10)
customer 0 real:    [517 678 889 678 481]
customer 0 rebuilt: [618 675 684 608 682]
RMSE Rs 132.7

Rebuilding is projecting back: Z @ W.T (2 columns × the transposed 2 × 10 weights = 10 columns), then add the means. The RMSE is Rs 132.7, exactly inverse_transform's number. Customer 0's rebuild is close for bread (678 vs 675) but off for eggs (889 vs 684): the part of eggs spending that is noise, not habit, is lost.

2. Add a third hidden habit, "fruit lovers" (more fruit and some veg). What changes in the explained variance?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
fruitlove = r.normal(0, 1, n)                                            # a third hidden habit (made up)
X3 = X.copy(); X3["fruit"] += 300 * fruitlove; X3["veg"] += 150 * fruitlove
p = PCA().fit(X3)
print("ratio PC1..PC5:", p.explained_variance_ratio_[:5].round(3))
print("cumulative:", np.cumsum(p.explained_variance_ratio_)[:4].round(3))
w = pd.Series(p.components_[2], X3.columns); print("PC3 biggest weights:", w.abs().nlargest(2).index.tolist(), w[w.abs().nlargest(2).index].round(2).tolist())

Output:

ratio PC1..PC5: [0.531 0.242 0.091 0.022 0.021]
cumulative: [0.531 0.772 0.863 0.885]
PC3 biggest weights: ['fruit', 'rice'] [0.78, -0.27]

PC3 jumps from about 2% to 9.1%, and the cliff moves to after three components (then 2.2%, 2.1%: rubble). PC3 is mostly fruit (weight 0.78): the new habit. Two components now keep only 77.2%, three keep 86.3%. The scree plot tells you how many habits are hiding.

3. Fit a scaler + PCA pipeline on the training rows only and transform the test rows. How different are the axes from a fit on all rows?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA; from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler; from sklearn.model_selection import train_test_split
X_tr, X_te = train_test_split(X, test_size=0.25, random_state=0)
pipe = make_pipeline(StandardScaler(), PCA(n_components=2)).fit(X_tr)   # learns means, stds, axes from TRAIN only
print("train", X_tr.shape, "test", X_te.shape, "  new test columns:", pipe.transform(X_te).shape)
print("ratio learned on train:", pipe[-1].explained_variance_ratio_.round(3))
leak = make_pipeline(StandardScaler(), PCA(n_components=2)).fit(X)       # WRONG: test rows seen
print("PC1 train-only vs all rows, same axis?", round(abs(pipe[-1].components_[0] @ leak[-1].components_[0]), 4))

Output:

train (750, 10) test (250, 10)   new test columns: (250, 2)
ratio learned on train: [0.565 0.265]
PC1 train-only vs all rows, same axis? 0.9999

The pipeline learns the means, standard deviations and axes from the 750 training rows and turns the 250 test rows into 2 columns. The absolute dot product of the two PC1 unit vectors is 0.9999 (1 would be the identical axis): with 1,000 well-behaved rows the leak changes little, but it is still a leak, and with few rows or many columns it matters more. The habit is what counts: fit on train, transform test.

4. Project customer 0 onto PC1 and PC2 by hand, using mean_ and components_. Does it match transform?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
from sklearn.decomposition import PCA
p = PCA(n_components=2).fit(X); c = X.values[:1]                          # project ONE customer by hand
by_hand = (c - p.mean_) @ p.components_.T
print("mean_ (first 3):", p.mean_[:3].round(1))
print("by hand:", by_hand.round(1), "  transform:", p.transform(X.iloc[:1]).round(1))

Output:

mean_ (first 3): [991.4 998.6 997.2]
by hand: [[-902.2  315.9]]   transform: [[-902.2  315.9]]

Centre with the stored means (milk 991.4, bread 998.6, eggs 997.2, ...), then take the dot product with each component: (−902.2, 315.9), the same as transform. That is all transform does (without whitening).

5. Eigenvalues 6, 3, 0.5, 0.3, 0.2. Compute the ratios, the cumulative ratio, and the fewest components for 90%.

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(8); n = 1000                                   # Riya's 1,000 customers (made up)
family, snacks = r.normal(0, 1, n), r.normal(0, 1, n)                    # two hidden habits nobody recorded
cols = ["milk", "bread", "eggs", "veg", "fruit", "rice", "dal", "chips", "soda", "chocolate"]
load = np.array([[1, .9, .8, 1, .7, 1, .9, .1, 0, .2], [.1, .2, 0, 0, -.2, 0, 0, 1, .9, .8]])   # who drives what
X = pd.DataFrame(np.c_[family, snacks] @ load * 300 + 1000 + r.normal(0, 150, (n, 10)), columns=cols).round()
vals = np.array([6, 3, 0.5, 0.3, 0.2])                                  # the quiz eigenvalues
print("ratio:", vals / vals.sum(), "  cumulative:", np.cumsum(vals) / vals.sum())
print("fewest for 90%:", int(np.argmax(np.cumsum(vals) / vals.sum() >= 0.9 - 1e-12)) + 1)

Output:

ratio: [0.6  0.3  0.05 0.03 0.02]   cumulative: [0.6  0.9  0.95 0.98 1.  ]
fewest for 90%: 2

Total = 10. Ratios 0.6, 0.3, 0.05, 0.03, 0.02; cumulative 0.6, 0.9, 0.95, ... so 2 components. np.argmax on a True/False array gives the position of the first True; - 1e-12 is a tiny tolerance so that a sum like 0.8999999999 from floating-point rounding still counts as 0.9.

Quick quiz

1. The eigenvalues of a covariance matrix are 6, 3, 0.5, 0.3 and 0.2. What is the fewest number of components that keeps at least 90% of the variance? Options: 1, 2, 3, 4.

Answer

2. The total is 6 + 3 + 0.5 + 0.3 + 0.2 = 10. One component keeps 6 / 10 = 60%; two keep 9 / 10 = 90%.

2. A covariance matrix is [[5, 4], [4, 5]]. What share of the variance does PC1 keep?

Answer

90%. Eigenvalues 9 and 1 (worked example), total 10, so 9 / 10.

3. Two principal components have weights (0.707, 0.707) and (−0.707, 0.707). What is their dot product, and what does it mean?

Answer

−0.5 + 0.5 = 0: they are orthogonal (at right angles), as principal components always are.

4. With 1,000 rows and 11 columns, what is the largest valid n_components?

Answer

11 = min(1000, 11). Asking for 12 raised ValueError: n_components=12 must be between 0 and min(n_samples, n_features)=11 with svd_solver='covariance_eigh'.

5. Your scratch PCA says PC1 explains 95% and every weight is the same. What did you probably forget?

Answer

Centring. Without subtracting the means, the first direction points at the average row (the mean), not along the spread.

6. The SVD of the centred data gives s₁ = 30 with n = 101 rows. What is the variance along PC1?

Answer

s² / (n − 1) = 900 / 100 = 9.

Interview questions

  1. What is PCA? An unsupervised, linear dimensionality-reduction method: it finds orthogonal directions (principal components) of maximum variance, sorted by variance, and projects the data onto the top k of them.
  2. Feature selection vs feature extraction? Selection keeps some original columns (easy to explain); extraction builds new columns as mixes (PCA). When columns share information, extraction keeps far more: 82.5% vs 23% here.
  3. Walk me through PCA from scratch. Centre the columns; covariance matrix C = Aᵀ A / (n − 1); eigen-decomposition (eigh, symmetric); sort by eigenvalue, biggest first; project: A @ top-k eigenvectors.
  4. What do the eigenvectors and eigenvalues mean? Eigenvectors = the directions (components); eigenvalues = the variance along each; their sum = the total variance.
  5. How does the SVD give PCA? A = U diag(s) Vᵀ for centred A; rows of Vᵀ are the components and s² / (n − 1) the variances. It avoids forming the covariance matrix, which is cheaper and numerically safer with many columns; scikit-learn's PCA offers SVD solvers (its "auto" choice picked "covariance_eigh" for our 1000 × 10 table).
  6. Why must you centre? PCA describes the spread around the mean. Uncentred, the first direction points at the mean (our fake 95% with equal weights).
  7. When should you standardise before PCA? When columns have different units or very different scales; otherwise the largest-scale column takes PC1 (distance in metres: weight 1.00). If all columns share a unit, raw PCA keeps that meaning.
  8. How do you choose the number of components? Scree plot (keep the components before the cliff), a cumulative-ratio target (n_components=0.90), reconstruction error, or the downstream model's validation score. A fixed target can keep noise: 90% needed 6 here, 2 was the honest answer.
  9. What is reconstruction error? Project to k components, map back with inverse_transform, and measure the difference (e.g. RMSE): Rs 132.7 for k = 2, 0 for k = 10.
  10. What is whitening, and when would you use it? Scaling each component to standard deviation 1 (uncorrelated, equal spread), useful before distance-based models; it hides which component carried more variance.
  11. Why can component signs differ between libraries? v and −v are the same axis; solvers may return either. Compare absolute values or align signs.
  12. Limitations of PCA? Linear only; ignores the label (high variance is not always predictive); components are harder to explain; sensitive to scale; must be fitted on training rows only to avoid leakage.

Glossary

Term Meaning (plain words + example)
Row One record: one customer
Column / feature One measured thing per row: monthly Rs on milk
Dimension One column, one axis; Riya's data is 10-dimensional
Dimensionality reduction Fewer columns, losing as little as possible: 10 → 2
Feature selection Keep some original columns: chips and veg (23%)
Feature extraction Build new columns as mixes of the old: PCA (82.5% in 2)
Mean The average: milk's mean is 991.4
Variance Average squared distance from the mean; spread = information
Standard deviation Square root of the variance, in the column's units: PC1's is 751.98
Total variance Sum of all columns' variances = sum of all eigenvalues
Variance as information A column with no spread tells you nothing about who is who
Centring Subtract each column's mean so it has mean 0
Covariance Whether two columns rise together: milk and rice 92,205
Correlation Covariance rescaled to −1..+1; PCA columns have 0
Covariance matrix Square table of all variances (diagonal) and covariances: 10 × 10
Symmetric matrix Cell (i, j) equals cell (j, i), like a covariance matrix
Vector A list of numbers; an arrow
Unit vector An arrow of length 1: (0.707, 0.707)
Matrix A table of numbers
@ (matrix multiplication) (1000 × 10) @ (10 × 2) = (1000 × 2)
Transpose (ᵀ, .T) Rows become columns
Dot product Multiply element by element and add: (1, 2) · (3, 4) = 11
Eigenvector A direction a matrix only stretches: C v = λ v; (1, 1) for [[5, 4], [4, 5]]
Eigenvalue (λ) The stretch factor; in PCA the variance along a component: 9 and 1
Eigen-decomposition Finding all eigenvectors and eigenvalues; np.linalg.eigh
np.linalg.eigh, eigvalsh NumPy's eigen-solvers for symmetric matrices (smallest first)
Principal component (PC) A new axis: an eigenvector of the covariance matrix; PC1 = staples, PC2 = snacks
Loading / weight How much an old column contributes to a component: veg −0.42 in PC1
Orthogonal At right angles; dot product 0
Projection Dropping a point straight onto an axis; position = x · w
PCA via eigendecomposition Centre, covariance matrix, eigenvectors, project
SVD (singular value decomposition) A = U diag(s) Vᵀ
U Left singular vectors: each row's coordinates along the components (before scaling)
s (singular values) Non-negative numbers, biggest first; variance = s² / (n − 1)
Vᵀ Its rows are the principal components
PCA via SVD Components from the SVD of the centred data, no covariance matrix needed
Economy SVD (full_matrices=False) U is n × d instead of n × n
Sign flip v and −v are the same axis; solvers may return either
Explained variance Variance along a component: 565,468 for PC1 (Rs²)
Explained variance ratio Its share of the total: 0.562 for PC1
Cumulative ratio Ratios added up: 0.825 for 2 components
Scree plot Ratios as bars; keep what is before the cliff
Scree Loose rocks at the bottom of a cliff: the small components
Cliff The big drop in the scree plot: after PC2 here
Noise Random variation with no pattern: Rs 150 per value here
n_components (int) Exactly this many components
n_components (float) Fewest components reaching this share: 0.90 → 6
n_components_ How many components the fitted PCA kept
Reconstruction error How far the rebuild from k components is from the original
RMSE (root mean squared error) √(average squared difference): Rs 132.7 for k = 2
transform Project rows onto the components
inverse_transform Map k columns back to the original columns (adds the means back)
fit_transform fit then transform in one call
PCA scikit-learn's class; centres but does not scale
components_ Fitted components, one per row
mean_ Column means PCA subtracted: milk 991.4
explained_variance_, explained_variance_ratio_ Variances along components, and their shares
svd_solver The method PCA uses to compute components: 'covariance_eigh' here
Whitening (whiten=True) Divide each component by its std: all stds 1, still uncorrelated
White noise Noise with equal strength everywhere; gives whitening its name
Scaling with mixed units Standardise when columns use different units (metres vs rupees)
StandardScaler Rescale columns to mean 0, std 1 (Episode 16)
Forgetting to centre Scratch PCA finds the mean: fake 95%
Fit on training rows only Learn means and axes from train, transform test (Episode 15)
Data leakage Test rows influencing what the model learned
Pipeline (make_pipeline) Steps fitted together: scaler then PCA
train_test_split Split rows into training and test parts
Unsupervised Learns without a label; PCA ignores the label
Label The answer column a supervised model predicts
Linear Straight-line structure only; PCA cannot unroll curves
ValueError Python error for a bad value: n_components=12 must be between 0 and ...
textwrap.wrap Splits a long text into short lines
min(n_samples, n_features) Most components possible: min(1000, 11) = 11

Summary

  • Dimensionality reduction gives fewer columns. Keeping the 2 widest original columns (selection) kept only 23%, because the information is shared across columns that move together.
  • PCA builds new columns (extraction) along the directions of greatest spread: centre, covariance matrix, eigenvectors (principal components) and eigenvalues (variance along each), sorted, then project.
  • On Riya's data two components keep 82.5% and recover the hidden habits: PC1 = staples, PC2 = snacks.
  • The SVD of the centred data gives the same components (rows of Vᵀ) and variances (s² / (n − 1)); scikit-learn agrees up to sign flips.
  • Choose k with the explained variance ratio, the scree plot (cliff after 2) and the reconstruction error (Rs 132.7 at k = 2, below the Rs 150 noise); a 90% target asked for 6 components that mostly rebuild noise.
  • Whitening gives every component standard deviation 1.
  • Centre (or get a fake 95%), scale mixed units (or metres take PC1), keep k ≤ min(rows, columns), and fit PCA on training rows only.

Next episode: t-SNE, UMAP and other reductions: methods that unroll curved shapes, and when to trust their pretty pictures.