SyntaxPilot
Courses/Machine Learning/Lesson 49
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 49 of 90

t-SNE, UMAP and other reductions

manifoldmanifold learningt-SNEperplexityUMAPn_neighbors and min_distlinear discriminant analysis (LDA)truncated SVDlimits of 2-D plots

Machine Learning Full Course · Episode 49. Every code sample below was run with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1 and umap-learn 0.5.12; the output shown is the real output. The digits data ships with scikit-learn; the three groups, the library register and the sentences are made up for teaching.

The big idea: last episode PCA squeezed ten columns into two by turning straight axes. Give it 1,797 scanned handwritten digits (64 pixel columns each) and its 2-D map is a pile: only 28.5% of the variance survives, and a dot's 5 nearest neighbours on the map carry the right digit only 60.3% of the time (96.3% with all 64 columns). The reason: the digits lie on a curved, low-dimensional sheet inside the 64-dimensional space, a manifold, and straight axes squash it flat. Manifold learning keeps each point's neighbours as its neighbours and lets long distances bend. t-SNE does this and scores 0.976; its perplexity sets how many neighbours count. But a 2-D map lies about sizes (a group 10 times wider came out the smallest) and gaps (gaps of 316 and 633 both became ~43), its axes mean nothing, and t-SNE cannot place new points (AttributeError: 'TSNE' object has no attribute 'transform'). UMAP builds a neighbour graph and lays it out; it is fast and has transform: digits it never saw land among the right digit 97.8% of the time. Its n_neighbors trades local detail for global shape and min_dist sets how tightly points pack. LDA uses the labels: 2 axes score 0.687 vs PCA's 0.618, and it allows at most (classes − 1) axes. Truncated SVD skips centring, so a sparse table stays sparse (2.5 MB instead of 1,600 MB).

Sources: L. van der Maaten and G. Hinton, "Visualizing Data using t-SNE", Journal of Machine Learning Research 9:2579-2605, 2008 (jmlr.org); L. McInnes, J. Healy and J. Melville, "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction", arXiv:1802.03426, 2018 (abstract); the installed scikit-learn 1.9.1: load_digits description (1,797 images of 8 × 8 pixels with values 0..16, a copy of the test set of the UCI handwritten-digits data, written by 13 people), the TSNE docstring (init default "pca" since 1.2), the TruncatedSVD docstring (does not centre the data, so it works on sparse matrices; on word counts it is known as latent semantic analysis).

Words you need first

  • Row, column, feature: one row per digit picture, one column per pixel. A column used as input is a feature. The digits table has 1,797 rows and 64 features.
  • Label (y): the answer column: which digit (0 to 9) the picture shows.
  • Dimension (Episode 48): one column, one axis. The digits are 64-dimensional.
  • Dimensionality reduction and PCA (Episode 48): fewer columns while losing as little as possible; PCA builds new straight axes along the biggest spread.
  • Variance (Episode 10): how spread out values are. Example: 2, 4, 6 has variance 4 (sample variance); 4, 4, 4 has 0.
  • k-nearest neighbours, kNN (Episode 27): classify a point by a majority vote of the k closest training points. Example: if 4 of your 5 nearest neighbours are 3s, you are called a 3.
  • Cross-validation (Episode 15 onwards): split the rows into 5 parts, test on each part once while training on the other 4, average the 5 scores.
  • Train/test split and stratify: hold back rows the model never sees; stratify=y keeps the same share of each digit in both parts.
  • Probability (Episode 9): a number between 0 and 1 for how likely something is.
  • KL divergence (Episode 12): a number for how different two probability distributions are; 0 when they are identical.
  • Gradient descent (Episode 8): improve something step by step by moving downhill on its error.
  • Unsupervised vs supervised (Episode 1): unsupervised methods never look at the label; supervised ones learn from it.
  • SVD (Episodes 5 and 48): splits a table A into U · diag(s) · Vᵀ; the rows of Vᵀ are directions, the s values their strengths.

The real-life hook: reading handwritten numbers

A sorting machine must read the PIN code someone wrote by hand on an envelope. Each digit is a tiny picture. Our data is a public set that comes with scikit-learn: 1,797 digits written by 13 people, each scanned to 8 × 8 = 64 pixels, each pixel a brightness from 0 to 16. So every digit is a row of 64 numbers.

Before training any model you want to look: draw every digit as a dot on a flat map and see whether the 3s sit apart from the 8s, and which digits get confused. A screen has two axes, so we need 64 columns down to 2.

Embedding and the map score

  • An embedding is the new, smaller set of coordinates for each row. A 2-D embedding is a map: each digit becomes one dot.
  • A good map keeps similar things together. To put a number on it, for every dot ask its 5 nearest neighbours on the map which digit they are and take the majority vote; score this with 5-fold cross-validation. We call it the map score. With all 64 columns it is 0.963.

Every digits program below starts with these six lines (we call them HDR):

from sklearn.datasets import load_digits
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
X, y = load_digits(return_X_y=True)                    # 1,797 scanned digits, 8 x 8 pixels each
def map_score(Z):                                      # do your 5 nearest neighbours on the map share your digit?
    return round(cross_val_score(KNeighborsClassifier(5), Z, y, cv=5).mean(), 3)

The obvious attempt: PCA to two columns

# HDR (6 lines above), then:
from sklearn.decomposition import PCA
print("digits:", X.shape, "  pixel brightness", X.min(), "to", X.max())
print("all 64 columns, map score:", map_score(X))
p = PCA(n_components=2).fit(X); Z = p.transform(X)    # obvious attempt: PCA down to 2 columns
print("PCA 2-D keeps", p.explained_variance_ratio_.sum().round(3), "of the variance")
print("PCA 2-D map score:", map_score(Z))

Output:

digits: (1797, 64)   pixel brightness 0.0 to 16.0
all 64 columns, map score: 0.963
PCA 2-D keeps 0.285 of the variance
PCA 2-D map score: 0.603
  • Two PCA columns keep only 28.5% of the variance (last episode, two kept 82.5% of Riya's data).
  • The map score falls from 0.963 to 0.603: four times in ten, your neighbours on the map are a different digit.
  • On the plot, 0s and 4s sit apart, but 2, 3, 5 and 8 pile up in the middle.

Why it failed: manifolds

Think of the surface of the Earth. Two numbers (north-south, east-west) find any place, so the surface is a 2-D sheet, but that sheet is curved through 3-D space.

  • A manifold is a curved sheet with few directions of its own, sitting inside a space with many. Key property: close up it looks flat, like the ground under your feet.
  • The meaningful distance is along the sheet (the road from Mumbai to Delhi), not a straight tunnel through the Earth. Only short straight-line distances can be trusted.
  • Handwritten digits behave like this: a 3 can only bend, slant and thicken in a few ways, so all the 3s lie on a thin curved sheet inside the 64-D space.

The classic toy example is the Swiss roll: a 2-D sheet rolled up in 3-D. PCA can only turn straight axes and drop one, so it squashes the roll and the inner and outer layers land side by side. Points that are close in a straight line (across layers) are far apart along the sheet.

Manifold learning is the family of methods that unroll the sheet: trust only short distances, keep every point's neighbours as its neighbours, and let long distances bend.

t-SNE

t-SNE (t-distributed stochastic neighbour embedding; van der Maaten and Hinton, 2008) in four steps:

  1. In the original 64 columns, for every point, turn distances into neighbour probabilities: close points get a high probability of being "picked as my neighbour", far ones nearly zero.
  2. Drop every point onto a 2-D map (scikit-learn starts from the PCA map by default).
  3. Compute the same neighbour probabilities on the map, but using a t-distribution, a bell curve with fat tails, so moderately far points can sit really far apart. This is what separates the islands.
  4. Move the dots step by step (gradient descent) until the map's neighbour probabilities match the original ones, measured by the KL divergence.

"Stochastic" means it involves probabilities and randomness.

# HDR (6 lines above), then:
from sklearn.manifold import TSNE; import time
t = time.time()
T = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)
print("t-SNE 2-D:", T.shape, f"  took {time.time() - t:.1f} s")
print("t-SNE 2-D map score:", map_score(T))

Output:

t-SNE 2-D: (1797, 2)   took 4.2 s
t-SNE 2-D map score: 0.976

The neighbours survive the squeeze: 0.976, above even the 64 raw columns, and far above PCA's 0.603. (The seconds depend on your computer.) On the plot, each digit gets its own island.

Perplexity

  • Perplexity is roughly how many neighbours each point pays attention to. Small: only the closest few. Large: a wider circle.
  • scikit-learn's default is 30, and it must be smaller than the number of rows.
  • No single value is right: try a few and believe only what stays the same.
Perplexity What the map looked like Map score
2 Tiny crumbs scattered everywhere 0.962
30 Clean islands 0.976
100 Clean islands, a little more global layout 0.964

What stayed the same in all three: ten groups, and the 1s sitting next to the 8s. That is what you can trust.

The limits of 2-D plots

A metro map shows which stations connect, in the right order, but it is not to scale, and every station gets the same dot. A t-SNE map is the same.

import numpy as np
from sklearn.manifold import TSNE
r = np.random.default_rng(0)                           # three made-up groups, 10 columns each
A = r.normal(0, 1, (200, 10))                          # tight
B = r.normal(0, 10, (200, 10)) + 100                   # 10 times wider
C = r.normal(0, 1, (200, 10)) + 300                    # tight, and twice as far from B as A is
X = np.r_[A, B, C]; T = TSNE(n_components=2, random_state=0).fit_transform(X)
g = [slice(0, 200), slice(200, 400), slice(400, 600)]
for name, s in zip("ABC", g):
    print(f"group {name}: spread before {X[s].std():5.1f}   on the map {T[s].std():5.1f}")
gap = lambda D, i, j: np.linalg.norm(D[g[i]].mean(0) - D[g[j]].mean(0)).round()
print("gap A-B before", gap(X, 0, 1), " on the map", gap(T, 0, 1))
print("gap B-C before", gap(X, 1, 2), " on the map", gap(T, 1, 2))
TSNE(n_components=2).fit(X).transform(X[:5])          # MISTAKE: place new points on the map

Output:

group A: spread before   1.0   on the map  22.5
group B: spread before  10.0   on the map   4.7
group C: spread before   1.0   on the map  21.5
gap A-B before 316.0  on the map 43.0
gap B-C before 633.0  on the map 44.0
Traceback (most recent call last):
  File "main.py", line 14, in <module>
    TSNE(n_components=2).fit(X).transform(X[:5])          # MISTAKE: place new points on the map
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TSNE' object has no attribute 'transform'. Did you mean: 'fit_transform'?
  • np.r_[A, B, C] stacks the three groups into one 600 × 10 table; slice(0, 200) picks rows 0 to 199.
  • Sizes lie: B was 10 times wider than A, yet on the map A is wide (22.5) and B the smallest (4.7). t-SNE gives each group roughly its own space.
  • Gaps lie: B to C was twice as far as A to B (633 vs 316); on the map both gaps are ~43.
  • No new points: t-SNE moves the dots it was given; it learns no rule. Adding one digit means re-running everything, and the map comes out different.
Trust Don't trust
Who is next to whom Island sizes
Groups that appear in every run and setting Gaps between islands
The axes: no meaning, no units (unlike PCA's "PC1 = family size")
One single run: change perplexity, init or the seed and check again

Squeezing 64 directions into 2 always throws something away; a beautiful map can still hide overlaps. Use maps to look and ask questions, never as proof on their own.

UMAP

UMAP (uniform manifold approximation and projection; McInnes, Healy and Melville, 2018):

  1. Build a graph: link each point to its nearest neighbours, like a friendship network.
  2. Lay the graph out in 2-D so linked points stay close and unlinked ones are pushed apart.

Two settings:

  • n_neighbors (default 15): how many links each point gets. Small = local detail (tight islands, stray crumbs); large = more of the overall shape.
  • min_dist (default 0.1): how tightly points may pack on the map. 0 = dense clumps; larger = puffed-out islands.

Practical win: UMAP learns a reusable map. Fit once, then transform new points. It lives in the separate package umap-learn (pip install umap-learn, then import umap).

# HDR (6 lines above), then:
import umap, warnings; warnings.filterwarnings("ignore")   # hide UMAP's note about using one CPU core
from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
u = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit(Xtr)   # learn the map on training digits
Ztr, Zte = u.embedding_, u.transform(Xte)             # then PLACE new digits on it
print("training map:", Ztr.shape, "  new digits placed:", Zte.shape)
acc = KNeighborsClassifier(5).fit(Ztr, ytr).score(Zte, yte)
print(f"new digits, 5 nearest neighbours on the map: {acc:.3f} correct")

Output:

training map: (1347, 2)   new digits placed: (450, 2)
new digits, 5 nearest neighbours on the map: 0.978 correct
  • u.embedding_ holds the map coordinates of the training digits.
  • The warning we hide says that fixing random_state makes UMAP use one CPU core (reproducible but slower).
  • 97.8% of digits the map never saw landed among neighbours of the right digit: a fair, out-of-sample test.
Setting What the map looked like
n_neighbors=5, min_dist=0.1 Very tight islands, a few stray crumbs
n_neighbors=50, min_dist=0.1 Related digits (1, 8, 3, 9) huddle together: more global layout
n_neighbors=15, min_dist=0.9 Same islands, puffed out so you see the points inside

LDA: a reduction that uses the labels

PCA, t-SNE and UMAP are unsupervised. Linear discriminant analysis (LDA) is supervised: like PCA it finds straight axes, but it chooses axes where the class centres are far apart while each class stays tight, like seating a wedding so each family sits together and families are clearly apart.

  • At most (number of classes − 1) axes: 10 digits give at most 9. (Two centres define a line, three a plane, and so on.)
  • Not to be confused with latent Dirichlet allocation, a text-mining method also called LDA.
# HDR (6 lines above), then:
import textwrap
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
for name, m, k in [("PCA", PCA, 2), ("LDA", LDA, 2), ("LDA", LDA, 9)]:
    red = m(n_components=k).fit(Xtr, ytr)                # PCA ignores y, LDA uses it
    acc = KNeighborsClassifier(5).fit(red.transform(Xtr), ytr).score(red.transform(Xte), yte)
    print(f"{name} {k}-D: new digits {acc:.3f} correct")
try: LDA(n_components=10).fit(Xtr, ytr)                  # MISTAKE: 10 classes, ask for 10
except ValueError as e: print(*textwrap.wrap("ValueError: " + str(e), 62), sep="\n")

Output:

PCA 2-D: new digits 0.618 correct
LDA 2-D: new digits 0.687 correct
LDA 9-D: new digits 0.967 correct
ValueError: n_components cannot be larger than min(n_features,
n_classes - 1).
  • PCA.fit(Xtr, ytr) accepts y but ignores it; LDA uses it.
  • With 2 axes, using the labels helps (0.687 vs 0.618). With all 9 allowed axes LDA reaches 0.967.
  • Asking for 10 raises the ValueError.

Truncated SVD for sparse tables

A library register: 20,000 readers × 10,000 books, a cell is 1 if that reader borrowed that book. Each reader borrowed a handful, so almost every cell is 0. A table that is mostly zeros is sparse, and a sparse matrix stores only the non-zero cells and their positions.

PCA's first step, centring (subtracting each column's mean), turns every 0 into a small negative number, so nothing is zero any more and the whole table must be stored. Truncated SVD runs the SVD without centring: the zeros stay zeros. On word-count tables it is called latent semantic analysis (LSA).

import numpy as np, scipy.sparse as sp
from sklearn.decomposition import TruncatedSVD
W = sp.random(20000, 10000, density=0.001, format="csr", random_state=0, data_rvs=np.ones)  # 1 = borrowed
print("readers x books:", W.shape, "  filled cells:", W.nnz, f"({W.nnz / W.shape[0] / W.shape[1]:.1%})")
print("stored sparse, MB:", round((W.data.nbytes + W.indices.nbytes + W.indptr.nbytes) / 1e6, 1))
small = W[:2000, :1000].toarray()                      # centre a small corner, the PCA way
print("corner: filled before", np.count_nonzero(small), "  after centring",
      np.count_nonzero(small - small.mean(axis=0)), "of", small.size)
print("centring all of it needs MB:", 20000 * 10000 * 8 / 1e6)
svd = TruncatedSVD(n_components=50, random_state=0).fit(W)   # no centring: the zeros stay zeros
print("TruncatedSVD:", svd.transform(W).shape, "  variance kept by 50:", svd.explained_variance_ratio_.sum().round(3))

Output:

readers x books: (20000, 10000)   filled cells: 200000 (0.1%)
stored sparse, MB: 2.5
corner: filled before 1982   after centring 1738000 of 2000000
centring all of it needs MB: 1600.0
TruncatedSVD: (20000, 50)   variance kept by 50: 0.014
  • sp.random(..., density=0.001, data_rvs=np.ones) makes a random sparse table with 0.1% of cells set to 1. format="csr" is the compressed sparse row format; W.nnz is the number of stored (non-zero) cells.
  • The sparse table takes 2.5 MB. A dense table of 200 million 8-byte numbers needs 1,600 MB.
  • Centring a corner turned 1,982 filled cells into 1.7 million (columns with no borrowings in the corner have mean 0 and stay 0).
  • These borrowings are pure random, so 50 components keep only 1.4% of the variance. Real borrowing has patterns (readers of one cricket book borrow others), and then a few components capture a lot.

Note: scikit-learn 1.9's PCA can take sparse input with some solvers ("arpack", "covariance_eigh") by centring implicitly; TruncatedSVD is the standard tool for sparse and text data.

Which reduction when?

Method Learns from labels? Straight or curved? New points (transform)? Best for
PCA No Straight axes Yes Features for models, compression, straight-line structure
t-SNE No Curved (neighbours) No Pictures only
UMAP No (optional) Curved (neighbour graph) Yes Pictures and features, big data
LDA Yes Straight axes Yes Separating known classes, at most classes − 1 axes
Truncated SVD No Straight axes, no centring Yes Sparse tables, word counts (LSA)

Common mistakes

Mistake Exact message / symptom Why Fix
PCA to 2-D on curved data map score 0.603 (0.963 with 64 columns) Straight axes squash the manifold t-SNE or UMAP to look
Reading island sizes spread 10 → 4.7, spread 1 → 22.5 t-SNE equalises local density Trust neighbours only
Reading gaps between islands gaps 316 and 633 → 43 and 44 Long distances are not kept Don't compare gaps
Placing new points with t-SNE AttributeError: 'TSNE' object has no attribute 'transform'. Did you mean: 'fit_transform'? No mapping is learnt UMAP or PCA
Asking LDA for as many axes as classes ValueError: n_components cannot be larger than min(n_features, n_classes - 1). Max classes − 1 n_components ≤ 9 for 10 digits
Centring a huge sparse table 2.5 MB → 1,600 MB Zeros stop being zero TruncatedSVD
Trusting one map crumbs at perplexity 2, islands at 30 Settings change the picture Try several settings
Feeding t-SNE coordinates to a model — No transform for test rows; leakage if fitted on all rows Use UMAP or PCA inside a pipeline

Mini project: which digits get confused?

Learn a UMAP map on the training digits, place the test digits on it, and list which test digits landed among the wrong digit.

import umap, warnings; warnings.filterwarnings("ignore")
import numpy as np, pandas as pd
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
u = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit(Xtr)        # 1) learn the map on training digits
Zte = u.transform(Xte)                                                      # 2) place the test digits
pred = KNeighborsClassifier(5).fit(u.embedding_, ytr).predict(Zte)           # 3) what do their map neighbours say?
wrong = pd.DataFrame({"true": yte, "map says": pred})[pred != yte]
print("test digits:", len(yte), "  landed among the wrong digit:", len(wrong))
print(wrong.value_counts().head(5).to_string())

Output:

test digits: 450   landed among the wrong digit: 10
true  map says
8     1           3
4     7           2
9     3           1
8     7           1
      3           1
  • Only 10 of 450 test digits landed in the wrong place. The most common confusion: an 8 drawn like a 1 (3 times), then 4 vs 7 (twice).
  • value_counts() on two columns counts each (true, map says) pair.
  • Next step a human would take: plot those 10 pictures (X[...].reshape(8, 8)) and look at them.

Practice

1. Run t-SNE on the digits with random_state 0, 1 and 2. Does the map change?

Solution

With the default init="pca", the start is the same every time, so the seed barely matters. Use init="random" to see the effect of the seed:

from sklearn.datasets import load_digits
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.manifold import TSNE
X, y = load_digits(return_X_y=True)
for seed in (0, 1, 2):
    T = TSNE(n_components=2, init="random", random_state=seed).fit_transform(X)
    s = cross_val_score(KNeighborsClassifier(5), T, y, cv=5).mean()
    print(f"random_state={seed}: map score {s:.3f}   first digit at ({T[0, 0]:.1f}, {T[0, 1]:.1f})")

Output:

random_state=0: map score 0.973   first digit at (-2.1, -58.6)
random_state=1: map score 0.973   first digit at (-19.0, 55.0)
random_state=2: map score 0.970   first digit at (57.2, -4.4)

The score barely moves, but the first digit lands in a completely different place each time: the islands are rotated, flipped and rearranged. Only neighbourhoods are stable; positions are not.

2. UMAP with min_dist=0.0 and min_dist=0.5: how tightly packed is each digit's island?

Solution
import umap, warnings; warnings.filterwarnings("ignore")
import numpy as np
from sklearn.datasets import load_digits
from scipy.spatial.distance import cdist
X, y = load_digits(return_X_y=True)
for md in (0.0, 0.5):
    Z = umap.UMAP(n_neighbors=15, min_dist=md, random_state=0).fit_transform(X)
    inside = np.mean([cdist(Z[y == d], Z[y == d]).mean() for d in range(10)])
    print(f"min_dist={md}: average distance between dots of the same digit {inside:.2f}")

Output:

min_dist=0.0: average distance between dots of the same digit 1.86
min_dist=0.5: average distance between dots of the same digit 2.38

cdist(A, A) gives every pairwise distance inside one digit's dots. A bigger min_dist spreads each island out (2.38 vs 1.86).

3. Use LDA with 9 axes as the first step of a classifier. Compare with logistic regression on all 64 pixels.

Solution
from sklearn.datasets import load_digits
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
for name, model in [("64 pixels", make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))),
                    ("LDA 9-D", make_pipeline(LDA(n_components=9), LogisticRegression(max_iter=2000)))]:
    print(f"{name:9}: test accuracy {model.fit(Xtr, ytr).score(Xte, yte):.3f}")

Output:

64 pixels: test accuracy 0.969
LDA 9-D  : test accuracy 0.958

From 64 columns to 9, accuracy drops only from 96.9% to 95.8%. Inside a pipeline, LDA is fitted on the training rows only.

4. Truncated SVD on the word counts of ten short sentences: do cricket and market sentences separate?

Solution
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import TruncatedSVD
docs = ["Kohli scores a century in the test match", "the bowler takes five wickets in the match",
        "India wins the test series", "the batsman hits a six and a four", "the captain wins the toss",
        "onion prices rise in the market", "tomato prices fall after the rains", "the market sells rice and dal",
        "vegetable prices rise again this week", "farmers bring onions and tomatoes to the market"]
W = CountVectorizer(stop_words="english").fit(docs); M = W.transform(docs)
print("word counts:", M.shape, "  filled:", M.nnz, "of", M.shape[0] * M.shape[1])
Z = TruncatedSVD(n_components=2, random_state=0).fit_transform(M)
for d, z in zip(docs, Z): print(f"{z[0]:5.2f} {z[1]:5.2f}  {d[:38]}")

Output:

word counts: (10, 31)   filled: 39 of 310
 0.00  1.94  Kohli scores a century in the test mat
 0.00  0.94  the bowler takes five wickets in the m
-0.00  1.12  India wins the test series
 0.00 -0.00  the batsman hits a six and a four
-0.00  0.37  the captain wins the toss
 1.67  0.00  onion prices rise in the market
 0.91  0.00  tomato prices fall after the rains
 0.85  0.00  the market sells rice and dal
 1.30  0.00  vegetable prices rise again this week
 1.11  0.00  farmers bring onions and tomatoes to the market

CountVectorizer turns each sentence into word counts (a sparse matrix, 39 of 310 cells filled); stop_words="english" drops words like "the" and "and". Component 1 is "market and prices", component 2 "test and match". The batsman sentence scores 0, 0: after dropping stop words its words ("batsman", "hits") appear in no other sentence, so they share nothing with either topic.

Quick quiz

1. A flower dataset has 3 species and 20 measurement columns. How many axes can LDA give at most? Options: 1, 2, 3, 20.

Answer

2 = min(20 columns, 3 classes − 1).

2. On a t-SNE map, island A is three times bigger than island B. What can you conclude?

Answer

Nothing about their real spread. In our test, a group 10 times wider came out the smallest island.

3. Which of PCA, t-SNE, UMAP can place a new point on an existing map?

Answer

PCA and UMAP (both have transform). t-SNE raises AttributeError: 'TSNE' object has no attribute 'transform'.

4. Your UMAP map shows tiny islands and many stray crumbs. Which setting would you raise to see more global structure?

Answer

n_neighbors (more links per point, more of the overall shape).

5. Why does truncated SVD work on sparse tables when textbook PCA does not?

Answer

It does not centre the data, so zeros stay zeros and the table stays sparse. Centring would fill almost every cell.

Interview questions

  1. Why can PCA fail as a visualisation? It is linear: it can only turn straight axes. Data on a curved manifold gets squashed and different parts overlap (digits: map score 0.603 vs 0.976 for t-SNE).
  2. What is a manifold, in plain words? A low-dimensional, possibly curved sheet inside a high-dimensional space that looks flat close up, like the Earth's surface.
  3. Explain t-SNE. Turn distances into neighbour probabilities in the original space (Gaussian), do the same on a low-dimensional map with a heavy-tailed t-distribution, and move the map points by gradient descent to minimise the KL divergence between the two.
  4. What does perplexity control? Roughly the effective number of neighbours per point. Low values give fragmented maps; it must be below the number of rows; default 30 in scikit-learn.
  5. What can and can't you read from a t-SNE plot? Can: local neighbourhoods and clusters that are stable across settings. Can't: cluster sizes, distances between clusters, the axes, or anything from a single run.
  6. t-SNE vs UMAP? Both preserve local neighbourhoods. UMAP builds a neighbour graph and optimises a layout of it; it is usually faster on large data, tends to keep more global layout, and has transform for new points. Settings: n_neighbors, min_dist.
  7. Should you use t-SNE output as model features? Generally no: there is no transform for new rows, and fitting on all rows leaks test information. Use PCA, UMAP or LDA inside a pipeline.
  8. LDA vs PCA? PCA (unsupervised) maximises variance; LDA (supervised) maximises separation between class centres relative to spread within classes, with at most classes − 1 components.
  9. When do you use truncated SVD? For large sparse matrices (word counts, TF-IDF, user-item tables), because it skips centring and keeps the matrix sparse; on text it is latent semantic analysis.
  10. How would you check that a 2-D map is trustworthy? Measure neighbour preservation (a kNN score or trustworthiness), compare several settings and seeds, and confirm patterns with the original features.

Glossary

Term Meaning (plain words + example)
Embedding The new low-dimensional coordinates of each row: a 2-D embedding is a map with one dot per digit
Map score (this episode) 5-NN cross-validated accuracy on the map: 0.603 for PCA, 0.976 for t-SNE
Manifold A curved sheet with few directions inside a space with many; the Earth's surface is a 2-D manifold in 3-D
Locally flat Close up, a manifold looks like a flat plane, like the ground under your feet
Manifold learning Methods that unroll the curved sheet by keeping neighbours together: t-SNE, UMAP
Swiss roll A toy dataset: a 2-D sheet rolled up in 3-D, which PCA squashes
Neighbour probability In t-SNE, how likely point j is picked as point i's neighbour; close = high
t-distribution A bell curve with fatter tails than the normal; lets moderately far points sit far apart on the map
t-SNE t-distributed stochastic neighbour embedding (2008): matches neighbour probabilities between the original space and a map
Stochastic Involving probabilities and randomness
Perplexity t-SNE's setting for roughly how many neighbours count; default 30
init t-SNE's starting map: "pca" (default, stable) or "random"
Limits of 2-D plots Sizes, gaps and axes on a t-SNE/UMAP map don't mean anything; one run is one opinion
UMAP Uniform manifold approximation and projection (2018): lays out a nearest-neighbour graph; has transform
n_neighbors UMAP: links per point; small = local detail, large = global shape; default 15
min_dist UMAP: how tightly points may pack; 0 = dense clumps; default 0.1
umap-learn The Python package that provides umap.UMAP
LDA (linear discriminant analysis) Supervised straight-line reduction that pushes class centres apart; at most classes − 1 axes
Latent Dirichlet allocation A different "LDA": a topic model for text
Supervised / unsupervised Uses the labels / ignores them
Sparse table, sparse matrix Mostly zeros; stored as only the non-zero cells and their positions (CSR format)
nnz Number of stored non-zero cells: 200,000 in the register
Centring Subtracting each column's mean; turns zeros into non-zeros
Truncated SVD SVD that keeps only the top k components and skips centring; works on sparse data
LSA (latent semantic analysis) Truncated SVD applied to word-count or TF-IDF tables
CountVectorizer Turns sentences into a sparse table of word counts
Stop words Very common words ("the", "and") dropped before counting
AttributeError Python error when an object has no such attribute or method: 'TSNE' object has no attribute 'transform'
ValueError Python error for a bad value: n_components cannot be larger than min(n_features, n_classes - 1).

Summary

  • On 64-pixel digits, PCA to 2-D keeps 28.5% of the variance and its map score falls to 0.603: straight axes squash the data's curved manifold.
  • Manifold learning keeps neighbours as neighbours. t-SNE matches neighbour probabilities (t-distribution on the map, KL divergence) and scores 0.976; perplexity sets how many neighbours count.
  • Limits of 2-D plots: island sizes and gaps lie (spread 10 → 4.7; gaps 316 and 633 → 43 and 44), axes mean nothing, settings change the picture, and t-SNE has no transform.
  • UMAP lays out a neighbour graph, has transform (new digits: 0.978), and is tuned with n_neighbors and min_dist.
  • LDA uses the labels (2-D: 0.687 vs PCA 0.618) and allows at most classes − 1 axes.
  • Truncated SVD skips centring, so sparse tables stay sparse (2.5 MB instead of 1,600 MB); on text it is LSA.

Next episode: Time series 1: fundamentals: dates as an index, trend, seasonality, stationarity, ACF and PACF.