Machine Learning Full Course · Episode 32. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The homes and lassi sales are made up for teaching; the mini project uses the handwritten digits dataset bundled with scikit-learn.
The big idea: when no straight road separates the classes, add new features made from the old ones (a feature map) so that a straight road works in the new space; it becomes a curved road in the original space. Building those features explodes in size, but the SVM only ever needs dot products between points, and a kernel computes those dot products directly without building the features: the kernel trick. The polynomial and RBF (Gaussian) kernels are the common choices. gamma sets how far one point's influence reaches (big gamma overfits), C is still the price of mistakes, and features must be scaled. Support vector regression (SVR) uses the same machinery for numbers, with an ε-tube inside which errors are free.
Words you need first
- SVM, hyperplane, margin, support vectors, C, hinge loss: Episode 31.
- Dot product, L2 norm, distance: Episode 3.
- Polynomial features: adding squares and products of features so a linear model can bend (Episode 19).
- Normal (Gaussian) distribution, binomial, variance: Episode 10.
- Overfitting / underfitting: memorising noise / missing the pattern (Episode 20).
- Scaling, pipeline: Episodes 16-17. Cross-validation: Episode 26. MAE (mean absolute error): the average size of the errors (Episode 19).
The problem: no straight road works
Raju's sweet shop delivers to 400 made-up homes, recorded as km east and km north of the shop. Homes within about 3.5 km order (+1); farther ones don't (−1). The ordering homes form a round patch around the shop.
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(33); n = 400 # 400 made-up homes around Raju's sweet shop
X = rng.uniform(-6, 6, (n, 2)) # km east, km north of the shop
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1) # +1 = orders (lives close), -1 = doesn't
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
lin = SVC(kernel="linear").fit(X_tr, y_tr)
print(f"straight road: train {lin.score(X_tr, y_tr):.3f} test {lin.score(X_te, y_te):.3f}")
print("homes it says will order:", (lin.predict(X_te) == 1).sum(), "of", len(X_te))
def phi(X): return np.c_[X, (X ** 2).sum(1)] # feature map: add east^2 + north^2
lin3 = SVC(kernel="linear").fit(phi(X_tr), y_tr)
print(f"with x1^2 + x2^2 added: train {lin3.score(phi(X_tr), y_tr):.3f} test {lin3.score(phi(X_te), y_te):.3f}")
w, b = lin3.coef_[0], lin3.intercept_[0]
print("weights:", w.round(2), " boundary radius:", np.sqrt(-b / w[2]).round(2), "km")
Output:
straight road: train 0.730 test 0.770
homes it says will order: 0 of 100
with x1^2 + x2^2 added: train 0.960 test 0.960
weights: [ 0.04 -0.05 -0.47] boundary radius: 3.48 km
- The linear SVM looks like 0.77 accurate but predicts zero orders: it answers "no" for everyone, which is right for the ~73-77% of homes that really don't order. Accuracy hides it (the accuracy paradox, Episode 24).
- Why it fails: a circle can't be cut out by one straight line.
- Adding one feature, east² + north² (the squared distance), makes the classes separable by a flat cut: 0.960, and the boundary radius 3.48 km recovers the hidden 3.5 km rule.

Feature maps
- Feature map φ (phi): a function that turns each point into new features, e.g. φ(east, north) = (east, north, east² + north²).
- A straight boundary in the new space is a curved boundary in the original space.
- Same idea as polynomial features (Episode 19).
- Problem: in real data you don't know which features to add, so you'd add all products up to some degree. The count explodes: with n features and degree d there are C(n + d, d) terms (see below).
The kernel trick
Key fact: solving an SVM (its "dual form") and making predictions only use the training points through dot products between pairs of points:
f(x) = Σ over support vectors of αᵢ yᵢ (φ(xᵢ) · φ(x)) + b
The αᵢ are weights found by the solver (zero for non-support vectors).
- Kernel: a function k(x, z) that returns φ(x) · φ(z) without computing φ.
- Kernel trick: replace every dot product in the SVM with k(x, z). The SVM then behaves as if it worked in the big feature space, at the cost of the original small one.
Analogy: comparing two giant gift boxes of sweets. Unpacking every sweet (building φ) is slow; a formula that gives the same similarity from the outside (the kernel) is fast.
import numpy as np
from math import comb
x, z = np.array([2.0, 3.0]), np.array([1.0, 2.0]) # two homes (km east, km north)
def phi(v): # degree-2 feature map: 6 new features
a, b = v
return np.array([a*a, b*b, np.sqrt(2)*a*b, np.sqrt(2)*a, np.sqrt(2)*b, 1])
print("phi(x) =", phi(x).round(3))
print("phi(x) . phi(z) =", round(phi(x) @ phi(z), 6)) # build features, then dot product
print("(x . z + 1)^2 =", (x @ z + 1) ** 2) # kernel: no features built
for n_feat, deg in [(2, 2), (100, 2), (100, 5), (1000, 5)]:
print(f"{n_feat:>5} features, degree {deg}: {comb(n_feat + deg, deg):>16,} mapped features")
gamma = 0.5 # RBF kernel: similarity from distance
for a, b in [(x, x), (x, x + 0.5), (x, z), (x, x + 10)]:
print(f"rbf, distance {np.linalg.norm(a - b):5.2f}: {np.exp(-gamma * np.sum((a - b) ** 2)):.4f}")
Output:
phi(x) = [4. 9. 8.485 2.828 4.243 1. ]
phi(x) . phi(z) = 81.0
(x . z + 1)^2 = 81.0
2 features, degree 2: 6 mapped features
100 features, degree 2: 5,151 mapped features
100 features, degree 5: 96,560,646 mapped features
1000 features, degree 5: 8,459,043,543,951 mapped features
rbf, distance 0.00: 1.0000
rbf, distance 0.71: 0.7788
rbf, distance 1.41: 0.3679
rbf, distance 14.14: 0.0000
Why (x · z + 1)² equals φ(x) · φ(z) for x = (a, b), z = (c, d):
(x · z + 1)² = (ac + bd + 1)²
= a²c² + b²d² + 2abcd + 2ac + 2bd + 1
= [a², b², √2ab, √2a, √2b, 1] · [c², d², √2cd, √2c, √2d, 1] = φ(x) · φ(z)
The √2 factors are exactly what makes the cross terms come out right.
- With 100 features and degree 5, φ would have 96,560,646 entries per row; with 1,000 features, over 8 trillion. The kernel still costs one dot product of the original features plus a power.
The common kernels
| Kernel | Formula (scikit-learn) | Boundary shapes | Main knobs |
|---|---|---|---|
| Linear | k(x, z) = x · z | straight (Episode 31) | C |
| Polynomial | k(x, z) = (γ x · z + coef0)^degree | circles, parabolas, curves of that degree | degree, coef0, gamma, C |
| RBF (Gaussian) | k(x, z) = exp(−γ ‖x − z‖²) | any smooth shape | gamma, C |
| Sigmoid | k(x, z) = tanh(γ x · z + coef0) | rarely used | gamma, coef0 |
coef0defaults to 0 in scikit-learn; with coef0 = 0 the polynomial kernel has only the highest-degree terms, socoef0=1is often a better choice.- RBF = radial basis function: "radial" because it depends only on the distance between the points. It is a bell curve (Gaussian) around each support vector: similarity 1 for the same point, falling to 0 far away (1.0000 → 0.7788 → 0.3679 → 0.0000 above).
- The RBF kernel corresponds to a feature map with infinitely many features; only the kernel can work with it.
SVC()'s default kernel is"rbf".
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(33); n = 400 # the same 400 made-up homes
X = rng.uniform(-6, 6, (n, 2)) # km east, km north of the shop
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1) # +1 = orders, -1 = doesn't
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
for k in [SVC(kernel="poly", degree=2, coef0=1), SVC(kernel="rbf")]:
k.fit(X_tr, y_tr)
print(f"{k.kernel:<5}: train {k.score(X_tr, y_tr):.3f} test {k.score(X_te, y_te):.3f} SVs {len(k.support_)}")
for g in [0.01, 0.1, 1, 10, 100]: # gamma: how far one home's influence reaches
k = SVC(kernel="rbf", gamma=g).fit(X_tr, y_tr)
print(f"gamma={g:<5} train {k.score(X_tr, y_tr):.3f} test {k.score(X_te, y_te):.3f} SVs {len(k.support_):>3}")
Output:
poly : train 0.960 test 0.960 SVs 81
rbf : train 0.953 test 0.950 SVs 83
gamma=0.01 train 0.947 test 0.930 SVs 143
gamma=0.1 train 0.957 test 0.930 SVs 74
gamma=1 train 0.970 test 0.920 SVs 156
gamma=10 train 0.990 test 0.820 SVs 287
gamma=100 train 1.000 test 0.770 SVs 300

gamma and C
- gamma (γ): how far a single training point's influence reaches. Small γ = wide bumps, smooth boundary (can underfit). Big γ = narrow bumps, an island around each point (overfits: γ = 100 gives train 1.000, test 0.770, and all 300 points become support vectors).
- C: the price of a margin violation (Episode 31). Big C also lets the boundary bend around single points, so C and γ interact.
- Default
gamma="scale"= 1 / (n_features × X.var()), where X.var() is the variance of all the input values together. Sensible only if features are on similar scales. (gamma="auto"= 1 / n_features.) - Tune C and γ together on a log grid with cross-validation.
| small | large | |
|---|---|---|
| gamma | smooth, may underfit | wiggly islands, overfits |
| C | wide margin, more violations | narrow margin, fits training points |
Scaling and tuning
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
rng = np.random.default_rng(33); n = 400 # the same 400 made-up homes
X = rng.uniform(-6, 6, (n, 2))
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)
X = X * [1000, 1] # east now in METRES, north still in km
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
raw = SVC(kernel="rbf", gamma=0.1).fit(X_tr, y_tr)
print(f"unscaled: train {raw.score(X_tr, y_tr):.3f} test {raw.score(X_te, y_te):.3f}")
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.01, 0.1, 1, 10]}
search = GridSearchCV(pipe, grid, cv=5).fit(X_tr, y_tr)
print("best:", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"scaled + tuned: test {search.score(X_te, y_te):.3f}")
Output:
unscaled: train 0.997 test 0.780
best: {'svc__C': 100, 'svc__gamma': 0.1} CV 0.960
scaled + tuned: test 0.960
- With east in metres, distances are almost entirely about east, so each home becomes an island (train 0.997, test 0.780).
- GridSearchCV: tries every combination in the grid (4 × 4 = 16), each with 5-fold cross-validation, and refits the best on all training data. Parameter names in a pipeline are
<step>__<param>, e.g.svc__C. (Full treatment: Episode 42.)
Support vector regression (SVR)
Raju's lassi counter: predict glasses sold (a number) from temperature.
- ε-tube (epsilon tube): a band of ±ε around the prediction. Errors inside it cost nothing.
- ε-insensitive loss: max(0, |y − f(x)| − ε). Same flat-then-ramp shape as hinge loss.
- Support vectors in SVR: points on or outside the tube; points inside have no effect.
SVR(kernel="rbf", C=..., gamma=..., epsilon=...); ε is in the units of the target (here, glasses), so scale or think about the target's size when choosing it.
import numpy as np
from sklearn.svm import SVR
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(34) # 200 made-up days at Raju's lassi counter
temp = np.sort(rng.uniform(10, 45, 200)) # temperature, degrees C
glasses = 20 + 0.08 * (temp - 10) ** 2 + rng.normal(0, 5, 200) # sales rise faster on hot days
X = ((temp - 27.5) / 10).reshape(-1, 1) # scaled by hand: centre 27.5, unit 10 C
X_tr, y_tr, X_te, y_te = X[::2], glasses[::2], X[1::2], glasses[1::2]
def eps_loss(err, eps): return np.maximum(0, np.abs(err) - eps) # errors inside the tube are free
print("epsilon=5 loss of errors -8, -3, 0, 3, 8:", eps_loss(np.array([-8, -3, 0, 3, 8]), 5))
def mae(m): return np.mean(np.abs(m.predict(X_te) - y_te))
print(f"LinearRegression: test MAE {mae(LinearRegression().fit(X_tr, y_tr)):.2f} glasses")
for eps in [0.5, 5, 15]:
m = SVR(kernel="rbf", C=100, epsilon=eps).fit(X_tr, y_tr)
print(f"SVR rbf, epsilon={eps:<4}: test MAE {mae(m):.2f} SVs {len(m.support_):>3} of 100")
Output:
epsilon=5 loss of errors -8, -3, 0, 3, 8: [3 0 0 0 3]
LinearRegression: test MAE 8.51 glasses
SVR rbf, epsilon=0.5 : test MAE 4.22 SVs 94 of 100
SVR rbf, epsilon=5 : test MAE 4.05 SVs 36 of 100
SVR rbf, epsilon=15 : test MAE 5.82 SVs 4 of 100
- A straight line can't follow the curve (MAE 8.51). SVR with an RBF kernel can.
- Small ε: almost every point is a support vector. ε = 5 (about the noise level): best error with 36 support vectors. ε = 15: tube too wide, real errors ignored.

When to use an SVM
| Good fit | Poor fit |
|---|---|
| Small to medium data (hundreds to tens of thousands of rows) | Hundreds of thousands+ rows with a kernel: the kernel matrix is n × n (100,000 rows → 10 billion entries) |
| Many features, e.g. text word counts (linear kernel) | Need probabilities out of the box |
| A clear margin; few support vectors | Mixed tables of categories and numbers with little tuning time: try tree ensembles (Episodes 33-36) |
For very large data with a linear boundary use LinearSVC or SGDClassifier(loss="hinge"); for approximate kernels on large data, scikit-learn offers Nystroem and RBFSampler (explicit approximate feature maps followed by a linear model).
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Linear SVM on curved data | test 0.770 but predicts 0 orders | No line cuts a circle | Feature map or kernel |
| Building huge feature maps by hand | 96,560,646 features per row | Grows like C(n + d, d) | Kernel trick |
| Very large gamma | train 1.000, test 0.770, 300/300 SVs | An island per point | Tune gamma with CV |
| Kernel on unscaled features | metres: test 0.780 | Distance = biggest unit | StandardScaler in a pipeline |
| SVR tube too wide | ε = 15: MAE 5.82 | Ignores real errors | Tune ε (start near the noise level) |
| Polynomial kernel with default coef0 = 0 | Only highest-degree terms | No lower-degree terms | Set coef0=1 |
| Wrong pipeline parameter name | ValueError: Invalid parameter 'C' for estimator Pipeline(...) |
Pipeline params need the step prefix | svc__C |
The last mistake, reproduced:
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import make_circles
X, y = make_circles(200, noise=0.1, factor=0.4, random_state=0)
GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"C": [1, 10]}, cv=3).fit(X, y)
Output:
Traceback (most recent call last):
... (library frames) ...
ValueError: Invalid parameter 'C' for estimator Pipeline(steps=[('standardscaler', StandardScaler()), ('svc', SVC())]). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].
Mini project: handwritten digits with an RBF SVM
The digits dataset bundled with scikit-learn has 1,797 images of handwritten digits, each 8 × 8 pixels (64 features, values 0-16), with labels 0-9. SVMs handle many classes by training several binary SVMs (scikit-learn's SVC uses one-vs-one internally).
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = load_digits(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
print("train", X_tr.shape, " classes", sorted(set(y.tolist())))
lin = make_pipeline(StandardScaler(), LinearSVC(max_iter=10000)).fit(X_tr, y_tr)
print(f"linear SVM test {lin.score(X_te, y_te):.3f}")
grid = {"svc__C": [1, 10, 100], "svc__gamma": [0.001, 0.01, 0.1]}
search = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")), grid, cv=5).fit(X_tr, y_tr)
print("best", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"RBF SVM test {search.score(X_te, y_te):.3f}")
print("support vectors:", search.best_estimator_[-1].support_.size, "of", len(X_tr))
Output:
train (1347, 64) classes [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
linear SVM test 0.951
best {'svc__C': 10, 'svc__gamma': 0.01} CV 0.983
RBF SVM test 0.980
support vectors: 619 of 1347
- The RBF kernel beats the linear SVM on this image data.
- 619 of the 1,347 training images end up as support vectors: with ten classes and one-vs-one, many images sit near some pair's boundary.
Practice
1. Check (x · z + 1)² = φ(x) · φ(z) for three more random pairs.
Solution
import numpy as np
def phi(v):
a, b = v
return np.array([a*a, b*b, np.sqrt(2)*a*b, np.sqrt(2)*a, np.sqrt(2)*b, 1])
rng = np.random.default_rng(0)
for _ in range(3):
x, z = rng.normal(size=2), rng.normal(size=2)
print(f"{phi(x) @ phi(z):.6f} {(x @ z + 1) ** 2:.6f}")
Output:
1.137769 1.137769
0.414668 0.414668
1.921898 1.921898
Equal every time (up to rounding).
2. Print a table of 5-fold CV accuracy over C × gamma on the 400 homes (scaled).
Solution
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
rng = np.random.default_rng(33); n = 400
X = rng.uniform(-6, 6, (n, 2)); d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)
Cs, gs = [0.1, 1, 10, 100], [0.01, 0.1, 1, 10]
s = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": Cs, "svc__gamma": gs}, cv=5).fit(X[:300], y[:300])
table = s.cv_results_["mean_test_score"].reshape(len(Cs), len(gs))
print("C \\ gamma " + "".join(f"{g:>7}" for g in gs))
for C, row in zip(Cs, table): print(f"{C:<9} " + "".join(f"{v:7.3f}" for v in row))
Output:
C \ gamma 0.01 0.1 1 10
0.1 0.730 0.730 0.887 0.747
1 0.730 0.937 0.947 0.940
10 0.730 0.953 0.953 0.923
100 0.940 0.960 0.937 0.903
Low C with low gamma underfits (too smooth); high gamma drops as it starts memorising. Good scores lie along a band of moderate C and gamma: the two interact.
3. SVR with a linear kernel on the lassi data. How does it compare?
Solution
import numpy as np
from sklearn.svm import SVR
rng = np.random.default_rng(34)
temp = np.sort(rng.uniform(10, 45, 200))
glasses = 20 + 0.08 * (temp - 10) ** 2 + rng.normal(0, 5, 200)
X = ((temp - 27.5) / 10).reshape(-1, 1)
X_tr, y_tr, X_te, y_te = X[::2], glasses[::2], X[1::2], glasses[1::2]
for k in ["linear", "rbf"]:
m = SVR(kernel=k, C=100, epsilon=5).fit(X_tr, y_tr)
print(f"{k:<6} test MAE {np.mean(np.abs(m.predict(X_te) - y_te)):.2f}")
Output:
linear test MAE 8.41
rbf test MAE 4.05
A linear-kernel SVR is a straight line, so like linear regression it can't follow the curve; the RBF kernel can.
4. Time SVC on 2,000 vs 8,000 rows. How much slower is 4× the data?
Solution
import time
from sklearn.datasets import make_classification
from sklearn.svm import SVC
for n in [2000, 8000]:
X, y = make_classification(n, 20, flip_y=0.1, random_state=0)
t = time.time(); SVC().fit(X, y); print(f"{n} rows: {time.time() - t:.2f} s")
Output (timings vary by machine):
2000 rows: 0.09 s
8000 rows: 1.12 s
Four times the rows takes much more than four times as long: kernel SVM training grows faster than linearly with n.
5. By hand: RBF kernel with γ = 0.5 for two points at distance 2. Value?
Solution
exp(−0.5 × 2²) = exp(−2) ≈ 0.135.
Quick quiz
- x = [1, 2], z = [3, 0]. (x · z + 1)² = ? 16.
- Which kernel depends only on the distance between points? RBF (Gaussian).
- Train 1.000, test 0.770, every point a support vector. Raise or lower gamma? Lower.
- In SVR, which points are support vectors? Points on or outside the ε-tube.
- Why can't you skip scaling with an RBF kernel? It's built on distance; the biggest unit dominates.
Interview questions
- What is the kernel trick? Replacing dot products in an algorithm that only uses dot products (like the SVM dual) with a kernel k(x, z) = φ(x) · φ(z), so it works in a high- or infinite-dimensional feature space without computing it.
- Why can SVMs use kernels but plain gradient-descent linear models usually don't? The SVM dual and prediction depend on data only through dot products between points.
- Explain the RBF kernel and gamma. exp(−γ‖x − z‖²): similarity decays with distance; γ controls the width; large γ → narrow bumps → overfitting.
- How do C and gamma interact? Both control flexibility; high C and high γ overfit, low C and low γ underfit; tune them jointly on a log grid with CV.
- What is a valid kernel? One that equals a dot product in some feature space; equivalently its Gram (kernel) matrix is symmetric positive semi-definite for any set of points (Mercer's condition).
- What is SVR and the ε-insensitive loss? Regression that ignores errors smaller than ε and penalises the excess linearly; points inside the tube are not support vectors.
- Time and memory cost of kernel SVMs? Roughly between O(n²) and O(n³) time and an n × n kernel matrix, so they struggle beyond tens of thousands of rows.
- How do SVMs handle more than two classes? Several binary SVMs: one-vs-one (SVC) or one-vs-rest (LinearSVC).
- When would you choose a linear kernel? Many features relative to rows (e.g. text), or very large data.
- Why does the polynomial kernel need coef0? With coef0 = 0 it contains only the highest-degree terms; coef0 > 0 adds all lower-degree terms.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Feature map φ (phi) | A recipe that builds new features from old ones | φ = (east, north, east² + north²) |
| Feature space | The space of the new features | 3-D space where a flat cut works |
| Kernel k(x, z) | A function returning φ(x) · φ(z) without building φ | (x · z + 1)² = 81 |
| Kernel trick | Swapping every dot product for a kernel | curved roads at the cost of straight ones |
| Dual form | The SVM written using only dot products between points | f(x) = Σ αᵢ yᵢ k(xᵢ, x) + b |
| αᵢ (alpha) | Solver weight of training point i; > 0 only for support vectors | |
| Linear kernel | k = x · z | Episode 31's straight road |
| Polynomial kernel | k = (γ x · z + coef0)^degree | degree 2, coef0 1: test 0.960 |
coef0 |
Constant added inside the polynomial kernel | default 0; we used 1 |
| RBF / Gaussian kernel | k = exp(−γ ‖x − z‖²), a bell curve on distance | test 0.950 |
| Radial basis function | A function of distance from a centre | the RBF kernel |
| gamma (γ) | How far one point's influence reaches | γ = 100 → islands |
gamma="scale" |
Default: 1 / (n_features × X.var()) | |
| GridSearchCV | Tries every combination of settings with CV | best C 100, γ 0.1 |
| Support vector regression (SVR) | SVM for numbers | lassi glasses vs temperature |
| ε-tube | A band ±ε around the prediction with no loss | ±5 glasses |
| ε-insensitive loss | max(0, | error |
| Kernel (Gram) matrix | Table of k(xᵢ, xⱼ) for all pairs | n × n numbers |
| One-vs-one | Multiclass by a binary model for each pair of classes | SVC on 10 digits |
Summary
- When no straight road works, a feature map makes one; the kernel trick computes its dot products without building it.
- Polynomial kernels give curves of a given degree; RBF kernels give any smooth shape and are the default.
- gamma = reach of each point (big = overfit), C = price of mistakes; scale, then tune both with CV.
- SVR predicts numbers with an ε-tube: errors inside are free.
- SVMs suit small/medium data and many features; for huge data use linear models or tree ensembles.
- Next episode: ensembles: voting and bagging.