Machine Learning Full Course · Episode 65. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU, one thread, as in the video's compiler). The output shown is the real output. MNIST is the real dataset; the programs load it from /data/mnist.pt (see "Getting MNIST" below for one line that downloads it with torchvision).
The big idea: an image is a grid of numbers. A plain network that flattens it gives every pixel position its own weights, so a pattern learned in one place doesn't help anywhere else, and a digit moved 3 pixels drops its accuracy from 89.5% to 44.0%. A convolution slides one small grid of weights (a kernel) over the whole image, so the same detector is reused at every position (parameter sharing), and a moved object simply moves its response (translation). Pooling then shrinks the maps and tolerates small shifts. Together: the convolutional neural network (CNN).
Sources: Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE 86(11):2278−2324, 1998 (bottou.org). MNIST: 60,000 training and 10,000 test images of handwritten digits, 28 × 28 pixels, grey (Hugging Face: ylecun/mnist).
Words you need first
- Tensor (Episode 58): a multi-dimensional array of numbers with a shape, e.g.
(28, 28). - MLP (Episode 56): a multilayer perceptron:
nn.Linearlayers with activations in between; every input connects to every neuron. - Weights / parameters: the numbers a network learns.
- ReLU (Episode 56):
max(0, x). - Cross-entropy, Adam, epoch (Episodes 59, 61): the classification loss, a popular optimizer, one pass over the training data.
- Accuracy: the fraction of examples classified correctly.
An image is a tensor
Real-life example: a card show in a cricket stadium. Each fan holds up one card (a pixel), the rows and seats are the height and width, and if each fan holds a red, a green and a blue card, the three layers are the channels.
- Pixel: one square of the image; its value is brightness, 0 = black to 255 = white.
- Channel: one layer of values: 1 for grey, 3 for colour (red, green, blue, RGB).
- Shape: PyTorch stores one image as
(C, H, W)(channels, height, width) and a batch as(N, C, H, W). Many image libraries (PIL, OpenCV, matplotlib) use(H, W, C); convert withx.permute(2, 0, 1). uint8: whole numbers 0..255. Networks want floats:x.float() / 255gives 0..1.
import torch
d = torch.load("/data/mnist.pt") # MNIST: 70,000 handwritten digits
img, label = d["train_x"][0], d["train_y"][0]
print("label:", label.item(), "| shape:", tuple(img.shape))
print("dtype:", img.dtype, "| min", img.min().item(), "| max", img.max().item())
for row in img[4:24]: # 0 = black, 255 = white
print("".join("#" if v > 170 else "+" if v > 60 else "." for v in row.tolist()))
Output:
label: 5 | shape: (28, 28)
dtype: torch.uint8 | min 0 | max 255
............................
................++#.+##+....
..........+++##########+....
........##########+++.......
........##########..........
........+++###...+..........
...........+#+..............
...........+##..............
............##+.............
.............##++...........
.............+###+..........
...............###+.........
................+###........
.................###+.......
...............+####........
.............+######........
...........+######+.........
.........+######+...........
.......#######+.............
.....#######+...............
To the computer, the "5" is nothing but numbers.
The obvious attempt: flatten the image into an MLP
import torch, torch.nn as nn
torch.manual_seed(65); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000] # (N,1,28,28)
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
shift = lambda X, k: torch.roll(X, shifts=k, dims=3) # move every digit k pixels to the right
torch.manual_seed(0)
mlp = nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10)) # 784 = 28 x 28
opt = torch.optim.Adam(mlp.parameters(), lr=1e-3)
for epoch in range(3):
for i in range(0, 12000, 64):
loss = nn.functional.cross_entropy(mlp(X[i:i + 64]), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
acc = lambda X: f"{(mlp(X).argmax(1) == yt).float().mean().item():.1%}"
print("MLP weights:", sum(p.numel() for p in mlp.parameters()))
print("test accuracy:", acc(Xt))
print("digits moved 1, 2, 3 pixels right:", acc(shift(Xt, 1)), acc(shift(Xt, 2)), acc(shift(Xt, 3)))
print("224 x 224 colour photo -> 1,000 neurons:", f"{(224 * 224 * 3 + 1) * 1000:,} weights")
Output:
MLP weights: 101770
test accuracy: 89.5%
digits moved 1, 2, 3 pixels right: 82.6% 63.7% 44.0%
224 x 224 colour photo -> 1,000 neurons: 150,529,000 weights
- 784 inputs (28 × 28), one hidden layer of 128: 101,770 weights, 89.5% on test digits.
- Move every test digit 1, 2, 3 pixels to the right (
torch.roll; MNIST borders are black, so nothing important wraps around): 82.6%, 63.7%, 44.0%. A person wouldn't notice the move. - A normal colour photo (224 × 224 × 3 = 150,528 numbers) into just 1,000 neurons needs 150,529,000 weights in the first layer alone.
Why it failed: after flattening, every pixel position has its own weights. A stroke at column 10 and the same stroke at column 13 are unrelated inputs to the MLP, so what it learned in one place doesn't transfer. Flattening also throws away which pixels are neighbours. We want one small detector reused at every position.
Convolution
Real-life example: a ticket checker (TTE) in a train coach. The railway doesn't hire 72 checkers with 72 rule books: one TTE with one checklist walks seat by seat doing the same check, and ends with a chart of one result per seat. The checklist is the kernel, the walk is the convolution, the chart is the feature map.
- Kernel (filter): a small grid of weights, e.g. 3 × 3.
- At one position: lay the kernel on a 3 × 3 patch, multiply matching numbers, add them all up → one number.
- Slide the kernel one step and repeat over the whole image.
- Feature map (activation map): the grid of results. It answers "where in the image is this pattern?". A 6 × 6 image and a 3 × 3 kernel give a 4 × 4 map (the kernel fits in 4 positions across and 4 down).
image = [[0, 0, 0, 9, 9, 9], # a 6 x 6 picture: dark on the left, bright on the right
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9]]
kernel = [[-1, 0, 1], # a 3 x 3 "vertical edge" detector
[-1, 0, 1],
[-1, 0, 1]]
def conv(img, k): # slide k over img; multiply and add at each spot
n, m = len(img), len(k)
return [[sum(img[i + a][j + b] * k[a][b] for a in range(m) for b in range(m))
for j in range(n - m + 1)] for i in range(n - m + 1)]
fmap = conv(image, kernel)
print("feature map", len(fmap), "x", len(fmap[0]))
for row in fmap: print(row)
Output:
feature map 4 x 4
[0, 27, 27, 0]
[0, 27, 27, 0]
[0, 27, 27, 0]
[0, 27, 27, 0]
Where the patch is one flat colour, the −1s and +1s cancel: 0. Where it sits on the edge, three 9s times +1: 27. This kernel is a vertical edge detector. In a CNN nobody designs kernels: their numbers are learned by gradient descent like any other weights.
Note: what deep learning calls "convolution" is, strictly, cross-correlation: the kernel is not flipped before sliding. Since the kernel is learned, the difference doesn't matter.
Stride, padding and the output size
- Stride
S: how many pixels the kernel jumps each step. Stride 1 visits every position; stride 2 skips every other one, so the map is about half as wide and tall. - Padding
P: a border of zeros around the image, so the kernel can also sit on edge pixels and the output doesn't shrink. A 3 × 3 kernel with padding 1 keeps 28 × 28 at 28 × 28 ("same" padding). - Output size:
out = floor((W − K + 2P) / S) + 1(W = input size, K = kernel size).
import torch, torch.nn as nn, textwrap
x = torch.randn(1, 1, 28, 28) # (batch, channels, height, width)
for k, s, p in [(3, 1, 0), (3, 1, 1), (3, 2, 1), (5, 1, 2), (5, 2, 0)]:
conv = nn.Conv2d(1, 1, kernel_size=k, stride=s, padding=p)
formula = (28 - k + 2 * p) // s + 1 # (W - K + 2P) / S + 1
print(f"kernel {k}, stride {s}, padding {p}: output {tuple(conv(x).shape[2:])} formula {formula}")
img = torch.tensor([[0, 0, 0, 9, 9, 9]] * 6).float().view(1, 1, 6, 6)
edge = nn.Conv2d(1, 1, 3, bias=False)
with torch.no_grad(): edge.weight[:] = torch.tensor([[-1.0, 0, 1]] * 3) # our edge kernel
print("nn.Conv2d on our 6 x 6:", edge(img)[0, 0, 0].tolist(), "(first row)")
try:
nn.Conv2d(1, 1, kernel_size=7)(torch.randn(1, 1, 5, 5))
except RuntimeError as e:
print(textwrap.fill("RuntimeError: " + str(e), 58, subsequent_indent=" "))
Output:
kernel 3, stride 1, padding 0: output (26, 26) formula 26
kernel 3, stride 1, padding 1: output (28, 28) formula 28
kernel 3, stride 2, padding 1: output (14, 14) formula 14
kernel 5, stride 1, padding 2: output (28, 28) formula 28
kernel 5, stride 2, padding 0: output (12, 12) formula 12
nn.Conv2d on our 6 x 6: [0.0, 27.0, 27.0, 0.0] (first row)
RuntimeError: Calculated padded input size per channel: (5
x 5). Kernel size: (7 x 7). Kernel size can't be greater
than actual input size
The formula matches every case; nn.Conv2d with our edge kernel gives exactly our hand-made numbers; and a kernel bigger than the (padded) input is an error.
nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0) expects input (N, C_in, H, W) and returns (N, C_out, H_out, W_out).
Why convolution works: parameter sharing and translation
- Parameter sharing: the same kernel weights are used at every position, so a layer has very few weights, and their number does not depend on the image size.
- Translation (moving something without changing it): if the object moves, its feature map moves the same way (translation equivariance). A pattern learned in one corner is detected in every corner.
- Output channels: a layer has many kernels, one per pattern; each makes its own feature map. A kernel for a colour image is 3 channels deep: weight shape
(out_channels, in_channels, K, K).
import torch, torch.nn as nn
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3) # 16 detectors for RGB images
print("weight shape:", tuple(conv.weight.shape), "| bias:", tuple(conv.bias.shape))
print("parameters:", sum(p.numel() for p in conv.parameters()), "= 16 x (3 x 3 x 3) + 16")
for side in (28, 224, 1000): # the SAME 448 numbers for any image size
print(f"{side} x {side} image -> output", tuple(conv(torch.randn(1, 3, side, side)).shape))
torch.manual_seed(0); edge = nn.Conv2d(1, 1, 3, padding=1)
x = torch.zeros(1, 1, 1, 8); x[0, 0, 0, 2] = 1.0 # a bright dot at position 2
x = x.expand(1, 1, 3, 8).clone()
moved = torch.roll(x, 3, dims=3) # the same dot, 3 pixels right
peak = lambda t: edge(t)[0, 0, 1].argmax().item()
print("strongest response at", peak(x), "| after moving the dot 3 right:", peak(moved))
Output:
weight shape: (16, 3, 3, 3) | bias: (16,)
parameters: 448 = 16 x (3 x 3 x 3) + 16
28 x 28 image -> output (1, 16, 26, 26)
224 x 224 image -> output (1, 16, 222, 222)
1000 x 1000 image -> output (1, 16, 998, 998)
strongest response at 2 | after moving the dot 3 right: 5
448 = 16 kernels × (3 × 3 × 3) weights + 16 biases, for a 28-pixel, a 224-pixel or a 1,000-pixel image alike (compare 150 million). The bright dot's strongest response is at position 2; moved 3 pixels right, it is at 5.
Receptive field
The receptive field of a value is the patch of the original image it can see. One 3 × 3 convolution: 3 × 3 pixels. Each layer looks at the layer below, so the view grows with depth: rf_new = rf + (K − 1) × jump, and every stride (or pooling) multiplies the jump.
| Layer | Receptive field | Jump |
|---|---|---|
| input | 1 × 1 | 1 |
| conv 3 × 3 | 3 × 3 | 1 |
| max pool 2 | 4 × 4 | 2 |
| conv 3 × 3 | 8 × 8 | 2 |
| max pool 2 | 10 × 10 | 4 |
That is how CNNs build meaning: early layers find edges, the next ones corners and curves, deeper ones whole loops and digits.
Pooling
Real-life example: a class monitor reports only the loudest student in each group of benches (max pooling), or the group's average marks (average pooling): one number per group instead of one per student.
- Max pooling
nn.MaxPool2d(2): keep the largest value in each 2 × 2 block (stride 2 by default). - Average pooling
nn.AvgPool2d(2): keep the mean of each block. - Pooling halves the width and height, has zero parameters, and makes the network a little more tolerant of small moves (a shift inside a block disappears): local translation invariance.
import torch, torch.nn as nn
fmap = torch.tensor([[1., 3., 0., 2.],
[4., 6., 1., 0.],
[0., 2., 9., 5.],
[1., 1., 7., 3.]]).view(1, 1, 4, 4) # a 4 x 4 feature map
print("max pool 2x2:", nn.MaxPool2d(2)(fmap)[0, 0].tolist()) # biggest in each 2 x 2 block
print("avg pool 2x2:", nn.AvgPool2d(2)(fmap)[0, 0].tolist()) # average of each block
a = torch.zeros(1, 1, 4, 4); a[0, 0, 0, 0] = 5.0 # a strong feature at (0, 0)
b = torch.zeros(1, 1, 4, 4); b[0, 0, 1, 1] = 5.0 # the same feature, moved 1 down, 1 right
pool = nn.MaxPool2d(2)
print("pooled, original:", pool(a)[0, 0].tolist())
print("pooled, moved: ", pool(b)[0, 0].tolist(), "-> same!")
print("8 maps of 28 x 28 after pooling:", tuple(pool(torch.randn(1, 8, 28, 28)).shape))
print("parameters in a pooling layer:", sum(p.numel() for p in pool.parameters()))
Output:
max pool 2x2: [[6.0, 2.0], [2.0, 9.0]]
avg pool 2x2: [[3.5, 0.75], [1.0, 6.0]]
pooled, original: [[5.0, 0.0], [0.0, 0.0]]
pooled, moved: [[5.0, 0.0], [0.0, 0.0]] -> same!
8 maps of 28 x 28 after pooling: (1, 8, 14, 14)
parameters in a pooling layer: 0
A first CNN on MNIST
Conv (8 kernels) → ReLU → max pool (28 → 14) → conv (16) → ReLU → max pool (14 → 7) → flatten (16 × 7 × 7 = 784) → linear (10 classes). Convolutions find patterns, pooling shrinks, the linear layer decides.
import torch, torch.nn as nn
torch.manual_seed(65); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]; torch.manual_seed(0)
cnn = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), # 28 -> 14
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), # 14 -> 7
nn.Flatten(), nn.Linear(16 * 7 * 7, 10))
opt = torch.optim.Adam(cnn.parameters(), lr=1e-3)
for epoch in range(3):
for i in range(0, 12000, 64):
loss = nn.functional.cross_entropy(cnn(X[i:i + 64]), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
acc = lambda X: f"{(cnn(X).argmax(1) == yt).float().mean().item():.1%}"
print("CNN weights:", sum(p.numel() for p in cnn.parameters()), "| test accuracy:", acc(Xt))
print("digits moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])
h = Xt[:1] # follow ONE image through the layers
for layer in cnn:
h = layer(h)
if not isinstance(layer, nn.ReLU): print(f" after {type(layer).__name__:9}", tuple(h.shape))
Output:
CNN weights: 9098 | test accuracy: 91.0%
digits moved 1, 2, 3 pixels right: 89.1% 79.7% 58.3%
after Conv2d (1, 8, 28, 28)
after MaxPool2d (1, 8, 14, 14)
after Conv2d (1, 16, 14, 14)
after MaxPool2d (1, 16, 7, 7)
after Flatten (1, 784)
after Linear (1, 10)
| Model | Weights | Test | Moved 1 px | Moved 2 px | Moved 3 px |
|---|---|---|---|---|---|
| MLP (flatten) | 101,770 | 89.5% | 82.6% | 63.7% | 44.0% |
| CNN | 9,098 | 91.0% | 89.1% | 79.7% | 58.3% |
Eleven times fewer weights, better accuracy, much more robust to moves. Not perfect: 3 pixels still hurts, because the final flatten + linear layer is again tied to positions. Global average pooling, deeper networks and data augmentation fix most of that (Episode 66).
Getting MNIST
from torchvision import datasets, transforms
train = datasets.MNIST("data", train=True, download=True, transform=transforms.ToTensor()) # floats 0..1, shape (1, 28, 28)
test = datasets.MNIST("data", train=False, download=True, transform=transforms.ToTensor())
The video's programs use the same data saved as tensors (train_x, train_y, test_x, test_y) in one file, so they load in a second.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Flattening images into an MLP | 89.5% → 44.0% when digits move 3 pixels | one weight per position | convolutions |
| Kernel bigger than the input | RuntimeError: Calculated padded input size per channel: (5 x 5). Kernel size: (7 x 7). Kernel size can't be greater than actual input size |
nowhere to place it | smaller kernel or padding |
(H, W, C) images |
wrong shapes / channel errors in Conv2d |
PyTorch is channels-first | x.permute(2, 0, 1) |
| No batch dimension | shape (C, H, W) where (N, C, H, W) is expected |
one image is not a batch | x.unsqueeze(0) |
Raw uint8 pixels |
dtype errors, or huge activations | conv needs floats on a small scale | x.float() / 255 |
Wrong in_features after flatten |
mat1 and mat2 shapes cannot be multiplied |
the map size was miscounted | use the formula, or print shapes layer by layer |
Mini project: see what a learned kernel finds
Train the CNN, then look at what one first-layer kernel responds to on a digit.
import torch, torch.nn as nn
torch.manual_seed(65); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
torch.manual_seed(0)
cnn = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(16 * 7 * 7, 10))
opt = torch.optim.Adam(cnn.parameters(), lr=1e-3)
for epoch in range(3):
for i in range(0, 12000, 64):
loss = nn.functional.cross_entropy(cnn(X[i:i + 64]), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
k = cnn[0].weight[0, 0].detach() # the first learned 3 x 3 kernel
print("kernel 0:\n", k.round(decimals=2))
fmap = torch.relu(cnn[0](X[:1]))[0, 0] # its feature map on the first digit
for row in fmap[4:24].tolist():
print("".join("#" if v > fmap.max() * 0.6 else "+" if v > fmap.max() * 0.25 else "." for v in row))
Output:
kernel 0:
tensor([[ 0.0500, 0.0800, -0.4400],
[-0.4200, -0.3300, 0.0600],
[ 0.0500, 0.3600, 0.1300]])
++++++++++++++++###+###+++++
+++++++++++######+++#....+++
++++++++###++............+++
+++++++++..............+++++
++++++................++++++
++++++.............+++++++++
+++++++..........+++++++++++
++++++++.+.....+++++++++++++
++++++++++.....+++++++++++++
++++++++++.....+++++++++++++
+++++++++++.......++++++++++
++++++++++++.......+++++++++
+++++++++++++........+++++++
+++++++++++++++++.....++++++
+++++++++++++###.....+++++++
+++++++++++###.......+++++++
++++++++++##........++++++++
+++++++###+.......++++++++++
+++++###........++++++++++++
++++#+........++++++++++++++
Compare the map with the "5" drawn at the top of these notes. This kernel's bias makes the empty background mildly active (+), the inside of the strokes is switched off (.), and the strongest responses (#) sit along one side of the strokes: it has learned to be an edge detector for one direction. Try weight[3, 0] for another kernel.
Practice
1. Write a horizontal-edge kernel and test it.
Solution
image = [[0] * 6] * 3 + [[9] * 6] * 3 # dark on top, bright at the bottom
kernel = [[-1, -1, -1], # a horizontal-edge detector
[ 0, 0, 0],
[ 1, 1, 1]]
def conv(img, k):
n, m = len(img), len(k)
return [[sum(img[i + a][j + b] * k[a][b] for a in range(m) for b in range(m))
for j in range(n - m + 1)] for i in range(n - m + 1)]
for row in conv(image, kernel): print(row)
vertical = [[0, 0, 0, 9, 9, 9]] * 6 # the vertical-edge picture from the video
print("on a vertical edge:", conv(vertical, kernel)[0])
Output:
[0, 0, 0, 0]
[27, 27, 27, 27]
[27, 27, 27, 27]
[0, 0, 0, 0]
on a vertical edge: [0, 0, 0, 0]
It fires (27) along the horizontal edge and gives 0 on the vertical-edge picture: each kernel answers one question.
2. Output size of a 64 × 64 input, kernel 3, stride 2, padding 1?
Solution
floor((64 − 3 + 2) / 2) + 1 = floor(31.5) + 1 = 32.
import torch, torch.nn as nn
conv = nn.Conv2d(3, 8, kernel_size=3, stride=2, padding=1)
print(tuple(conv(torch.randn(1, 3, 64, 64)).shape), "formula:", (64 - 3 + 2 * 1) // 2 + 1)
(1, 8, 32, 32) formula: 32
3 and 4. Train the CNN with more kernels, and with average pooling instead of max pooling.
Solution
import torch, torch.nn as nn
torch.manual_seed(65); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
shift = lambda X, k: torch.roll(X, shifts=k, dims=3)
def run(c1, c2, Pool=nn.MaxPool2d):
torch.manual_seed(0)
cnn = nn.Sequential(nn.Conv2d(1, c1, 3, padding=1), nn.ReLU(), Pool(2),
nn.Conv2d(c1, c2, 3, padding=1), nn.ReLU(), Pool(2),
nn.Flatten(), nn.Linear(c2 * 7 * 7, 10))
opt = torch.optim.Adam(cnn.parameters(), lr=1e-3)
for epoch in range(3):
for i in range(0, 12000, 64):
loss = nn.functional.cross_entropy(cnn(X[i:i + 64]), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
acc = lambda X: f"{(cnn(X).argmax(1) == yt).float().mean().item():.1%}"
n = sum(p.numel() for p in cnn.parameters())
print(f"{c1:2}/{c2:2} kernels, {Pool.__name__}: {n:6,} weights | test {acc(Xt)} | moved 1, 2, 3:",
acc(shift(Xt, 1)), acc(shift(Xt, 2)), acc(shift(Xt, 3)))
for c1, c2, Pool in [(8, 16, nn.MaxPool2d), (16, 32, nn.MaxPool2d), (32, 64, nn.MaxPool2d), (8, 16, nn.AvgPool2d)]:
run(c1, c2, Pool)
Output:
8/16 kernels, MaxPool2d: 9,098 weights | test 91.0% | moved 1, 2, 3: 89.1% 79.7% 58.3%
16/32 kernels, MaxPool2d: 20,490 weights | test 94.0% | moved 1, 2, 3: 93.2% 86.8% 68.8%
32/64 kernels, MaxPool2d: 50,186 weights | test 94.2% | moved 1, 2, 3: 94.0% 88.2% 70.0%
8/16 kernels, AvgPool2d: 9,098 weights | test 88.1% | moved 1, 2, 3: 84.1% 67.8% 50.2%
More kernels: more weights and more accuracy (91.0% → 94.0% → 94.2%), and more robust to moves (58.3% → 70.0% at 3 pixels), with diminishing returns. Average pooling instead of max pooling is worse here (88.1%, 50.2% at 3 pixels): max pooling keeps the strongest response in each block, so a stroke that moves a little inside a block still gives the same output; average pooling blurs it.
Quick quiz
x = torch.randn(1, 3, 32, 32)
conv = nn.Conv2d(3, 10, kernel_size=5, stride=1, padding=0)
print(conv(x).shape)
Answer
torch.Size([1, 10, 28, 28]): (32 − 5 + 0) / 1 + 1 = 28, and 10 kernels give 10 output channels.
More questions:
- How many parameters has
nn.Conv2d(3, 16, 3)?Answer
16 × 3 × 3 × 3 + 16 = 448. - What padding keeps a 5 × 5 kernel's output the same size at stride 1?
Answer
2 (in general(K − 1) / 2for odd K). - How many parameters does
nn.MaxPool2d(2)have?Answer
0. - What shape does PyTorch expect for a batch of 32 colour 64 × 64 images?
Answer
(32, 3, 64, 64).
Interview questions
- Why are CNNs better than MLPs for images? Local connectivity (neighbouring pixels), parameter sharing (far fewer weights, independent of image size), translation equivariance (a pattern is found anywhere), and hierarchy (edges → parts → objects through growing receptive fields).
- What do stride and padding do? Stride sets the step between kernel positions (downsampling); padding adds a border so edges are covered and sizes can be preserved. Output
floor((W − K + 2P)/S) + 1. - Equivariance vs invariance? Convolution is translation equivariant (the output moves with the input); pooling (and global pooling) adds some invariance (the output stays the same).
- How many parameters in a conv layer?
C_out × C_in × K × K + C_out. - What is a receptive field and how do you grow it? The input region a value depends on; grow it with depth, larger kernels, strides/pooling, or dilated convolutions.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Pixel | One square of an image with a brightness value: 0..255 |
| Channel | One layer of an image or feature map: grey = 1, RGB = 3 |
| RGB | Red, green, blue: the three colour channels |
| Image as tensor | (C, H, W) for one image, (N, C, H, W) for a batch |
| Channels-first | PyTorch's order (C, H, W); channels-last is (H, W, C) |
uint8 |
Whole numbers 0..255, how images are stored |
| MNIST | 70,000 handwritten digits, 28 × 28, grey (60,000 train / 10,000 test) |
| Flatten | Turning a grid into one long vector: 28 × 28 → 784 |
| Convolution | Sliding a kernel over an image, multiplying and adding at each position |
| Cross-correlation | Convolution without flipping the kernel; what deep learning actually computes |
| Kernel (filter) | A small learned grid of weights: 3 × 3 |
| Feature map (activation map) | The grid of a kernel's results: "where is this pattern?" |
| Edge detector | A kernel that responds where brightness changes: [-1, 0, 1] rows |
| Stride | The step between kernel positions: 2 halves the size |
| Padding | A border of zeros around the input: 1 keeps 28 → 28 with a 3 × 3 kernel |
| "Same" / "valid" padding | Output the same size as the input / no padding |
| Output-size formula | floor((W − K + 2P) / S) + 1 |
nn.Conv2d |
PyTorch's 2-D convolution layer: nn.Conv2d(3, 16, 3) |
| In / out channels | Channels coming into / going out of a conv layer |
| Parameter sharing | The same kernel weights used at every position |
| Translation | Moving an object without changing it |
| Translation equivariance | Input moves → feature map moves the same way |
| Translation invariance | Output unchanged when the input moves a little |
| Receptive field | The patch of the original image one value depends on: 10 × 10 after conv-pool-conv-pool |
| Pooling | Summarising blocks of a feature map with one number |
| Max pooling | Keeping the biggest value per block: nn.MaxPool2d(2) |
| Average pooling | Keeping the mean per block: nn.AvgPool2d(2) |
| CNN (convolutional neural network) | A network built from convolution and pooling layers, ending in a classifier |
torch.roll |
Shifting a tensor along a dimension, wrapping around |
Summary
- An image is a tensor
(C, H, W)of pixels; MNIST digits are(1, 28, 28). - Flattening into an MLP: one weight per pixel position, 150 million weights for a photo, and 89.5% → 44.0% when digits move 3 pixels.
- Convolution slides a small kernel over the image (multiply, add) to make a feature map; kernels are learned.
- Stride and padding set the output size:
floor((W − K + 2P)/S) + 1. - Parameter sharing (448 weights for any image size) and translation equivariance are why it works; stacked layers grow the receptive field.
- Max / average pooling halve the size with zero parameters and tolerate small shifts.
- Our first CNN: 9,098 weights, 91.0%, and far more robust to moved digits than the MLP.
Next episode: CNNs 2: architectures: LeNet, AlexNet, VGG, ResNet and skip connections, 1 × 1 convolutions and global average pooling.