Machine Learning Full Course · Episode 68. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29.2 on a CPU, and the output shown is the real output. Most pieces are written from scratch in plain Python first, then checked against torchvision.ops.
The big idea: one label per image is not enough when a photo holds several objects. Object detection gives each object a bounding box, a label and a confidence score; segmentation labels every pixel. To build a detector we need a way to score a box (IoU), a way to remove duplicate boxes (non-maximum suppression), ready-made box shapes to correct (anchor boxes), a fast single-network design (YOLO), and a way to grade the whole detector (AP and mAP). For segmentation, U-Net turns an image into a same-size mask using a contracting path, an expanding path and skip connections.
Sources: Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", 2015 (arXiv 1506.02640, full text). Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", 2015 (full text). Olaf Ronneberger, Philipp Fischer and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", 2015 (arXiv 1505.04597). COCO detection metrics as listed in the hf-vision detection_metrics README.
Words you need first
- Computer vision: getting computers to understand images and video.
- Image classification: one label for the whole image ("bean rust"). Episodes 65-67.
- Object detection: for every object, a box + a label + a confidence score.
- Segmentation: a label for every pixel of the image.
- Pixel coordinates: x goes right, y goes down; (0, 0) is the top-left pixel.
- Confidence score: a number from 0 to 1 saying how sure the detector is that a box holds an object (of that class).
- Precision and recall (Episode 24): precision = correct predictions ÷ all predictions; recall = objects found ÷ all real objects.
- 1 × 1 convolution and ResNet body (Episodes 66-67).
The story: three leaves in one photo
The farmer from last episode now sends a photo with three leaves: two sick, one healthy. Our classifier gives one label, "bean rust", for the whole photo. Which leaf? Where? How many are sick? A single label cannot answer; we need positions.
| Task | Output | Answers | Example |
|---|---|---|---|
| Image classification | one label (+ probability) | what is in the image? | "bean rust, 94%" |
| Object detection | a box + label + score per object | what and where? | 2 boxes "bean rust", 1 box "healthy" |
| Semantic segmentation | a class for every pixel | what is each pixel? | every leaf pixel = 1, background = 0 |
| Instance segmentation | class + object id for every pixel | which object is each pixel? | leaf 1, leaf 2, leaf 3 |
Bounding boxes
A bounding box is the smallest upright rectangle around an object. We store it as (x1, y1, x2, y2): top-left corner and bottom-right corner. Other common formats are (x, y, width, height) (COCO) and (centre x, centre y, width, height) (YOLO); torchvision.ops.box_convert converts between them. Always check which one a dataset or library uses.
IoU: intersection over union
Real-life picture: two sheets of paper on a table. The intersection is where they overlap; the union is the whole area covered by either.
IoU = intersection ÷ union, where union = area(A) + area(B) − intersection. IoU is 1 for a perfect match and 0 for no overlap. The overlap's width is min(right edges) − max(left edges), and the same for the height.
The obvious attempt:
def area(b): # box = (x1, y1, x2, y2): top-left and bottom-right corners
return (b[2] - b[0]) * (b[3] - b[1])
def iou_wrong(a, b): # the obvious attempt
w = min(a[2], b[2]) - max(a[0], b[0])
h = min(a[3], b[3]) - max(a[1], b[1])
inter = w * h
return inter / (area(a) + area(b) - inter)
truth = (50, 50, 150, 150) # where the leaf really is
for name, guess in [("perfect", (50, 50, 150, 150)), ("shifted", (70, 60, 170, 160)),
("far away", (300, 300, 400, 400))]:
print(f"{name:9} IoU = {iou_wrong(truth, guess):.3f}")
Output:
perfect IoU = 1.000
shifted IoU = 0.562
far away IoU = -9.000
IoU = −9.000 is impossible. Why: when the boxes do not touch, the "overlap" width and height are both negative (−150 × −150 = +22,500), a fake overlap bigger than either box. Fix: clamp both at zero.
def area(b):
return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b): # intersection over union
w = max(0, min(a[2], b[2]) - max(a[0], b[0])) # no overlap -> 0, never negative
h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
inter = w * h
return inter / (area(a) + area(b) - inter)
truth = (50, 50, 150, 150)
for name, guess in [("perfect", (50, 50, 150, 150)), ("shifted", (70, 60, 170, 160)),
("too big", (25, 25, 175, 175)), ("half off", (100, 50, 200, 150)),
("far away", (300, 300, 400, 400))]:
print(f"{name:9} IoU = {iou(truth, guess):.3f}")
import torch, torchvision.ops as ops # the library version agrees
print("torchvision:", ops.box_iou(torch.tensor([truth]).float(),
torch.tensor([(70, 60, 170, 160)]).float()).item())
Output:
perfect IoU = 1.000
shifted IoU = 0.562
too big IoU = 0.444
half off IoU = 0.333
far away IoU = 0.000
torchvision: 0.5625
torchvision.ops.box_iou gives the same 0.5625. A detection usually counts as correct when its IoU with a true box is at least a threshold such as 0.5 (PASCAL VOC style).
Non-maximum suppression (NMS)
A detector makes many guesses, each with a confidence score, and neighbouring guesses often find the same object. Counting every box above 0.5 confidence counts each leaf three times. Non-maximum suppression fixes it: sort by confidence, keep the best box, delete every remaining box whose IoU with it is above a threshold (its copies), and repeat.
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
return w * h / (area(a) + area(b) - w * h)
# a detector's raw output for a photo with TWO sick leaves: (box, confidence score)
raw = [((48, 52, 152, 148), 0.92), ((55, 45, 160, 155), 0.85), ((40, 60, 145, 150), 0.78),
((260, 80, 380, 200), 0.88), ((270, 90, 385, 210), 0.70), ((250, 75, 370, 190), 0.65),
((150, 300, 180, 330), 0.12)]
print("boxes over 0.5 confidence:", sum(s > 0.5 for _, s in raw), "-> 6 leaves?!")
def nms(dets, iou_threshold=0.5):
dets = sorted(dets, key=lambda d: d[1], reverse=True) # most confident first
keep = []
while dets:
best = dets.pop(0); keep.append(best) # keep the best box...
dets = [d for d in dets if iou(best[0], d[0]) < iou_threshold] # ...drop its copies
return keep
kept = nms([d for d in raw if d[1] > 0.5])
for box, score in kept:
print("kept", box, "score", score)
import torch, torchvision.ops as ops
B = torch.tensor([b for b, s in raw if s > 0.5]).float(); S = torch.tensor([s for b, s in raw if s > 0.5])
print("torchvision.ops.nms keeps indices:", ops.nms(B, S, 0.5).tolist())
Output:
boxes over 0.5 confidence: 6 -> 6 leaves?!
kept (48, 52, 152, 148) score 0.92
kept (260, 80, 380, 200) score 0.88
torchvision.ops.nms keeps indices: [0, 3]
The NMS IoU threshold is a setting: Faster R-CNN's region proposals use 0.7 ("We fix the IoU threshold for NMS at 0.7"); 0.5 is a common choice for final detections. Too low merges neighbouring objects; too high keeps duplicates (see Practice 2).
Anchor boxes
Real-life example: a parking lot. Spaces are painted at fixed spots (grid positions) in several shapes (bikes, cars, buses: anchor boxes), and a car parks a little off the lines (the offset the network learns).
An anchor box is a fixed reference box of a set size and shape at a grid position. The network predicts, for each anchor, whether it holds an object and how to shift and stretch it. Faster R-CNN (Ren, He, Girshick, Sun, 2015): "By default we use 3 scales and 3 aspect ratios, yielding k=9 anchors at each sliding position" (areas 128², 256², 512²; ratios 1:1, 1:2, 2:1). Aspect ratio = height ÷ width. During training, anchors with a high IoU to a true box learn to fit it.
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
return w * h / (area(a) + area(b) - w * h)
def anchors_at(cx, cy, scales=(32, 64, 128), ratios=(0.5, 1.0, 2.0)):
out = []
for s in scales: # same area s * s ...
for r in ratios: # ... different shapes: height / width = r
w, h = s / r ** 0.5, s * r ** 0.5
out.append((cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2))
return out
stride = 32 # one anchor position every 32 pixels
grid = [(x * stride + 16, y * stride + 16) for y in range(7) for x in range(7)]
anchors = [a for cx, cy in grid for a in anchors_at(cx, cy)]
print("positions:", len(grid), "| anchors per position:", len(anchors_at(0, 0)),
"| total anchors:", len(anchors))
leaf = (100, 24, 188, 200) # a tall, thin leaf: 88 wide, 176 high
best = max(anchors, key=lambda a: iou(a, leaf))
w, h = best[2] - best[0], best[3] - best[1]
print(f"best anchor: {w:.0f} wide x {h:.0f} high, IoU {iou(best, leaf):.2f}")
print("anchors with IoU > 0.5:", sum(iou(a, leaf) > 0.5 for a in anchors))
mid = lambda b: ((b[0] + b[2]) / 2, (b[1] + b[3]) / 2)
(ax, ay), (lx, ly) = mid(best), mid(leaf)
print(f"what the network learns: move {lx - ax:+.0f}, {ly - ay:+.0f} px,"
f" width x{(leaf[2] - leaf[0]) / w:.2f}, height x{(leaf[3] - leaf[1]) / h:.2f}")
Output:
positions: 49 | anchors per position: 9 | total anchors: 441
best anchor: 91 wide x 181 high, IoU 0.95
anchors with IoU > 0.5: 4
what the network learns: move +0, +0 px, width x0.97, height x0.97
The tall anchor (91 × 181) already overlaps the tall leaf with IoU 0.95, so only a tiny correction is needed.
YOLO: You Only Look Once
Earlier detectors first proposed regions and then classified each one. YOLO (Redmon, Divvala, Girshick, Farhadi, 2015): "A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation." It divides the image into an S × S grid; each cell predicts B boxes (x, y, w, h and a confidence = Pr(Object) × IOU) and C class probabilities, encoded as an S × S × (B·5 + C) tensor. For PASCAL VOC: S = 7, B = 2, C = 20, a 7 × 7 × 30 tensor. The base model ran at 45 frames per second (Fast YOLO: 155) and reached 63.4% mAP on VOC 2007. Later YOLO versions added anchor boxes and many other improvements.
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(68)
S, B, C = 7, 2, 20 # YOLO (2015) on PASCAL VOC: 7 x 7 grid, 2 boxes, 20 classes
print("numbers per grid cell: B * 5 + C =", B * 5 + C)
body = models.resnet18(weights=None)
body = nn.Sequential(*list(body.children())[:-2]) # keep the conv layers, drop pooling + fc
head = nn.Conv2d(512, B * 5 + C, kernel_size=1) # a 1 x 1 conv: 30 numbers per cell
photo = torch.randn(1, 3, 224, 224)
features = body(photo)
out = head(features)
print("photo", tuple(photo.shape), "-> features", tuple(features.shape), "-> output", tuple(out.shape))
cell = out[0, :, 3, 4] # the cell in row 3, column 4
x, y, w, h, conf = cell[:5].sigmoid().tolist() # box 1: centre, size, confidence
print(f"cell (3, 4), box 1: x {x:.2f} y {y:.2f} w {w:.2f} h {h:.2f} confidence {conf:.2f}")
print("class scores per cell:", cell[10:].shape[0], "| boxes per photo:", S * S * B)
Output:
numbers per grid cell: B * 5 + C = 30
photo (1, 3, 224, 224) -> features (1, 512, 7, 7) -> output (1, 30, 7, 7)
cell (3, 4), box 1: x 0.71 y 0.42 w 0.18 h 0.74 confidence 0.50
class scores per cell: 20 | boxes per photo: 98
A ResNet-18 body on a 224 × 224 photo already gives a 7 × 7 grid of 512 features; a 1 × 1 convolution turns it into 30 numbers per cell. The network is untrained, so these numbers mean nothing yet. 98 boxes per photo is why NMS always follows.
Scoring a detector: AP and mAP
- A prediction is a true positive if its IoU with a true box is at least the threshold and that true box was not already matched; otherwise it is a false positive.
- Go through predictions from most to least confident, recording recall and precision after each: the precision-recall curve.
- Average precision (AP) = area under that curve (we make precision non-increasing first, the usual "interpolated" AP).
- mAP (mean average precision) = mean of AP over all classes.
- COCO AP averages AP over IoU thresholds 0.50, 0.55, ..., 0.95 ("AP at IoU=.50:.05:.95"); AP50 is "similar to mAP PASCAL VOC metric" and AP75 is the "strict metric".
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
return w * h / (area(a) + area(b) - w * h)
truth = [(10, 10, 60, 60), (100, 20, 150, 80), (200, 200, 260, 250), (300, 40, 350, 90)]
dets = [((12, 8, 61, 58), 0.95), ((105, 25, 152, 84), 0.90), ((400, 400, 450, 450), 0.80),
((196, 205, 255, 252), 0.70), ((12, 12, 58, 63), 0.60), ((320, 60, 370, 110), 0.40)]
def average_precision(dets, truth, thr):
dets = sorted(dets, key=lambda d: d[1], reverse=True) # most confident first
used, tp, points = set(), 0, []
for k, (box, score) in enumerate(dets, 1):
best = max(range(len(truth)), key=lambda i: iou(box, truth[i]))
if iou(box, truth[best]) >= thr and best not in used:
used.add(best); tp += 1 # a true positive
points.append((tp / len(truth), tp / k)) # (recall, precision)
ap, last_r = 0.0, 0.0
for i, (r, p) in enumerate(points): # area under the P-R curve
best_p = max(p2 for r2, p2 in points[i:]) # (precision made non-increasing)
ap += (r - last_r) * best_p; last_r = r
return ap, points
ap, points = average_precision(dets, truth, 0.5)
for r, p in points:
print(f"recall {r:.2f} precision {p:.2f}")
print(f"AP at IoU 0.5: {ap:.3f}")
for thr in (0.75, 0.9):
print(f"AP at IoU {thr}: {average_precision(dets, truth, thr)[0]:.3f}")
coco = [average_precision(dets, truth, t / 100)[0] for t in range(50, 100, 5)]
print(f"COCO-style AP (IoU 0.50 to 0.95, 10 steps): {sum(coco) / len(coco):.3f}")
Output:
recall 0.25 precision 1.00
recall 0.50 precision 1.00
recall 0.50 precision 0.67
recall 0.75 precision 0.75
recall 0.75 precision 0.60
recall 0.75 precision 0.50
AP at IoU 0.5: 0.688
AP at IoU 0.75: 0.688
AP at IoU 0.9: 0.000
COCO-style AP (IoU 0.50 to 0.95, 10 steps): 0.463
At IoU 0.9 none of our boxes is precise enough, so AP falls to 0; averaging over many thresholds rewards tight boxes.
Segmentation: semantic vs instance
A mask is an image-sized grid of labels, one per pixel. Semantic segmentation gives each pixel's class; instance segmentation also says which object it belongs to. Exact shapes let us measure, for example, how much of a leaf is rusty.
photo = ["................", # a tiny 8 x 16 "photo": # = leaf pixel
"..####......###.",
".######....#####",
".######....#####",
"..####......###.",
"................",
"......##........",
"......##........"]
semantic = [[1 if c == "#" else 0 for c in row] for row in photo] # class per pixel: leaf or not
instance = [[0] * 16 for _ in photo]; count = 0
for y in range(8):
for x in range(16):
if semantic[y][x] and not instance[y][x]:
count += 1; todo = [(y, x)] # a new leaf: flood-fill it
while todo:
j, i = todo.pop()
if 0 <= j < 8 and 0 <= i < 16 and semantic[j][i] and not instance[j][i]:
instance[j][i] = count
todo += [(j + 1, i), (j - 1, i), (j, i + 1), (j, i - 1)]
print("semantic (class per pixel) instance (which leaf)")
for s, n in zip(semantic, instance):
print("".join(map(str, s)), " ", "".join(str(v) if v else "." for v in n))
print("classes in the semantic mask:", sorted({v for r in semantic for v in r}),
"| leaves found:", count)
Output:
semantic (class per pixel) instance (which leaf)
0000000000000000 ................
0011110000001110 ..1111......222.
0111111000011111 .111111....22222
0111111000011111 .111111....22222
0011110000001110 ..1111......222.
0000000000000000 ................
0000001100000000 ......33........
0000001100000000 ......33........
classes in the semantic mask: [0, 1] | leaves found: 3
The flood fill starts at an unlabelled leaf pixel and spreads to all touching leaf pixels, giving each connected leaf its own number.
U-Net
U-Net (Ronneberger, Fischer, Brox, 2015, for biomedical images) has "a contracting path to capture context and a symmetric expanding path that enables precise localization". Going down, convolutions and pooling make maps smaller with more channels (what); going up, transposed convolutions (nn.ConvTranspose2d, a learned upsampling) bring them back to full size (where). Skip connections copy each down-level's maps across and concatenate them (torch.cat) onto the way up, restoring fine detail lost by pooling. The final 1 × 1 convolution gives a score per class per pixel; the paper segments a 512 × 512 image in "less than a second on a recent GPU".
import torch, torch.nn as nn
torch.manual_seed(68)
def block(cin, cout): # two 3 x 3 convs, like the U-Net paper
return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(),
nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU())
class TinyUNet(nn.Module):
def __init__(self, classes=3):
super().__init__()
self.d1, self.d2, self.mid = block(3, 16), block(16, 32), block(32, 64)
self.pool = nn.MaxPool2d(2)
self.up2, self.u2 = nn.ConvTranspose2d(64, 32, 2, stride=2), block(64, 32)
self.up1, self.u1 = nn.ConvTranspose2d(32, 16, 2, stride=2), block(32, 16)
self.out = nn.Conv2d(16, classes, 1) # a score for every class, at every pixel
def forward(self, x):
a = self.d1(x); b = self.d2(self.pool(a)); m = self.mid(self.pool(b))
print("down:", tuple(a.shape), tuple(b.shape), "bottom:", tuple(m.shape))
u = self.u2(torch.cat([self.up2(m), b], 1)) # up, then glue on the skip
u = self.u1(torch.cat([self.up1(u), a], 1))
print("up: ", tuple(u.shape))
return self.out(u)
net = TinyUNet()
scores = net(torch.randn(1, 3, 64, 64))
print("output:", tuple(scores.shape), "-> mask:", tuple(scores.argmax(1).shape))
print("parameters:", f"{sum(p.numel() for p in net.parameters()):,}")
Output:
down: (1, 16, 64, 64) (1, 32, 32, 32) bottom: (1, 64, 16, 16)
up: (1, 16, 64, 64)
output: (1, 3, 64, 64) -> mask: (1, 64, 64)
parameters: 117,075
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Overlap without clamping | far away IoU = -9.000 |
negative width × negative height | max(0, ...) for width and height |
| Counting raw detections | 6 boxes over 0.5 for 2 leaves | neighbouring guesses find the same object | non-maximum suppression |
| Mixing box formats | boxes in the wrong place, IoU near 0 | (x, y, w, h) read as (x1, y1, x2, y2) |
torchvision.ops.box_convert |
| Reporting one loose AP threshold | AP 0.688 at 0.5 but 0.000 at 0.9 | IoU 0.5 hides sloppy boxes | also report COCO AP over 0.50-0.95 |
| Flipping the photo but not the labels | boxes/masks on the wrong side | augmentation must move labels too | flip boxes and masks the same way |
| NMS across different classes | a "rust" box deletes an overlapping "healthy" box | suppression should compare same-class boxes | per-class NMS (torchvision.ops.batched_nms) |
Mini project: count the sick leaves
Combine the pieces: filter by confidence, run NMS, then count the boxes per class. With the raw detections from the NMS section, the 6 confident boxes become 2 kept boxes: 2 sick leaves. Add a class to each detection and run NMS per class (torchvision.ops.batched_nms(boxes, scores, classes, 0.5)) to count each disease separately.
Practice
1. Write iou() for (x, y, width, height) boxes.
Solution
def iou_xywh(a, b): # boxes as (x, y, width, height), (x, y) = top-left corner
a = (a[0], a[1], a[0] + a[2], a[1] + a[3]) # convert to (x1, y1, x2, y2)
b = (b[0], b[1], b[0] + b[2], b[1] + b[3])
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
return w * h / (area(a) + area(b) - w * h)
print(iou_xywh((50, 50, 100, 100), (70, 60, 100, 100))) # same boxes as the video: 0.5625
print(iou_xywh((0, 0, 10, 10), (5, 0, 10, 10))) # the quiz: 0.333...
import torch, torchvision.ops as ops
print(ops.box_convert(torch.tensor([[50., 50., 100., 100.]]), "xywh", "xyxy"))
Output:
0.5625
0.3333333333333333
tensor([[ 50., 50., 150., 150.]])
2. Run NMS with thresholds 0.3 and 0.8.
Solution
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
return w * h / (area(a) + area(b) - w * h)
raw = [((48, 52, 152, 148), 0.92), ((55, 45, 160, 155), 0.85), ((40, 60, 145, 150), 0.78),
((260, 80, 380, 200), 0.88), ((270, 90, 385, 210), 0.70), ((250, 75, 370, 190), 0.65)]
def nms(dets, iou_threshold):
dets = sorted(dets, key=lambda d: d[1], reverse=True); keep = []
while dets:
best = dets.pop(0); keep.append(best)
dets = [d for d in dets if iou(best[0], d[0]) < iou_threshold]
return keep
print("IoUs with the best box:", [round(iou(raw[0][0], b), 2) for b, _ in raw[1:3]])
for t in (0.3, 0.5, 0.8, 0.9):
print(f"threshold {t}: {len(nms(raw, t))} boxes kept")
Output:
IoUs with the best box: [0.76, 0.78]
threshold 0.3: 2 boxes kept
threshold 0.5: 2 boxes kept
threshold 0.8: 6 boxes kept
threshold 0.9: 6 boxes kept
The copies overlap the best box with IoU 0.76-0.78, so a threshold above that (0.8) keeps every duplicate.
3. Add a second class to the AP code and compute mAP.
Solution
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
return w * h / (area(a) + area(b) - w * h)
def average_precision(dets, truth, thr):
dets = sorted(dets, key=lambda d: d[1], reverse=True)
used, tp, points = set(), 0, []
for k, (box, score) in enumerate(dets, 1):
best = max(range(len(truth)), key=lambda i: iou(box, truth[i]))
if iou(box, truth[best]) >= thr and best not in used:
used.add(best); tp += 1
points.append((tp / len(truth), tp / k))
ap, last_r = 0.0, 0.0
for i, (r, p) in enumerate(points):
ap += (r - last_r) * max(p2 for _, p2 in points[i:]); last_r = r
return ap
data = { # class: (true boxes, detections)
"bean_rust": ([(10, 10, 60, 60), (100, 20, 150, 80), (200, 200, 260, 250), (300, 40, 350, 90)],
[((12, 8, 61, 58), 0.95), ((105, 25, 152, 84), 0.90), ((400, 400, 450, 450), 0.80),
((196, 205, 255, 252), 0.70), ((12, 12, 58, 63), 0.60), ((320, 60, 370, 110), 0.40)]),
"leaf_spot": ([(30, 120, 90, 170), (150, 150, 200, 210)],
[((32, 118, 88, 172), 0.85), ((10, 300, 40, 330), 0.75), ((152, 148, 205, 205), 0.50)])}
aps = {c: average_precision(d, t, 0.5) for c, (t, d) in data.items()}
for c, ap in aps.items(): print(f"AP@0.5 {c}: {ap:.3f}")
print(f"mAP@0.5 = mean over classes: {sum(aps.values()) / len(aps):.3f}")
Output:
AP@0.5 bean_rust: 0.688
AP@0.5 leaf_spot: 0.833
mAP@0.5 = mean over classes: 0.760
4. Add a third level to the tiny U-Net.
Solution
import torch, torch.nn as nn
def block(cin, cout):
return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(), nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU())
class UNet3(nn.Module): # three levels down, three up
def __init__(self, classes=3):
super().__init__()
self.d1, self.d2, self.d3, self.mid = block(3, 16), block(16, 32), block(32, 64), block(64, 128)
self.pool = nn.MaxPool2d(2)
self.up3, self.u3 = nn.ConvTranspose2d(128, 64, 2, stride=2), block(128, 64)
self.up2, self.u2 = nn.ConvTranspose2d(64, 32, 2, stride=2), block(64, 32)
self.up1, self.u1 = nn.ConvTranspose2d(32, 16, 2, stride=2), block(32, 16)
self.out = nn.Conv2d(16, classes, 1)
def forward(self, x):
a = self.d1(x); b = self.d2(self.pool(a)); c = self.d3(self.pool(b)); m = self.mid(self.pool(c))
u = self.u3(torch.cat([self.up3(m), c], 1))
u = self.u2(torch.cat([self.up2(u), b], 1))
u = self.u1(torch.cat([self.up1(u), a], 1))
return self.out(u)
net = UNet3()
print("bottom:", tuple(net.mid(net.pool(net.d3(net.pool(net.d2(net.pool(net.d1(torch.randn(1, 3, 64, 64)))))))).shape))
print("output:", tuple(net(torch.randn(1, 3, 64, 64)).shape))
print("parameters:", f"{sum(p.numel() for p in net.parameters()):,}")
Output:
bottom: (1, 128, 8, 8)
output: (1, 3, 64, 64)
parameters: 482,067
One more level makes the bottom 8 × 8 with 128 channels and grows the network from 117,075 to 482,067 parameters.
Quick quiz
truth = (0, 0, 10, 10), guess = (5, 0, 15, 10), overlap 50. IoU? (a) 0.25 (b) 0.333 (c) 0.5 (d) 1.0
Answer
(b) 0.333: union = 100 + 100 − 50 = 150, and 50 ÷ 150 = 0.333.
- Detection vs segmentation?
Answer
Detection gives a box per object; segmentation gives a label per pixel. - What does NMS remove?
Answer
Lower-scored boxes that overlap a kept box above the IoU threshold (duplicates). - What is YOLO's output size on PASCAL VOC?
Answer
7 × 7 × 30: S = 7, B = 2 boxes × 5 numbers + C = 20 classes. - How many anchors per position in Faster R-CNN?
Answer
9: 3 scales × 3 aspect ratios. - What do U-Net's skip connections carry?
Answer
Each down-level's feature maps, concatenated onto the up path, to restore fine detail.
Interview questions
- Explain IoU and where it is used. Intersection area ÷ union area of two boxes (or masks). Used to match predictions to ground truth, in NMS, in anchor assignment, and in mAP thresholds.
- How does non-maximum suppression work? Sort by score; repeatedly keep the top box and discard remaining boxes with IoU above a threshold with it; usually per class.
- What are anchor boxes and why use them? Predefined boxes of several sizes and aspect ratios at each grid position; the network predicts offsets and objectness relative to them, which makes learning box shapes easier.
- One-stage vs two-stage detectors? Two-stage (Faster R-CNN) proposes regions then classifies them; one-stage (YOLO) predicts boxes and classes in one pass, usually faster.
- What is mAP@0.5 vs COCO mAP? mAP@0.5 counts a match at IoU ≥ 0.5; COCO's main AP averages over IoU 0.50-0.95 in steps of 0.05, rewarding precise localisation.
- Semantic vs instance segmentation? Semantic labels pixel classes only; instance also separates individual objects of the same class.
- Why does U-Net need skip connections? Pooling loses spatial detail; copying encoder features to the decoder gives precise boundaries.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Computer vision | Getting computers to understand images: classify, detect, segment |
| Image classification | One label for the whole image: "bean rust" |
| Object detection | A box + label + confidence for every object in an image |
| Segmentation | A label for every pixel |
| Semantic segmentation | Each pixel's class: leaf (1) or background (0) |
| Instance segmentation | Each pixel's class and which object: leaf 1, leaf 2, leaf 3 |
| Mask | An image-sized grid of per-pixel labels |
| Bounding box | Smallest upright rectangle around an object, e.g. (50, 50, 150, 150) |
(x1, y1, x2, y2) / (x, y, w, h) |
Corner format / corner + size format of a box |
| Confidence score | How sure the detector is that a box holds an object (0 to 1) |
| Intersection | The overlapping area of two boxes |
| Union | The area covered by either box: area A + area B − intersection |
| IoU (intersection over union) | Intersection ÷ union: 1 = perfect, 0 = no overlap; shifted box = 0.5625 |
| Non-maximum suppression (NMS) | Keep the best box, drop boxes overlapping it above a threshold, repeat |
| Anchor box | A fixed reference box (size + shape) at a grid position that the network adjusts |
| Aspect ratio | Height ÷ width of a box: 1:2, 1:1, 2:1 |
| Offset | The small shift and stretch from an anchor to the real box |
| Faster R-CNN | Ren, He, Girshick, Sun (2015): two-stage detector with 9 anchors per position |
| YOLO | "You Only Look Once" (Redmon et al., 2015): one network, one pass, S × S grid, 45 fps |
| Grid cell (YOLO) | One of the S × S squares; predicts boxes for objects centred in it |
| PASCAL VOC | A detection benchmark with 20 object classes |
| COCO | A large detection benchmark whose main metric averages AP over IoU 0.50-0.95 |
| True / false positive (detection) | A prediction that matches an unmatched true box at the IoU threshold / one that does not |
| Precision-recall curve | Precision vs recall as we accept more and more predictions |
| Average precision (AP) | Area under the precision-recall curve for one class |
| mAP (mean average precision) | Mean of AP over classes (and, for COCO, over IoU thresholds) |
| Flood fill | Spreading a label to all touching pixels of the same kind |
| U-Net | Ronneberger, Fischer, Brox (2015): contracting path + expanding path + skip connections |
| Contracting / expanding path | The down half (smaller maps, more channels) / the up half (back to full size) |
| Transposed convolution | A learned upsampling layer, nn.ConvTranspose2d(..., 2, stride=2) doubles height and width |
| Skip connection (U-Net) | Copying a down-level's maps across and concatenating them on the way up |
Summary
- Classification: one label; detection: boxes + labels + scores; segmentation: a label per pixel (semantic or instance).
- IoU = intersection ÷ union; clamp the overlap at 0 or far-apart boxes give −9.000.
- NMS keeps the best box and removes its copies: 6 boxes → 2 leaves.
- Anchor boxes: 9 shapes per position (Faster R-CNN), the network learns small offsets.
- YOLO: one network, one look, a 7 × 7 × 30 output on PASCAL VOC, 45 fps.
- AP = area under the precision-recall curve (0.688 at IoU 0.5, 0.463 COCO-style); mAP = mean over classes.
- U-Net: down, up, skip connections: a 64 × 64 photo → a 64 × 64 mask.
Next episode: RNNs and sequences: sequence data, recurrent neural networks, the hidden state, unrolling through time, backpropagation through time and vanishing gradients.