Machine Learning Full Course · Episode 59. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU). The output shown is the real output; fixed random seeds make it repeatable, and timings vary from machine to machine.
The big idea: training a real network by hand means creating, listing, updating and zeroing every weight tensor yourself, and forgetting one is silent (our output bias stayed at exactly 0.0000 for 2,000 steps). PyTorch's torch.nn fixes the bookkeeping: an nn.Linear layer owns its weight and bias, a model is a class that inherits from nn.Module with a forward method (or a quick nn.Sequential), and model.parameters() finds every tensor (337 for our circle network). Loss functions take logits (raw scores): BCEWithLogitsLoss stays finite where sigmoid-then-log gives inf, CrossEntropyLoss needs int64 class indices, MSELoss is for numbers. An optimiser updates every parameter with step() and clears gradients with zero_grad(). The training loop runs those five lines for every batch; one pass over the data is an epoch. Forgetting zero_grad() dropped test accuracy to 78.0%, exactly what "always say outside" scores. model.eval() switches off dropout so one input always gives one answer.
Words you need first
- Tensor, requires_grad, backward(), .grad, torch.no_grad() (Episode 58).
- Neuron, layer, hidden layer, ReLU (Episode 56). ReLU: max(0, z).
- Sigmoid, log loss / binary cross-entropy (Episode 22); softmax, categorical cross-entropy (Episode 23).
- Accuracy (Episode 24): the share of predictions that are right.
- Train / test split (Episode 15): data the model learns from vs data kept aside to measure it.
- SGD, mini-batch gradient descent (Episode 18).
- Class: a Python blueprint for objects; inheritance: a class built on another class, getting its abilities.
The problem: too many tensors by hand
The task: 1,000 random points in a square from −2 to 2; label 1 if the point is inside a circle of radius 1, else 0. One straight line can't separate inside from outside, so we need hidden layers: 2 inputs → 16 → 16 → 1 output, which means six tensors.
import torch; torch.manual_seed(59) # is a point inside a circle of radius 1?
X = torch.rand(1000, 2) * 4 - 2 # 1,000 points in a 4 x 4 square
y = (X.pow(2).sum(1) < 1).float() # 1 = inside, 0 = outside
W1 = (torch.randn(2, 16) * 0.5).requires_grad_(); b1 = torch.zeros(16, requires_grad=True)
W2 = (torch.randn(16, 16) * 0.5).requires_grad_(); b2 = torch.zeros(16, requires_grad=True)
W3 = (torch.randn(16, 1) * 0.5).requires_grad_(); b3 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2, W3] # six tensors to keep track of... we listed five
for step in range(2001):
h = torch.relu(torch.relu(X @ W1 + b1) @ W2 + b2)
p = torch.sigmoid(h @ W3 + b3).squeeze(1)
loss = ((p - y) ** 2).mean()
loss.backward()
with torch.no_grad():
for t in params: t -= 0.5 * t.grad; t.grad.zero_()
if step % 500 == 0: print(f"step {step:4}: loss {loss.item():.4f} b3 = {b3.item():.4f}")
Output:
step 0: loss 0.1633 b3 = 0.0000
step 500: loss 0.0100 b3 = 0.0000
step 1000: loss 0.0059 b3 = 0.0000
step 1500: loss 0.0043 b3 = 0.0000
step 2000: loss 0.0034 b3 = 0.0000
The loss falls, so it looks fine, but b3 was never in the list and was never trained. No error, no warning. Why: we did the bookkeeping. What we want: layers that own their tensors, and a model that lists them itself.
nn.Linear and nn.Module
A tiffin carrier: each container holds its own food (a layer holds its own weight and bias); containers stack in order (a model); "what's for lunch?" opens every container (model.parameters()).
import torch, torch.nn as nn; torch.manual_seed(59)
layer = nn.Linear(2, 16) # 2 inputs -> 16 outputs: holds a weight AND a bias
print("weight:", layer.weight.shape, " bias:", layer.bias.shape, " grad:", layer.weight.requires_grad)
class CircleNet(nn.Module): # our own model: a class that inherits from nn.Module
def __init__(self):
super().__init__()
self.l1 = nn.Linear(2, 16); self.l2 = nn.Linear(16, 16); self.out = nn.Linear(16, 1)
def forward(self, x): # the forward pass: input -> output
x = torch.relu(self.l1(x))
x = torch.relu(self.l2(x))
return self.out(x)
model = CircleNet()
print(model)
for name, p in model.named_parameters(): print(f" {name:10} {str(list(p.shape)):9} {p.numel()}")
print("total parameters:", sum(p.numel() for p in model.parameters()))
print("model(x) on 3 points:", model(torch.rand(3, 2)).shape)
Output:
weight: torch.Size([16, 2]) bias: torch.Size([16]) grad: True
CircleNet(
(l1): Linear(in_features=2, out_features=16, bias=True)
(l2): Linear(in_features=16, out_features=16, bias=True)
(out): Linear(in_features=16, out_features=1, bias=True)
)
l1.weight [16, 2] 32
l1.bias [16] 16
l2.weight [16, 16] 256
l2.bias [16] 16
out.weight [1, 16] 16
out.bias [1] 1
total parameters: 337
model(x) on 3 points: torch.Size([3, 1])
Rules:
nn.Linear(in, out)computes y = x · Wᵀ + b; W has shape[out, in], b has shape[out]; both start random-ish and already require grad.nn.Module: the base class for every model and layer. In__init__, callsuper().__init__()first, then create layers.forward(self, x): describes the forward pass.- Layers stored as
self.somethingare registered:parameters()andnamed_parameters()find them. - Call
model(x), notmodel.forward(x): calling the model runsforwardplus PyTorch's extra machinery (hooks). - Forgetting
super().__init__()raisesAttributeError: cannot assign module before Module.__init__() callthe moment a layer is assigned. - Parameters: the learnable numbers of a model (weights and biases): 32 + 16 + 256 + 16 + 16 + 1 = 337.
nn.Sequential
import torch, torch.nn as nn; torch.manual_seed(59)
model = nn.Sequential( # the same network, as a stack of layers
nn.Linear(2, 16), nn.ReLU(),
nn.Linear(16, 16), nn.ReLU(),
nn.Linear(16, 1),
)
print(model)
print("parameters:", sum(p.numel() for p in model.parameters()))
try:
model(torch.rand(3, 3)) # 3 features in, but the first layer expects 2
except RuntimeError as e:
print("RuntimeError:", e)
Output:
Sequential(
(0): Linear(in_features=2, out_features=16, bias=True)
(1): ReLU()
(2): Linear(in_features=16, out_features=16, bias=True)
(3): ReLU()
(4): Linear(in_features=16, out_features=1, bias=True)
)
parameters: 337
RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (3x3 and 2x16)
nn.Sequential runs its layers in order; nn.ReLU() is the activation as a layer. Use a full nn.Module class when the forward pass is not a simple chain (branches, loops, several inputs). The shape error means: the input's last dimension (3) must equal the first layer's in_features (2).
Loss functions and logits
Logits are the raw scores a final Linear layer outputs (any number, −200 … 30) before sigmoid or softmax turns them into probabilities.
import torch, torch.nn as nn
logits = torch.tensor([2.0, -1.0, 30.0, -200.0]) # raw scores from the last layer: logits
y = torch.tensor([1.0, 0.0, 0.0, 1.0]) # the true labels
p = torch.sigmoid(logits) # the obvious way: sigmoid, then the log-loss formula
print("probabilities:", [round(v, 4) for v in p.tolist()])
manual = -(y * torch.log(p) + (1 - y) * torch.log(1 - p)).mean()
print("manual log loss:", manual.item())
print("BCEWithLogitsLoss:", round(nn.BCEWithLogitsLoss()(logits, y).item(), 4))
pred, true = torch.tensor([50.0, 72.0]), torch.tensor([52.0, 70.0]) # prices in lakh
print("MSELoss:", nn.MSELoss()(pred, true).item())
Output:
probabilities: [0.8808, 0.2689, 1.0, 0.0]
manual log loss: inf
BCEWithLogitsLoss: 57.61
MSELoss: 4.0
sigmoid(−200) is so tiny that float32 stores it as exactly 0, and log(0) = −∞, so the loss is infinite and training breaks. BCEWithLogitsLoss combines sigmoid and log in one numerically stable formula, so it takes logits. MSELoss: ((50 − 52)² + (72 − 70)²) / 2 = 4.
Multiclass:
import torch, torch.nn as nn
logits = torch.tensor([[2.0, 0.5, -1.0], # 2 photos, 3 classes: cat, dog, bird
[0.1, 0.2, 3.0]])
target = torch.tensor([0, 2]) # class indices: photo 1 is a cat, photo 2 a bird
ce = nn.CrossEntropyLoss()
print("softmax:", [[round(v, 3) for v in row] for row in torch.softmax(logits, dim=1).tolist()])
print("CrossEntropyLoss:", round(ce(logits, target).item(), 4))
print("predicted classes:", logits.argmax(dim=1).tolist())
try:
ce(logits, torch.tensor([0.0, 2.0])) # labels as floats instead of class indices
except RuntimeError as e:
print("RuntimeError:", e)
Output:
softmax: [[0.786, 0.175, 0.039], [0.049, 0.054, 0.896]]
CrossEntropyLoss: 0.1755
predicted classes: [0, 2]
RuntimeError: expected target dtype to be Long or Byte, but got Float
CrossEntropyLoss applies softmax inside and averages −log(probability of the true class): (−log 0.786 − log 0.896) / 2 ≈ 0.1755. argmax returns the position of the biggest score: the predicted class.
| Task | Last layer | Loss | Labels |
|---|---|---|---|
| Regression (a number) | Linear(…, 1) |
nn.MSELoss() |
float |
| Binary (yes / no) | Linear(…, 1), one logit |
nn.BCEWithLogitsLoss() |
float 0. / 1., same shape as output |
| Multiclass (K classes) | Linear(…, K), K logits |
nn.CrossEntropyLoss() |
int64 class index 0 … K−1 |
Never put sigmoid or softmax before BCEWithLogitsLoss or CrossEntropyLoss; apply torch.sigmoid / torch.softmax only when showing probabilities.
Optimisers
A driving lesson: drive a lap (forward + loss), the instructor notes every mistake (backward()), the student adjusts and the notes are cleared (step(), zero_grad()).
- Optimiser: an object that updates parameters.
torch.optim.SGD(model.parameters(), lr=0.5)is plain stochastic gradient descent. opt.step(): w ← w − lr × w.grad for every parameter it was given.opt.zero_grad(): empties every parameter's.grad.
The five lines of every training step:
pred = model(xb) # 1. forward
loss = loss_fn(pred, yb) # 2. loss
opt.zero_grad() # 3. empty old gradients
loss.backward() # 4. fill every .grad
opt.step() # 5. update every parameter
The training loop: epochs, batches, iterations
- Batch: a small group of examples used for one update (here 100).
- Iteration: one update, i.e. one batch through the five lines.
- Epoch: one full pass over all training data. 1,000 examples ÷ batch size 100 = 10 iterations per epoch.
- Shuffle the order every epoch so batches differ.
- Training loop:
model.train()+ the five lines per batch. Evaluation loop:model.eval()+torch.no_grad(), only measuring.
import torch, torch.nn as nn; torch.manual_seed(59)
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:] # 1,000 to train, 200 to test
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5) # EVERY parameter
batch_size = 100 # 1,000 / 100 = 10 iterations per epoch
for epoch in range(1, 61):
model.train() # training mode
perm = torch.randperm(1000) # shuffle every epoch
for i in range(0, 1000, batch_size):
idx = perm[i:i + batch_size]
loss = loss_fn(model(Xtr[idx]), ytr[idx]) # 1. forward 2. loss
opt.zero_grad() # 3. empty old gradients
loss.backward() # 4. backward
opt.step() # 5. update every parameter
model.eval() # evaluation mode
with torch.no_grad(): acc = ((model(Xte) > 0).float() == yte).float().mean() # evaluation loop
if epoch in (1, 5, 20, 60): print(f"epoch {epoch:2}: loss {loss.item():.4f} test accuracy {acc:.1%}")
Output:
epoch 1: loss 0.4104 test accuracy 78.0%
epoch 5: loss 0.2768 test accuracy 93.0%
epoch 20: loss 0.0803 test accuracy 98.0%
epoch 60: loss 0.0205 test accuracy 99.0%
A logit above 0 means a probability above 0.5, so model(Xte) > 0 predicts "inside". 99% on 200 points the model never trained on.
Forgetting zero_grad()
import torch, torch.nn as nn; torch.manual_seed(59)
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]
for forget in (False, True):
torch.manual_seed(1)
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 1))
opt = torch.optim.SGD(model.parameters(), lr=0.5); loss_fn = nn.BCEWithLogitsLoss()
for epoch in range(60):
perm = torch.randperm(1000)
for i in range(0, 1000, 100):
idx = perm[i:i + 100]; loss = loss_fn(model(Xtr[idx]), ytr[idx])
if not forget: opt.zero_grad() # the line we "forget"
loss.backward(); opt.step()
with torch.no_grad(): acc = ((model(Xte) > 0).float() == yte).float().mean()
print(f"forgot zero_grad = {forget}: test accuracy {acc:.1%}, final loss {loss.item():.4f}")
print(f"always saying 'outside': {(yte == 0).float().mean().item():.1%}")
Output:
forgot zero_grad = False: test accuracy 98.5%, final loss 0.0135
forgot zero_grad = True: test accuracy 78.0%, final loss 2.2883
always saying 'outside': 78.0%
Without zero_grad() the gradients from every old batch pile up, the steps grow huge, and the model ends up no better than always answering "outside".
model.train() vs model.eval()
Dropout (studied in episode 63) randomly zeroes values during training so the network can't rely on one neuron, like a cricket team practising with random players resting. In a real match everyone plays.
import torch, torch.nn as nn; torch.manual_seed(59)
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Dropout(0.5), nn.Linear(16, 1))
point = torch.tensor([[0.3, -0.4]])
with torch.no_grad():
print("forgot eval():", [round(model(point).item(), 3) for _ in range(3)]) # 1 point, 3 calls
model.eval()
print("after eval(): ", [round(model(point).item(), 3) for _ in range(3)])
drop = nn.Dropout(p=0.5); x = torch.ones(8)
print("dropout, train mode:", drop(x).tolist()) # zeroed with chance 0.5, the rest doubled
print("dropout, eval mode: ", drop.eval()(x).tolist())
print("model.training =", model.training, "| after model.train():", model.train().training)
Output:
forgot eval(): [-0.229, -0.373, -0.345]
after eval(): [-0.118, -0.118, -0.118]
dropout, train mode: [2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0]
dropout, eval mode: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
model.training = False | after model.train(): True
model.train()(the default): training mode, dropout on. Each value is zeroed with probability p and the survivors are multiplied by 1/(1 − p) = 2 to keep the average; here, by chance, only one of eight was zeroed.model.eval(): evaluation mode, dropout off, batch normalisation (episode 63) uses its saved statistics.eval()does not stop gradient tracking; still usetorch.no_grad()when only measuring.model.trainingtells you the current mode.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Listing tensors by hand | b3 = 0.0000 after 2,000 steps |
one tensor forgotten | model.parameters() |
Forgot super().__init__() |
AttributeError: cannot assign module before Module.__init__() call |
Module not set up | call it first |
| Wrong number of input features | RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (3x3 and 2x16) |
in_features = 2 |
match the input's last dimension |
| sigmoid + log by hand | manual log loss: inf |
log(0) | BCEWithLogitsLoss on logits |
| Float class labels | RuntimeError: expected target dtype to be Long or Byte, but got Float |
needs class indices | target.long() |
softmax before CrossEntropyLoss |
trains worse, no error | softmax applied twice | pass logits |
No opt.zero_grad() |
test accuracy 78.0%, final loss 2.2883 |
gradients pile up | zero every step |
No model.eval() |
[-0.229, -0.373, -0.345] for one point |
dropout still on | model.eval() before predicting |
Mini project: the circle network as an nn.Module class
import torch, torch.nn as nn; torch.manual_seed(59)
class CircleNet(nn.Module):
def __init__(self):
super().__init__()
self.l1 = nn.Linear(2, 16); self.l2 = nn.Linear(16, 16); self.out = nn.Linear(16, 1)
def forward(self, x):
return self.out(torch.relu(self.l2(torch.relu(self.l1(x)))))
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]
model = CircleNet(); loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(60):
model.train(); perm = torch.randperm(1000)
for i in range(0, 1000, 100):
idx = perm[i:i + 100]; loss = loss_fn(model(Xtr[idx]), ytr[idx])
opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad(): print(f"CircleNet test accuracy: {((model(Xte) > 0).float() == yte).float().mean():.1%}")
Output:
CircleNet test accuracy: 99.0%
Practice
1. Change the circle to a ring (0.5 < distance < 1).
Solution
import torch, torch.nn as nn; torch.manual_seed(59)
X = torch.rand(1200, 2) * 4 - 2; r = X.pow(2).sum(1).sqrt()
y = ((r > 0.5) & (r < 1)).float().unsqueeze(1) # a ring: 0.5 < distance < 1
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 201):
model.train(); perm = torch.randperm(1000)
for i in range(0, 1000, 50):
idx = perm[i:i + 50]; loss = loss_fn(model(Xtr[idx]), ytr[idx])
opt.zero_grad(); loss.backward(); opt.step()
if epoch in (1, 50, 200):
model.eval()
with torch.no_grad(): acc = ((model(Xte) > 0).float() == yte).float().mean()
print(f"epoch {epoch:3}: test accuracy {acc:.1%}")
print(f"always 'not in ring': {(yte == 0).float().mean():.1%}")
Output:
epoch 1: test accuracy 84.0%
epoch 50: test accuracy 97.5%
epoch 200: test accuracy 96.5%
always 'not in ring': 84.0%
A ring is harder: a wider network (32 neurons) and more epochs. Epoch 1 equals the lazy baseline; by epoch 50 it has learned the ring.
2. Try batch sizes 10, 100 and 1,000 for 30 epochs.
Solution
import torch, torch.nn as nn, time; torch.manual_seed(59)
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]
for bs in (10, 100, 1000):
torch.manual_seed(1)
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 1))
opt = torch.optim.SGD(model.parameters(), lr=0.5); loss_fn = nn.BCEWithLogitsLoss(); t = time.time()
for epoch in range(30):
perm = torch.randperm(1000)
for i in range(0, 1000, bs):
idx = perm[i:i + bs]; loss = loss_fn(model(Xtr[idx]), ytr[idx])
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad(): acc = ((model(Xte) > 0).float() == yte).float().mean()
print(f"batch {bs:4}: {1000 // bs:3} updates/epoch, 30 epochs -> accuracy {acc:.1%} ({time.time() - t:.1f} s)")
Output (times vary):
batch 10: 100 updates/epoch, 30 epochs -> accuracy 96.5% (0.8 s)
batch 100: 10 updates/epoch, 30 epochs -> accuracy 99.0% (0.1 s)
batch 1000: 1 updates/epoch, 30 epochs -> accuracy 78.0% (0.0 s)
With batch 1,000 there is only one update per epoch, 30 in total: not enough to learn. Tiny batches make many noisy updates and take longer per epoch.
3. Turn it into 3 classes (inner disc, ring, outside) with CrossEntropyLoss.
Solution
import torch, torch.nn as nn; torch.manual_seed(59)
X = torch.rand(1200, 2) * 4 - 2; d = X.pow(2).sum(1).sqrt()
y = torch.where(d < 0.8, 0, torch.where(d < 1.6, 1, 2)) # 3 classes: inner disc, ring, outside
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 3)) # 3 logits
loss_fn, opt = nn.CrossEntropyLoss(), torch.optim.SGD(model.parameters(), lr=0.3)
for epoch in range(1, 101):
model.train(); perm = torch.randperm(1000)
for i in range(0, 1000, 50):
idx = perm[i:i + 50]; loss = loss_fn(model(Xtr[idx]), ytr[idx])
opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad(): pred = model(Xte).argmax(dim=1)
print("labels dtype:", ytr.dtype, "| test accuracy:", f"{(pred == yte).float().mean():.1%}")
Output:
labels dtype: torch.int64 | test accuracy: 96.5%
Quick quiz
10,000 training examples, batch_size = 50, 3 epochs. How many times does opt.step() run?
Answer
600. 10,000 ÷ 50 = 200 iterations per epoch, × 3 epochs.
More questions:
- What shape is the weight of
nn.Linear(784, 128)?Answer
[128, 784](out, in); the bias is[128]. - How many parameters does
nn.Linear(10, 5)have?Answer
10 × 5 + 5 = 55. - Which loss for predicting a house price? For spam / not spam? For 10 digits?
Answer
MSELoss,BCEWithLogitsLoss,CrossEntropyLoss. - What are logits?
Answer
The raw, unbounded scores from the last layer before sigmoid or softmax. - Does
model.eval()turn off gradient tracking?Answer
No. It changes layer behaviour (dropout, batch norm);torch.no_grad()stops tracking.
Interview questions
- Walk through a PyTorch training step. Forward pass, compute the loss,
optimizer.zero_grad(),loss.backward(),optimizer.step(). - Why
BCEWithLogitsLossinstead of sigmoid +BCELoss? It fuses sigmoid and log using the log-sum-exp trick, avoiding log(0) and overflow for large logits. - What target does
CrossEntropyLossexpect? Raw logits of shape[N, C]and int64 class indices of shape[N](or class probabilities of the same shape as the input). - Why call
model(x)instead ofmodel.forward(x)?__call__runs registered hooks and other machinery aroundforward. - Difference between epoch, batch and iteration? Batch: subset used for one update; iteration: one update; epoch: one pass over the dataset.
- What do
train()andeval()change? Behaviour of layers such as dropout and batch normalisation; nothing about gradients.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
torch.nn |
PyTorch's neural-network package: layers, losses, models |
| nn.Module | Base class of every model and layer; define __init__ and forward |
| forward method | Describes how an input flows through the layers to the output |
| nn.Linear | A layer computing x · Wᵀ + b that owns its weight [out, in] and bias [out] |
| nn.Sequential | A model that runs a list of layers in order |
| nn.ReLU | The ReLU activation, max(0, z), as a layer |
| Parameters | A model's learnable numbers (weights and biases): 337 in our circle net |
model.parameters() |
Yields every parameter tensor of every registered layer |
named_parameters() |
The same, with names: l1.weight, out.bias |
| Registered layer | A layer stored as self.something, so the model can find it |
| Logits | Raw scores from the last layer before sigmoid/softmax: 2.0, −200.0 |
| Loss function | A function measuring how wrong predictions are |
| MSELoss | Mean squared error, for numbers: errors 2 and 2 → 4.0 |
| BCEWithLogitsLoss | Binary cross-entropy taking logits, numerically stable |
| CrossEntropyLoss | Softmax + negative log-likelihood for several classes; int64 labels |
| Numerical stability | Avoiding inf/NaN from tiny or huge numbers, e.g. log(0) |
| argmax | Position of the biggest value: predicted class |
| Optimiser | Object that updates parameters from their gradients: torch.optim.SGD |
optimizer.step() |
Apply the update rule to every parameter |
optimizer.zero_grad() |
Empty every parameter's .grad |
| Batch | A small group of examples for one update: 100 points |
| Iteration | One update = one batch through the five lines |
| Epoch | One full pass over all training data |
| Training loop | Repeating forward, loss, zero_grad, backward, step over batches and epochs |
| Evaluation loop | Measuring on held-out data with model.eval() and torch.no_grad() |
| Dropout | Randomly zeroing values during training (episode 63) |
model.train() / model.eval() |
Switch between training mode (dropout on) and evaluation mode (dropout off) |
| Accuracy | Share of correct predictions: 99.0% |
| Baseline | A trivial model to beat: always "outside" = 78.0% |
Summary
- Hand-managed tensors fail silently;
nn.Linearlayers own their parameters andnn.Module/nn.Sequentialmodels find them all withparameters(). - Losses take logits:
MSELoss(numbers),BCEWithLogitsLoss(yes/no),CrossEntropyLoss(classes, int64 labels). - An optimiser updates everything with
step();zero_grad()every step or training collapses (78.0%). - Batches → iterations → epochs; the training loop learns, the evaluation loop measures in
eval()mode underno_grad().
Next episode: PyTorch 3: data loading: Dataset, DataLoader, batching, shuffling, collate functions and workers.