Machine Learning Full Course · Episode 60. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU, 2 cores). The output shown is the real output; timings vary from machine to machine and run to run.
The big idea: last episode every batch was a slice of one tensor already in memory. Real data breaks that: 50,000 photos would need 30.1 GB as one float32 tensor, hand-cut batches leave a short last batch (40 instead of 64), and sentences of different lengths can't even become one tensor. PyTorch splits the job in two. A Dataset answers two questions: how many examples (__len__) and "give me example number i" (__getitem__), loading each example only when asked. A DataLoader turns a Dataset into batches: batch_size, shuffle (a new order every epoch), drop_last, a collate_fn that joins examples into a batch (we pad uneven sentences), and num_workers helper processes that load in parallel (2.61 s → 1.36 s per epoch here). Transforms fix or randomly vary (augment) each example inside __getitem__. Training and validation each get their own loader.
Source: torchvision, PyTorch's computer-vision library, provides ready-made transforms in torchvision.transforms.v2, including Compose, Normalize and RandomHorizontalFlip (torchvision docs).
Words you need first
- Tensor, shape, dtype, padding a shape (Episode 58).
- Training loop, batch, iteration, epoch,
model.train()/model.eval()(Episode 59). - Train / validation / test split (Episode 15): validation data is kept aside to measure the model while you build it.
- Normalisation (Episode 16): rescaling numbers to a standard range, e.g. −1 … 1.
- CSV file: a text file of rows with values separated by commas.
- Class, method: a Python blueprint and the functions that belong to it.
- Process: a running program; several processes can work at the same time.
The problem: loading everything and batching by hand
import torch
n, h, w = 50_000, 224, 224 # 50,000 colour photos, 224 x 224 pixels
bytes_needed = n * 3 * h * w * 4 # 3 colours, float32 = 4 bytes each
print(f"all photos as one tensor: {bytes_needed / 1e9:.1f} GB of memory")
X = torch.arange(1000.0).unsqueeze(1) # 1,000 examples that DO fit
batches = [X[i:i + 64] for i in range(0, 1000, 64)] # hand-made batches of 64
print("batches:", len(batches), "| last 3 sizes:", [len(b) for b in batches[-3:]])
sentences = [[4, 9, 2], [7, 1, 8, 3, 5]] # two sentences as word ids: lengths 3 and 5
try:
print(torch.tensor(sentences))
except ValueError as e:
print("ValueError:", e)
Output:
all photos as one tensor: 30.1 GB of memory
batches: 16 | last 3 sizes: [64, 64, 40]
ValueError: expected sequence of length 3 at dim 1 (got 5)
- 50,000 × 3 × 224 × 224 × 4 bytes = 30.1 GB: more than most laptops have.
- 1,000 is not a multiple of 64, so the last batch has 40 examples.
- A tensor must be a perfect rectangle; sentences of 3 and 5 word ids are not. (Word id: a number standing for a word in a vocabulary.)
Why: we loaded all the data before training and wrote the batching ourselves. What we want: load one example only when needed, and a helper that groups, shuffles and batches them.
The Dataset class
A library: the catalogue says how many books there are (__len__); the librarian fetches book No. 25 only when asked (__getitem__); a helper with a trolley brings 4 random books per trip (the DataLoader).
Dataset(from torch.utils.data import Dataset): a class that answers two questions.__len__(self): how many examples are there?len(ds)calls it.__getitem__(self, i): return example number i, usually as tensors(x, y).ds[i]calls it.- Dunder methods: names with double underscores on both sides; Python calls them for built-in syntax.
__init__keeps only cheap things (file names, rows); the heavy loading (opening a photo) happens in__getitem__. This is lazy loading.
import torch, csv
from torch.utils.data import Dataset
with open("marks.csv", "w") as f: # a small file: study hours, sleep hours, passed?
f.write("hours,sleep,passed\n" + "".join(f"{h},{s},{int(h + s / 2 > 7)}\n"
for h in range(10) for s in range(4, 10)))
class MarksDataset(Dataset):
def __init__(self, path):
with open(path) as f: self.rows = list(csv.DictReader(f)) # keep the rows, not tensors
def __len__(self): return len(self.rows) # how many examples?
def __getitem__(self, i): # give me example number i, as tensors
r = self.rows[i]
x = torch.tensor([float(r["hours"]), float(r["sleep"])])
return x, torch.tensor(float(r["passed"]))
ds = MarksDataset("marks.csv")
print("len(ds):", len(ds))
print("ds[0]: ", ds[0], " ds[59]:", ds[59])
x, y = ds[25]; print("ds[25] -> x =", x.tolist(), " y =", y.item())
Output:
len(ds): 60
ds[0]: (tensor([0., 4.]), tensor(0.)) ds[59]: (tensor([9., 9.]), tensor(1.))
ds[25] -> x = [4.0, 5.0] y = 0.0
Forgetting __len__:
import torch
from torch.utils.data import Dataset, DataLoader
class NoLen(Dataset): # forgot __len__
def __getitem__(self, i): return torch.tensor([float(i)])
try:
for xb in DataLoader(NoLen(), batch_size=4, shuffle=True): pass
except Exception as e:
print(type(e).__name__ + ":", e)
Output:
TypeError: object of type 'NoLen' has no len()
To shuffle, the loader must know how many example numbers exist.
The DataLoader
import torch
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(60)
X = torch.arange(10).unsqueeze(1); y = X.squeeze(1) * 2 # 10 tiny examples: x -> 2x
ds = TensorDataset(X, y) # a ready-made Dataset for tensors
loader = DataLoader(ds, batch_size=4) # no shuffle
print("batches per epoch:", len(loader))
for xb, yb in loader: print(" x:", xb.squeeze(1).tolist(), " shape:", list(xb.shape))
for epoch in (1, 2):
order = [xb.squeeze(1).tolist() for xb, _ in DataLoader(ds, batch_size=4, shuffle=True)]
print(f"shuffle, epoch {epoch}:", order)
print("drop_last=True:", [len(xb) for xb, _ in DataLoader(ds, batch_size=4, drop_last=True)])
Output:
batches per epoch: 3
x: [0, 1, 2, 3] shape: [4, 1]
x: [4, 5, 6, 7] shape: [4, 1]
x: [8, 9] shape: [2, 1]
shuffle, epoch 1: [[0, 2, 5, 3], [6, 4, 7, 9], [8, 1]]
shuffle, epoch 2: [[3, 0, 7, 1], [5, 4, 2, 8], [6, 9]]
drop_last=True: [4, 4]
| Option | Meaning | Typical value |
|---|---|---|
batch_size |
examples per batch | 32, 64, 128 |
shuffle |
a new random order every epoch | True for training, False for validation |
drop_last |
skip the short final batch | True when every batch must be full |
collate_fn |
function that joins a list of examples into one batch | default stacks tensors |
num_workers |
helper processes loading in parallel | 0 (main process) … number of CPU cores |
len(loader) is the number of batches = iterations per epoch: ⌈10 / 4⌉ = 3, or ⌊10 / 4⌋ = 2 with drop_last. TensorDataset(X, y) is a ready-made Dataset whose example i is (X[i], y[i]).
collate_fn: building a batch
The loader picks example numbers, calls __getitem__ for each, then gives the list to a collate function, which joins them into one batch. The default stacks tensors, so every example must have the same shape.
import torch
from torch.utils.data import DataLoader
sentences = [torch.tensor([4, 9, 2]), torch.tensor([7, 1, 8, 3, 5]), torch.tensor([6, 2])] # word ids
try:
next(iter(DataLoader(sentences, batch_size=3))) # default: stack them into one tensor
except RuntimeError as e:
print("RuntimeError:", e)
def pad_collate(batch): # our own way to build a batch
longest = max(len(s) for s in batch)
padded = torch.zeros(len(batch), longest, dtype=torch.long) # 0 = padding
for i, s in enumerate(batch): padded[i, :len(s)] = s
lengths = torch.tensor([len(s) for s in batch])
return padded, lengths
xb, lengths = next(iter(DataLoader(sentences, batch_size=3, collate_fn=pad_collate)))
print(xb); print("lengths:", lengths.tolist())
Output:
RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1
tensor([[4, 9, 2, 0, 0],
[7, 1, 8, 3, 5],
[6, 2, 0, 0, 0]])
lengths: [3, 5, 2]
Padding: filling short examples with a placeholder (0) up to the longest length. The lengths tell the model which positions are real. (A plain Python list works as a Dataset too: it has len() and [i]. next(iter(loader)) takes just the first batch.)
Transforms and data augmentation
- Transform: a function applied to each example inside
__getitem__, as it is loaded. - Preprocessing: fixed changes such as scaling pixels 0 … 255 to 0 … 1 and normalising to −1 … 1.
- Data augmentation: random small changes (noise, flips, crops) so the model sees a slightly different copy every epoch: more variety without more data.
import torch; from torch.utils.data import Dataset
class Photos(Dataset): # pretend photos: 4 pixels each, values 0..255
def __init__(self, transform=None):
g = torch.Generator().manual_seed(60) # the same 6 "photos" every time
self.data = torch.randint(0, 256, (6, 4), generator=g).float(); self.transform = transform
def __len__(self): return len(self.data)
def __getitem__(self, i): # the transform runs on every example
return self.transform(self.data[i]) if self.transform else self.data[i]
def preprocess(x): # fixed: scale to 0..1, then normalise to -1..1
return (x / 255 - 0.5) / 0.5
def augment(x): # random: preprocess + a little noise
return preprocess(x) + 0.05 * torch.randn_like(x)
r = lambda t: [round(v, 3) for v in t.tolist()]
print("raw: ", Photos()[0].tolist())
print("transformed:", r(Photos(preprocess)[0]))
aug = Photos(augment)
print("augmented 1:", r(aug[0]))
print("augmented 2:", r(aug[0]), "<- same photo, new noise")
Output (the noise differs each run):
raw: [205.0, 65.0, 70.0, 138.0]
transformed: [0.608, -0.49, -0.451, 0.082]
augmented 1: [0.651, -0.319, -0.462, 0.111]
augmented 2: [0.515, -0.446, -0.54, 0.103] <- same photo, new noise
205 / 255 = 0.804; (0.804 − 0.5) / 0.5 = 0.608. For real images, torchvision.transforms.v2 offers Compose, Normalize, RandomHorizontalFlip and many more. Augment only the training data; validation data gets the fixed preprocessing only.
num_workers: loading in parallel
With num_workers=0 the main program loads every example itself and the model waits. With num_workers=N, N helper processes (workers) prepare batches in parallel.
import torch, time
from torch.utils.data import Dataset, DataLoader
class SlowDisk(Dataset): # each example takes 10 ms to read, like a file
def __len__(self): return 256
def __getitem__(self, i):
time.sleep(0.01)
return torch.tensor([float(i)])
for workers in (0, 2):
t = time.time()
for xb in DataLoader(SlowDisk(), batch_size=32, num_workers=workers): pass
print(f"num_workers={workers}: one epoch in {time.time() - t:.2f} s")
Output (this machine has 2 CPU cores):
num_workers=0: one epoch in 2.61 s
num_workers=2: one epoch in 1.36 s
Start near the number of CPU cores; PyTorch prints a UserWarning ("Our suggested max number of worker in current system is 2...") if you ask for more than it suggests. Workers cost memory and start-up time, so 0 is fine for small in-memory data. On Windows and macOS, put the training code under if __name__ == "__main__": when using workers.
Train and validation loaders
import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader, random_split
torch.manual_seed(60)
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1) # episode 59's circle
train_ds, val_ds = random_split(TensorDataset(X, y), [1000, 200]) # 1,000 train, 200 validation
train_loader = DataLoader(train_ds, batch_size=100, shuffle=True) # shuffle the training data
val_loader = DataLoader(val_ds, batch_size=100) # never shuffle validation
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 41):
model.train()
for xb, yb in train_loader: # the loader hands us batches
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
model.eval(); correct = 0
with torch.no_grad():
for xb, yb in val_loader: correct += ((model(xb) > 0).float() == yb).sum().item()
if epoch in (1, 10, 40): print(f"epoch {epoch:2}: validation accuracy {correct / len(val_ds):.1%}")
print("train batches:", len(train_loader), "| validation batches:", len(val_loader))
Output:
epoch 1: validation accuracy 80.5%
epoch 10: validation accuracy 96.5%
epoch 40: validation accuracy 97.0%
train batches: 10 | validation batches: 2
random_split(ds, [1000, 200])returns two Subset datasets that pick random, non-overlapping example numbers.- Training loader:
shuffle=True. Validation loader: no shuffling needed, only measuring. - Never train on validation data, or its score stops meaning anything (leakage, episode 15).
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Loading every photo into one tensor | all photos as one tensor: 30.1 GB of memory |
memory | load in __getitem__ |
| Hand-made batches | last 3 sizes: [64, 64, 40] |
leftovers | DataLoader (+ drop_last) |
Uneven lists to torch.tensor |
ValueError: expected sequence of length 3 at dim 1 (got 5) |
not a rectangle | pad |
No __len__ |
TypeError: object of type 'NoLen' has no len() |
shuffle needs the size | add __len__ |
| Uneven examples with default collate | RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1 |
default stacks | collate_fn with padding |
Heavy loading in __init__ |
slow start, huge memory | not lazy | keep file names; open in __getitem__ |
| Too many workers | UserWarning: This DataLoader will create 4 worker processes in total... |
more workers than cores | about the number of cores |
| Augmenting / training on validation data | a validation score that means nothing | leakage | separate loaders, fixed preprocessing only |
Mini project: train episode 59's model on MarksDataset
import torch, torch.nn as nn, csv
from torch.utils.data import Dataset, DataLoader, random_split
torch.manual_seed(60)
with open("marks.csv", "w") as f:
f.write("hours,sleep,passed\n" + "".join(f"{h},{s},{int(h + s / 2 > 7)}\n" for h in range(10) for s in range(4, 10)))
class MarksDataset(Dataset):
def __init__(self, path):
with open(path) as f: self.rows = list(csv.DictReader(f))
def __len__(self): return len(self.rows)
def __getitem__(self, i):
r = self.rows[i]
x = torch.tensor([float(r["hours"]) / 9, float(r["sleep"]) / 9]) # scale to about 0..1
return x, torch.tensor([float(r["passed"])])
train_ds, val_ds = random_split(MarksDataset("marks.csv"), [48, 12])
train_loader, val_loader = DataLoader(train_ds, batch_size=8, shuffle=True), DataLoader(val_ds, batch_size=12)
model = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 301):
model.train()
for xb, yb in train_loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
if epoch in (1, 100, 300):
model.eval()
with torch.no_grad():
xb, yb = next(iter(val_loader)); acc = ((model(xb) > 0).float() == yb).float().mean()
print(f"epoch {epoch:3}: validation accuracy {acc:.1%}")
Output:
epoch 1: validation accuracy 75.0%
epoch 100: validation accuracy 100.0%
epoch 300: validation accuracy 100.0%
The label is returned as [1.] (shape [1]) so a batch has shape [8, 1], matching the model's output.
Practice
1. Write a Dataset that reads one .txt file per example.
Solution
import torch, os
from torch.utils.data import Dataset, DataLoader
os.makedirs("reviews", exist_ok=True) # one small text file per example
for i, (text, label) in enumerate([("great chai", 1), ("cold and bland", 0), ("loved it", 1), ("too sweet", 0)]):
with open(f"reviews/{i}_{label}.txt", "w") as f: f.write(text)
class ReviewFiles(Dataset):
def __init__(self, folder):
self.paths = sorted(os.path.join(folder, n) for n in os.listdir(folder)) # only the file names
def __len__(self): return len(self.paths)
def __getitem__(self, i):
with open(self.paths[i]) as f: text = f.read() # open the file only now
label = int(self.paths[i].rsplit("_", 1)[1][0])
return text, torch.tensor(label)
ds = ReviewFiles("reviews")
print(len(ds), ds[1])
for texts, labels in DataLoader(ds, batch_size=2): print(texts, labels.tolist())
Output:
4 ('cold and bland', tensor(0))
('great chai', 'cold and bland') [1, 0]
('loved it', 'too sweet') [1, 0]
The default collate keeps strings as a tuple and stacks the label tensors.
2. Add a random flip transform (x.flip(0) with probability 0.5).
Solution
import torch
from torch.utils.data import Dataset
class Photos(Dataset):
def __init__(self, transform=None):
g = torch.Generator().manual_seed(60)
self.data = torch.randint(0, 256, (6, 4), generator=g).float(); self.transform = transform
def __len__(self): return len(self.data)
def __getitem__(self, i):
x = self.data[i]
return self.transform(x) if self.transform else x
def random_flip(x, p=0.5): # flip the pixel order with probability p
return x.flip(0) if torch.rand(()) < p else x
torch.manual_seed(0)
ds = Photos(random_flip)
print("original:", Photos()[0].tolist())
for k in range(4): print(f"call {k + 1}: ", ds[0].tolist())
Output:
original: [205.0, 65.0, 70.0, 138.0]
call 1: [138.0, 70.0, 65.0, 205.0]
call 2: [205.0, 65.0, 70.0, 138.0]
call 3: [138.0, 70.0, 65.0, 205.0]
call 4: [138.0, 70.0, 65.0, 205.0]
3. Time num_workers = 0, 1, 2 on your machine.
Solution
Use the SlowDisk program above with for workers in (0, 1, 2). Output on the 2-core recording machine:
num_workers=0: one epoch in 2.61 s
num_workers=1: one epoch in 2.69 s
num_workers=2: one epoch in 1.35 s
One worker just moves the loading to one helper process (no faster, plus a little start-up cost); two workers load two batches at once and roughly halve the time.
Quick quiz
len(ds) = 1,050, DataLoader(ds, batch_size=100, drop_last=True). What is len(loader)?
Answer
10. Ten full batches of 100; the leftover 50 are dropped. Without drop_last it would be 11.
More questions:
- Which two methods must a map-style Dataset define?
Answer
__len__and__getitem__. - Where should a photo be opened:
__init__or__getitem__?Answer
__getitem__, one example at a time;__init__keeps only the file names. - What does
collate_fnreceive?Answer
A list of examples (whatever__getitem__returned) for one batch. - Should the validation loader shuffle?
Answer
Not needed: it only measures; the order does not change the score. - What is data augmentation?
Answer
Random, label-preserving changes to training examples (flips, noise, crops) so the model sees more variety.
Interview questions
- Dataset vs DataLoader? A Dataset gives single examples by index and knows its length; a DataLoader samples indices, batches with a collate function, shuffles and loads in parallel.
- Why lazy loading? Data larger than memory; only the current batch is in memory, and loading overlaps with training when workers are used.
- How do you batch variable-length sequences? A custom
collate_fnthat pads to the longest in the batch and returns lengths (or an attention mask). - What does
num_workersdo, and trade-offs? Spawns worker processes that prepare batches in parallel; faster for I/O- or transform-heavy data but costs memory and start-up time. - What is
drop_lastfor? Dropping the final incomplete batch, useful when layers or losses assume a fixed batch size.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Dataset | A class giving example number i and the number of examples: MarksDataset |
__len__ |
Method answering "how many examples?": len(ds) → 60 |
__getitem__ |
Method returning example i: ds[25] → (x, y) |
| Dunder method | A method named with double underscores that Python calls for built-in syntax |
| Lazy loading | Loading each example only when it is requested |
| TensorDataset | Ready-made Dataset over existing tensors: TensorDataset(X, y) |
| DataLoader | Turns a Dataset into batches: shuffles, collates, uses workers |
| batch_size | Number of examples per batch: 4, 100 |
| Shuffling | A new random order of examples every epoch (shuffle=True) |
| drop_last | Skip the final short batch: [4, 4, 2] → [4, 4] |
Collate function (collate_fn) |
Joins a list of examples into one batch; the default stacks tensors |
| Padding | Filling short sequences with a placeholder (0) to a common length |
| Word id | A number standing for a word in a vocabulary |
| Transform | A function applied to each example as it is loaded: scale, normalise |
| Compose | Chaining several transforms into one |
| Data augmentation | Random small changes to training examples: noise, flips |
| num_workers | Number of helper processes loading batches in parallel |
| Worker | A helper process that runs __getitem__ and collate for the loader |
| random_split | Splits a Dataset into random non-overlapping parts: [1000, 200] |
| Subset | A Dataset that is a selection of another Dataset's examples |
| Training loader | Shuffled loader the model learns from |
| Validation loader | Loader used only to measure the model during development |
| torchvision | PyTorch's computer-vision library, with datasets and transforms |
Summary
- Don't load everything and slice by hand: memory runs out, leftovers and uneven examples break things.
- A Dataset answers
__len__and__getitem__, loading lazily. - A DataLoader gives batches:
batch_size,shuffle,drop_last, acollate_fn(padding for uneven data) andnum_workers. - Transforms preprocess and augment each example on the way in.
- Train with a shuffled training loader; measure with a separate validation loader.
Next episode: Optimisers: momentum, RMSProp, Adam and AdamW, compared on loss curves.