Machine Learning Full Course · Episode 77. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output. Data: MNIST, 60,000 training and 10,000 test images of handwritten digits (28 × 28 pixels), from the Hugging Face copy ylecun/mnist, saved as /data/mnist.pt by tools/get_mnist.py in the course repository. Images are printed as text: each 2 × 2 block of pixels becomes one character from " .:-=+*#%@" (dark → bright).
The big idea: a discriminative model maps an input to a label; a generative model learns what the data itself looks like, so it can make new examples. Random pixels give static and averaging gives one blur, so we need a model. An autoencoder squeezes an image through a bottleneck into a short code (a point in latent space) and rebuilds it; a denoising autoencoder learns to remove noise. Random codes decode badly because the latent space has holes. A VAE gives each image a small area (mean μ and spread σ) and a KL loss packs the areas around N(0, 1), so random codes decode to digits; the reparameterisation trick (z = μ + σ·ε) lets gradients flow through the random draw. A GAN trains a generator against a discriminator and makes sharp images, but can suffer mode collapse. A diffusion model adds noise step by step (forward noising) and learns to remove it (reverse denoising), starting new samples from pure noise.
Sources: Diederik P. Kingma and Max Welling, "Auto-Encoding Variational Bayes" (q(z|x) is a recognition model, a "probabilistic encoder"; reparameterisation z = μ + σ ⊙ ε with ε ~ N(0, I); for Gaussians the KL term has a closed form, −KL = ½ Σ (1 + log σ² − μ² − σ²); the KL term acts as a regulariser). Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio, "Generative Adversarial Nets", 2014 (two models trained simultaneously, G and D; a minimax two-player game; counterfeiters trying to produce fake currency vs police trying to detect it; G "must not be trained too much without updating D, in order to avoid 'the Helvetica scenario' in which G collapses too many values of z to the same value of x"). Jonathan Ho, Ajay Jain and Pieter Abbeel, "Denoising Diffusion Probabilistic Models", 2020 (the forward process adds Gaussian noise by a variance schedule β₁…β_T; x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε; T = 1000 with β rising linearly from 10⁻⁴ to 0.02; the simplified loss predicts the noise ε; sampling starts from x_T ~ N(0, I); FID 3.17 on CIFAR-10).
Words you need first
- Probability distribution (Episode 10): how likely each possible value is. N(0, 1), the standard normal: a bell curve with mean 0 and standard deviation 1.
- KL divergence (Episode 12): a measure of how different one distribution is from another (0 = identical).
- MLP, ReLU, sigmoid, tanh (Episodes 57-59): layers and activation functions. Sigmoid squeezes to 0..1, tanh to −1..1.
- Mean squared error and binary cross-entropy (Episodes 18, 22): losses.
- Gradient and backpropagation (Episodes 7, 58): how training finds which way to change each weight.
- MNIST (Episode 65).
The story: draw new handwritten digits
import torch
torch.manual_seed(77)
d = torch.load("/data/mnist.pt") # 60,000 handwritten digits, 28 x 28
X, Y = d["train_x"].float() / 255, d["train_y"]
def show(imgs, title): # print 28 x 28 images as text, 14 x 14
small = torch.nn.functional.avg_pool2d(imgs[:, None], 2)[:, 0]
print(title)
for r in range(14):
print(" ".join("".join(" .:-=+*#%@"[min(9, int(v * 10))] for v in img[r]) for img in small))
show(X[:4], "real digits")
show(torch.rand(4, 28, 28), "attempt 1: random pixels")
show(X[Y == 7].mean(0).expand(4, 28, 28), "attempt 2: the average 7, four times")
real digits
:.=- -%* : --
+#@@@+*: -@@%= = * :@*
*%@-* =@#@=% # :# ##
:# :@# . @: .% =+ =@:
#=. @- @- -* :@- .@+
#%. +# @- -%==*##=@ %%
+@ *= +# :--. :% =@=
.*@% *- .*+ -* @#
+@@+ *%+#@+ -* =@=
:*@@+ :%@*. -# *@
-=-. - :=
attempt 1: random pixels
+**+==*+*#=--* --++:-#*==+:=+ *+=*+%++-+=.++ *.=*=**=:+*#-=
=*++===-**:*-- ++=+-#==-+%=++ +==##===+--==# *#+-=*.=*-==+-
*#**=++#+=*#=: #=*==+=:.-.-=# +-##+====+-:+- *:**:*+-=*=:==
--*-*-*+=+=++* -==%==%*=#:++= :+:+=+*==+=*-= +=#=.==*=.:+-=
+-=---++**:*+- =++=++-=+*+*++ +=-%*+%=-*-#** ==%+=#-=*+--*+
*++%-==#=+#**: *=-===-**-+=+= -#**+*+=+=*=-= #==:=**--.---*
#+==++-+-=+*-- =:==*+*=+-*==* %=+%#-+-=+=:+= =**-*=+.==-=++
=--*=:+==+*+*+ =++**+%#+-++*= ++=:-=-=%+-#+* ===+-+:+=++=+=
:==-==++=-#-+% +*=**++*+-*++. **--+===*+*--= -=+*=%+%=+-#++
*+=#=++**#=++= -=+=%--+==+=#* +=:==++=:+.+-+ :-=+=**=:#=+=*
::#+#+:+#:#::* -=*#=+##+=#:== -+-:=+#+=+=**= =.==%#*+=+=:%*
=#*+=*:-+*#==+ ++==+#=--=@=+: +--+=*:=+:*-+= =*=+-+-++#+++=
=#==-=+-*=+=++ +=:==-++:+**+- -#-::=#*:=#++= +*==+*.-=-++=#
+++*-+*%+:++-+ ++=-+++:=+=:+= +*=+++%#++#*=: ++-==.-=++++*=
attempt 2: the average 7, four times
.::::: .::::: .::::: .:::::
.:+*###*- .:+*###*- .:+*###*- .:+*###*-
.:==--+*- .:==--+*- .:==--+*- .:==--+*-
.:. .++. .:. .++. .:. .++. .:. .++.
-*=. -*=. -*=. -*=.
.+*: .+*: .+*: .+*:
-*=. -*=. -*=. -*=.
.=+: .=+: .=+: .=+:
.-=-. .-=-. .-=-. .-=-.
.--:. .--:. .--:. .--:.
Why both attempts failed: almost every possible 28 × 28 image is noise; real digits are a tiny, structured corner of all images. The average of all 7s is blurry and always identical, with no variety. We need a model of what digits look like that can still give something different each time.
Real-life example: the food critic and the chef
| In a kitchen | In machine learning |
|---|---|
| A critic tastes and says "biryani" or "pulao": only needs the difference | Discriminative model: x → label (spam or not, 7 or 3), learns p(y given x) |
| A chef cooks a NEW biryani: must know what biryani is really like | Generative model: learns what the data looks like, p(x), and makes new x |
Autoencoders
Real-life example: the sketch artist on the phone. You see a face and must describe it in only 16 words; the artist draws it back from those words. If the drawing is right, the 16 words kept what matters.
- Encoder: a network that squeezes the input into a short code z (784 → 16 numbers).
- Decoder: a network that rebuilds the input from the code (16 → 784).
- Bottleneck: the narrow middle; it forces the network to keep only the important information.
- Latent space: the space of all possible codes ("latent" = hidden). Each image becomes a point in it.
- Reconstruction loss: how different the output is from the input; no labels needed.
import torch, torch.nn as nn, time
from art import ascii_row
torch.manual_seed(77)
d = torch.load("/data/mnist.pt")
X = d["train_x"].float().reshape(-1, 784) / 255; Xt = d["test_x"].float().reshape(-1, 784) / 255
encoder = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 16)) # 784 -> 16 numbers
decoder = nn.Sequential(nn.Linear(16, 256), nn.ReLU(), nn.Linear(256, 784), nn.Sigmoid())
opt = torch.optim.Adam([*encoder.parameters(), *decoder.parameters()], 1e-3)
t0 = time.time()
for epoch in range(1, 6):
for i in torch.randperm(60000).split(128):
loss = ((decoder(encoder(X[i])) - X[i]) ** 2).mean() # copy the input back out
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad(): tl = ((decoder(encoder(Xt)) - Xt) ** 2).mean()
print(f"epoch {epoch}: test reconstruction error {tl:.4f}")
print(f"trained in {time.time() - t0:.0f} s; 784 pixels squeezed into 16 numbers")
with torch.no_grad():
print("code for the first test digit:", encoder(Xt[:1])[0, :6].numpy().round(1), "...")
print(ascii_row(torch.cat([Xt[:3], decoder(encoder(Xt[:3]))]), ["input"] * 3 + ["rebuilt"] * 3))
z = torch.randn(4, 16) * encoder(Xt).std(0) + encoder(Xt).mean(0) # random codes
print(ascii_row(decoder(z), ["random code"] * 4))
torch.save({"enc": encoder.state_dict(), "dec": decoder.state_dict()}, "ae.pt")
(ascii_row from art.py prints images side by side:)
import torch
CH = " .:-=+*#%@"
def ascii_row(imgs, labels=None):
"""imgs: (n, 28, 28) in 0..1 -> one text block, digits side by side (14 x 14 each)"""
small = torch.nn.functional.avg_pool2d(imgs.reshape(-1, 1, 28, 28).float(), 2)[:, 0]
lines = []
if labels: lines.append(" ".join(f"{l:^14}" for l in labels))
for r in range(14):
lines.append(" ".join("".join(CH[min(9, int(v * 10))] for v in img[r]) for img in small))
return "\n".join(lines)
epoch 1: test reconstruction error 0.0212
epoch 2: test reconstruction error 0.0165
epoch 3: test reconstruction error 0.0149
epoch 4: test reconstruction error 0.0139
epoch 5: test reconstruction error 0.0133
trained in 42 s; 784 pixels squeezed into 16 numbers
code for the first test digit: [-3.4 -4.3 -0.5 -6.7 2. -0.6] ...
input input input rebuilt rebuilt rebuilt
:=- .:-:
+@%@* *. :#@%#- +.
:- .@: +* # ..:. .-. -+ .+
**%%%%%* .@+ :+ .=*#*#%#+ =+ -+
..++ *% *= . ..#+ =#. *=
.@. =@: % .#. .%: %.
*= @+ :# *+ *# -#
.% +@. *- -%. +#. +=
#= @+ :: %: @- %# .=-: %:
=# @@@@%@@@#. .@ =% *@%%%@@*. :%
.@. .::==: =+ .@- .===-. =+
*@ *@
-: --
random code random code random code random code
.
=%@* -. =+:-:=. .=+=.
#+.*= *# .. +@= .#= .%@+. .:.
+% .#: .%= := :@* #= -@- .::.
#+ ## .+: =. =@: :@+ :- -#+:.
+-+@: .. -- =@. =%@#. . :+=..
:*+#%: +: -@: =#-**. -*-.:=.
.%%*=%= .+ .+. . .. :+*:.%%.
:#+=+*. .-: .:. -- :-. .%+
.+*#%: .: . -*=--*@= . =*.
.=*+: :=: .-===+=. .::=*:
.
The rebuilt 7, 2 and 1 match the inputs. But random codes (in the same range as real ones) decode to broken shapes: the encoder placed real digits at scattered points, with holes in between that the decoder never learned to draw.
Denoising autoencoders
Give the network a spoiled input (image + random noise) but train it to output the clean image.
import torch, torch.nn as nn
from art import ascii_row
torch.manual_seed(77)
d = torch.load("/data/mnist.pt")
X = d["train_x"].float().reshape(-1, 784) / 255; Xt = d["test_x"].float().reshape(-1, 784) / 255
net = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 32), nn.ReLU(),
nn.Linear(32, 256), nn.ReLU(), nn.Linear(256, 784), nn.Sigmoid())
opt = torch.optim.Adam(net.parameters(), 1e-3)
noisy = lambda x: (x + 0.5 * torch.randn_like(x)).clamp(0, 1) # spoil the input
for epoch in range(1, 6):
for i in torch.randperm(60000).split(128):
loss = ((net(noisy(X[i])) - X[i]) ** 2).mean() # target = the CLEAN image
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
n = noisy(Xt[:3])
print(f"test error: noisy input {((n - Xt[:3]) ** 2).mean():.4f}"
f" -> cleaned {((net(n) - Xt[:3]) ** 2).mean():.4f}")
print(ascii_row(torch.cat([n, net(n)]), ["noisy input"] * 3 + ["cleaned"] * 3))
test error: noisy input 0.1188 -> cleaned 0.0175
noisy input noisy input noisy input cleaned cleaned cleaned
+:.- .-=.::+. :. :..: -:.:= ..: :-:..:-.
-- - .:.:-.: ::=- =:--.:.:. ::. :- .+.. .:.
.: .::::+- :-+ +:: +%%*+.:::. =.: ..=*:- =: =#%%*. -+:
- .-*: + .-: -.. + :#% - . .:. -.:.*--..- ..::: :::-+ :-
.::-=#*%#@+::- : +.: *=.. .- ..=::+.---: .-=*##*%: *+ ::
= ::: ---*... : ... +% .. : :: . =:::.=: .::..-%: *%. +:
:.-:--.*+ . . =: +@:: -..: =-.. .-#.:== - =% .#+ .#.
- . :- #*: =. :. -#+-..- . --= .=:#.:-.-. %+ +*..:. -*
:. - ..-%. :- -..+#:. -=: : ...:::--.=-... -%. =%- :: #=
=:= ..#+ = . : ::%*.: .* +: -: ..=#- .-:-- %= *%: .==. -%.
- .:.==: ...: +:= %%+@%##%% -...=-%. . : +% +@#*#%*. **.
. =:@: :. :. ..:: -#-=*: :. -. .=+.-:.- : :%- -+*+: =-
. -*%-+. -.: :::. + - . : --. .::::- **
.= *:....::= .- ::.+: .:..: =..:. :.:..+ . ::
Error on unseen test digits: 0.1188 (noisy) → 0.0175 (cleaned). Removing noise is exactly the skill a diffusion model repeats a thousand times.
Variational autoencoders (VAE)
Real-life example: an area, not an exact address. An autoencoder gives each image an exact address with empty plots in between. A VAE says "somewhere around Sector 5": every spot in the area is a real house. And all areas are packed around the city centre, so any random spot you pick has a house.
- The encoder outputs, for each code number, a mean μ and a log-variance log σ² (variance = spread²).
- Draw z near μ, decode, and compare with the input: the reconstruction loss (binary cross-entropy here).
- Add the KL divergence between N(μ, σ²) and N(0, 1):
KL = -0.5 * sum(1 + logvar - mu**2 - exp(logvar)). It pulls all areas toward the centre, filling the holes. - To generate: draw z ~ N(0, 1) and decode. No input image needed.
The reparameterisation trick
Training needs gradients from the loss back to μ and σ, but "pick a random number" has no derivative. Write the draw as arithmetic instead: eps = torch.randn_like(mu); z = mu + sigma * eps. Same distribution, but now z depends on μ and σ through + and ×, and ε is just a constant input. (Like a recipe: base salt + extra pinch × a random amount from a fixed jar; you can still tune the base and the pinch.)
import torch
from torch.distributions import Normal
mu = torch.tensor([0.5, -1.0], requires_grad=True) # the encoder's mean for 2 numbers
logvar = torch.tensor([0.0, 0.0], requires_grad=True) # and its log-variance
torch.manual_seed(77)
eps = torch.randn(2) # the randomness, on its own
z = mu + torch.exp(0.5 * logvar) * eps # reparameterisation trick
(z ** 2).sum().backward()
print("z =", z.detach().numpy().round(3))
print("gradient reaches mu:", mu.grad.numpy().round(3), " logvar:", logvar.grad.numpy().round(3))
z = Normal(mu, torch.exp(0.5 * logvar)).sample() # obvious way: just draw a random z
print("sampled z =", z.numpy().round(3), " connected to mu?", z.requires_grad)
(z ** 2).sum().backward()
z = [ 0.143 -0.399]
gradient reaches mu: [ 0.286 -0.799] logvar: [-0.051 -0.24 ]
sampled z = [-0.197 -1.524] connected to mu? False
Traceback (most recent call last):
...
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
Normal(...).sample() draws without the trick, so z is cut off from μ. PyTorch's .rsample() ("reparameterised sample") uses the trick for you.
import torch, torch.nn as nn, time
from art import ascii_row
torch.manual_seed(77)
d = torch.load("/data/mnist.pt"); X = d["train_x"].float().reshape(-1, 784) / 255
class VAE(nn.Module):
def __init__(s, k=16):
super().__init__()
s.enc = nn.Sequential(nn.Linear(784, 256), nn.ReLU())
s.mu, s.logvar = nn.Linear(256, k), nn.Linear(256, k) # a mean AND a spread per number
s.dec = nn.Sequential(nn.Linear(k, 256), nn.ReLU(), nn.Linear(256, 784), nn.Sigmoid())
def forward(s, x):
h = s.enc(x); mu, logvar = s.mu(h), s.logvar(h)
z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu) # reparameterisation trick
return s.dec(z), mu, logvar
vae = VAE(); opt = torch.optim.Adam(vae.parameters(), 1e-3); t0 = time.time()
for epoch in range(1, 11):
for i in torch.randperm(60000).split(128):
out, mu, logvar = vae(X[i])
rec = nn.functional.binary_cross_entropy(out, X[i], reduction="sum") / len(i)
kl = -0.5 * (1 + logvar - mu ** 2 - logvar.exp()).sum() / len(i) # pull codes toward N(0, 1)
loss = rec + kl; opt.zero_grad(); loss.backward(); opt.step()
if epoch % 2 == 0: print(f"epoch {epoch:2}: reconstruction {rec:.1f} KL {kl:.1f}")
print(f"trained in {time.time() - t0:.0f} s")
with torch.no_grad():
print(ascii_row(vae.dec(torch.randn(5, 16)), ["new: z ~ N(0,1)"] * 5))
torch.save(vae.state_dict(), "vae.pt")
epoch 2: reconstruction 102.6 KL 20.8
epoch 4: reconstruction 91.6 KL 21.4
epoch 6: reconstruction 90.7 KL 22.6
epoch 8: reconstruction 82.5 KL 22.7
epoch 10: reconstruction 88.2 KL 23.1
trained in 147 s
new: z ~ N(0,1) new: z ~ N(0,1) new: z ~ N(0,1) new: z ~ N(0,1) new: z ~ N(0,1)
--. .. :*%*. :.
:%+ . =%#: .#@@@+ .: .*@@
.#- *%%%- .*#*@# .#+ =#+%*
=: :*-+@+ .#%%@= .** .-. . :
- -- *. =#- *%@* :+:..:. .*. :@=
+*%@+=. *+.... %@* :#%#**: .@- .-=**
-#*:.-. .#@%-. =@*. ..:: @%*: -
. +@@%=:::. =%+. .=. .-: .
.= *@#:..:: +@%=+#+ ++ .=:
.: :: @@@%=-. +. +#.
. : :.
Brand-new digits from random z, but blurry: a VAE learns to be right on average, and averages are soft.
Generative adversarial networks (GAN)
Real-life example: fake notes and the bank cashier. A forger prints fake notes; a cashier checks every note; each learns from the other until the fakes are hard to spot. (The GAN paper itself uses counterfeiters and police.)
- Generator G: random noise z (64 numbers) → a fake image.
- Discriminator D: an image → the probability that it is real.
- D is trained to say 1 for real and 0 for fake; G is trained to make D say 1 on its fakes. This is a minimax game: one player's gain is the other's loss.
fake.detach()in D's step: D's update must not change G.
import torch, torch.nn as nn, time
from art import ascii_row
torch.manual_seed(77)
d = torch.load("/data/mnist.pt")
X = d["train_x"].float().reshape(-1, 784) / 255 * 2 - 1 # pixels in -1..1
G = nn.Sequential(nn.Linear(64, 256), nn.LeakyReLU(0.2), nn.Linear(256, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 784), nn.Tanh()) # generator: noise -> image
D = nn.Sequential(nn.Linear(784, 512), nn.LeakyReLU(0.2), nn.Linear(512, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 1)) # discriminator: image -> real?
optG = torch.optim.Adam(G.parameters(), 2e-4, betas=(0.5, 0.999))
optD = torch.optim.Adam(D.parameters(), 2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss(); t0 = time.time(); step = 0
for epoch in range(1, 9):
for i in torch.randperm(60000).split(128):
real, fake = X[i], G(torch.randn(len(i), 64))
lossD = bce(D(real), torch.ones(len(i), 1)) + bce(D(fake.detach()), torch.zeros(len(i), 1))
optD.zero_grad(); lossD.backward(); optD.step() # police: spot the fakes
lossG = bce(D(fake), torch.ones(len(i), 1)) # forger: make D say "real"
optG.zero_grad(); lossG.backward(); optG.step()
with torch.no_grad(): p = torch.sigmoid(D(G(torch.randn(1000, 64)))).mean()
if epoch % 2 == 0:
print(f"epoch {epoch}: D loss {lossD:.2f} G loss {lossG:.2f}"
f" D thinks fakes are real: {p:.0%}")
print(f"trained in {time.time() - t0:.0f} s")
with torch.no_grad():
print(ascii_row((G(torch.randn(5, 64)) + 1) / 2, ["GAN sample"] * 5))
epoch 2: D loss 0.50 G loss 3.88 D thinks fakes are real: 8%
epoch 4: D loss 0.47 G loss 2.58 D thinks fakes are real: 16%
epoch 6: D loss 0.50 G loss 2.99 D thinks fakes are real: 12%
epoch 8: D loss 0.32 G loss 2.64 D thinks fakes are real: 15%
trained in 160 s
GAN sample GAN sample GAN sample GAN sample GAN sample
: : :
. . .
:#: #* . .- = :#*
=# .++. . ++ +: :%#:.
%* +@- -@= +@ : **:=
-#: . : :@+ -:: . == : : %+ ::: -%- .: :
-= *@+ @* : -#: -
:#: : -#: ::%:- : ..+@: .=#:+ .
: .@@: : :@% : ..@@@ : .@@. . : .-+.
.@* . :@@: ## . .%@* : -==
: .- :.=#%: * : :#: : :. +==
.: %#. .. :*@#.
:
.:: ::: : .:
Sharper strokes than the VAE (with a few stray dots). D still catches most fakes during training (it rates about 15% of them as real); the two keep pushing each other.
Mode collapse
- Mode: one kind of thing in the data (each digit; each cluster).
- Mode collapse: G finds a few outputs that fool D and produces only those.
- The GAN paper: G must not be trained too much without updating D, to avoid "the Helvetica scenario" in which G collapses many values of z to the same output.
Test: eight clusters on a circle (ring.py), a balanced GAN vs one whose G trains 10 times per D step.
import torch, math
def ring(n): # 8 clusters in a circle: 8 "kinds" of data
k = torch.randint(0, 8, (n,)); a = k * 2 * math.pi / 8
return torch.stack([2 * a.cos(), 2 * a.sin()], 1) + 0.1 * torch.randn(n, 2)
CENTRES = torch.stack([torch.tensor([2 * math.cos(k * math.pi / 4), 2 * math.sin(k * math.pi / 4)]) for k in range(8)])
def modes(x): # how many of the 8 clusters are covered?
near = torch.cdist(x, CENTRES).min(1)
cnt = torch.bincount(near.indices[near.values < 0.4], minlength=8)
return int((cnt > 0.02 * len(x)).sum())
def plot(x, w=41, h=17):
g = [[" "] * w for _ in range(h)]
for px, py in x.tolist():
c, r = int((px + 3) / 6 * (w - 1) + .5), int((3 - py) / 6 * (h - 1) + .5)
if 0 <= c < w and 0 <= r < h: g[r][c] = "o" if g[r][c] == " " else "@"
return "\n".join("".join(r) for r in g)
import torch, torch.nn as nn
from ring import ring, modes, plot # 8 clusters on a circle; counting; text plot
def train_gan(g_steps):
torch.manual_seed(77)
G = nn.Sequential(nn.Linear(2, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 2))
D = nn.Sequential(nn.Linear(2, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 1))
oG = torch.optim.Adam(G.parameters(), 1e-3, betas=(.5, .999))
oD = torch.optim.Adam(D.parameters(), 1e-3, betas=(.5, .999)); bce = nn.BCEWithLogitsLoss()
for step in range(3000):
real, fake = ring(256), G(torch.randn(256, 2))
lD = bce(D(real), torch.ones(256, 1)) + bce(D(fake.detach()), torch.zeros(256, 1))
oD.zero_grad(); lD.backward(); oD.step()
for _ in range(g_steps): # G trained g_steps times per D step
lG = bce(D(G(torch.randn(256, 2))), torch.ones(256, 1))
oG.zero_grad(); lG.backward(); oG.step()
return G(torch.randn(2000, 2)).detach()
a, b = train_gan(1), train_gan(10)
spread = lambda x: x.std(0).mean().item()
print(f"balanced (1 G step per D step): clusters covered {modes(a)}/8, spread {spread(a):.2f}")
print(f"G rushed (10 G steps per D step): clusters covered {modes(b)}/8, spread {spread(b):.3f}")
print(f"{'real data':^31} {'balanced GAN':^31} {'G rushed: collapsed':^31}")
for r in zip(*[plot(x, 31, 13).split("\n") for x in (ring(2000), a, b)]): print(" ".join(r))
balanced (1 G step per D step): clusters covered 8/8, spread 1.43
G rushed (10 G steps per D step): clusters covered 0/8, spread 0.071
real data balanced GAN G rushed: collapsed
@ o o
@@@ o@@@@@@
@@@@o @ @@@@ @@@@@@@@@ @o@@@
@@o o@@ @@@ o o@@ o@@
o @@ o@@o @@
@@@ o@@@ oo@@@ o @@@
o @@ @@
o@o @@ o@o o@@
@@@ @@@ @@@@@@@ o o@@@@@o
@@@ o @@@@@o@
o
Balanced: 8/8 clusters, a little smeared along the ring. Rushed: 0/8, all 2,000 samples in one tiny blob (spread 0.071).
Diffusion models (DDPM)
Real-life example: a drop of ink, played backwards. Ink spreads in water until the glass is evenly grey (easy). Play the film backwards and grey water gathers into a shape. Learn each small backward step and you can turn noise into new data.
- Forward noising: over T = 1000 steps, add a little Gaussian noise each step; the amount β grows linearly from 0.0001 to 0.02.
- ᾱ_t ("a-bar"): the product of (1 − β) up to step t: how much of the original survives. Jump to any step at once:
x_t = sqrt(abar_t) * x_0 + sqrt(1 - abar_t) * eps. - Training: show the network x_t and t; it predicts the noise ε (mean squared error).
- Reverse denoising (sampling): start from pure noise x_T; at each step remove a little predicted noise and add a little fresh noise; after 1000 steps you have a sample.
import torch
torch.manual_seed(77)
x0 = torch.load("/data/mnist.pt")["test_x"][0].float() / 255 * 2 - 1 # one digit, pixels -1..1
T = 1000
beta = torch.linspace(1e-4, 0.02, T) # a little noise at every step
abar = torch.cumprod(1 - beta, 0) # how much of the original is left
row = []
for t in [0, 150, 300, 999]:
eps = torch.randn_like(x0)
xt = abar[t].sqrt() * x0 + (1 - abar[t]).sqrt() * eps # jump straight to step t
print(f"step {t + 1:4}: original kept {abar[t].sqrt():.3f}, noise {(1 - abar[t]).sqrt():.3f}")
row.append(((xt + 1) / 2).clamp(0, 1))
small = torch.nn.functional.avg_pool2d(torch.stack(row)[:, None], 2)[:, 0]
for r in range(14):
print(" ".join("".join(" .:-=+*#%@"[min(9, int(v * 10))] for v in img[r]) for img in small))
step 1: original kept 1.000, noise 0.010
step 151: original kept 0.887, noise 0.463
step 301: original kept 0.628, noise 0.778
step 1000: original kept 0.006, noise 1.000
..:. .. .. +- . .::-::-+ :*-#=:--#-=#++
:...... ..: *:: :.-:=-..: .-+++#=.-=+#**
. . . .:.:. . - :::::..#:..: ++=++:+=+.-=--
:- ..=:: ::::. ..+=:.-.:.- =- -=+*#---*+%#+#
**%%%%%* . ##%@%%#*. : :+-=*++##%-*-. =:*+:+=++:+- @
..++ :. . :=*+.: - :.::-.:%-=-. :#+%#+-=-*++*-
.@. . : . .:@: : . +.=:=+*= ::##*++#+--%--
*= : :. ...*+ ... =-=-::=.+*: -. =%*+*=#-#.-+::
.% . . ..# .. : :.:::+-. ::. +*#*+.=.:+++-*
#= - ....*+ .. -:=:-:=+# +=-- *+#*+*=-*+#=+=
=# . :. == .. =+-.*.-=::+-.- +**--===#+++=-
.@. .:...:@:. ... .**.-:%:-:.=-. *=*.+*=.+.#-:*
*@ .:.::+%.. . . :-- :**:: --.= #=*+=+#*.@:++=
-: ...=:...-. . -::..+*:.-::.- -*+#+---*+=*--
import torch, torch.nn as nn, time
from ring import ring, modes, plot
torch.manual_seed(77)
T = 1000
beta = torch.linspace(1e-4, 0.02, T) # DDPM noise schedule (Ho et al., 2020)
abar = torch.cumprod(1 - beta, 0) # how much of the original survives at step t
net = nn.Sequential(nn.Linear(3, 128), nn.SiLU(), nn.Linear(128, 128), nn.SiLU(),
nn.Linear(128, 128), nn.SiLU(), nn.Linear(128, 2)) # (x_t, t) -> predicted noise
opt = torch.optim.Adam(net.parameters(), 1e-3); t0 = time.time()
for step in range(6001):
x0 = ring(512); t = torch.randint(0, T, (512,)); eps = torch.randn_like(x0)
xt = abar[t, None].sqrt() * x0 + (1 - abar[t, None]).sqrt() * eps # forward noising, in one jump
loss = ((net(torch.cat([xt, t[:, None] / T], 1)) - eps) ** 2).mean() # learn to predict the noise
opt.zero_grad(); loss.backward(); opt.step()
if step % 2000 == 0: print(f"step {step}: noise-prediction loss {loss:.3f}")
print(f"trained in {time.time() - t0:.0f} s")
with torch.no_grad(): # reverse denoising: start from pure noise
x = torch.randn(2000, 2)
for t in reversed(range(T)):
eps = net(torch.cat([x, torch.full((2000, 1), t / T)], 1))
x = (x - beta[t] / (1 - abar[t]).sqrt() * eps) / (1 - beta[t]).sqrt()
if t > 0: x += beta[t].sqrt() * torch.randn_like(x)
print("modes covered:", modes(x), "/ 8")
print(plot(x))
step 0: noise-prediction loss 1.026
step 2000: noise-prediction loss 0.278
step 4000: noise-prediction loss 0.225
step 6000: noise-prediction loss 0.228
trained in 21 s
modes covered: 8 / 8
o@@@o
o @ o o@@@@@ o
@@@@@ o o@@@@@
@@@ @@@@o
oo o
o@o @@
@@@@@ o@@@@
oo @@@o
@oo ooo
o@@ o@@@
@@@@@@o o@@@@@@
o @@@@@@ o
@@@
All 8 clusters, with no discriminator and no collapse. The DDPM paper used the same recipe on real photographs.
Which one when?
| Model | How it learns | Strengths | Weaknesses |
|---|---|---|---|
| Autoencoder | Compress and rebuild | Compression, denoising, features | Latent space has holes; poor generator |
| VAE | Rebuild + KL to N(0, 1) | Smooth latent space; stable; sample from random z | Blurry samples |
| GAN | Generator vs discriminator | Sharp; one pass to generate | Unstable training; mode collapse |
| Diffusion | Learn to remove noise | High quality; covers all modes | Slow sampling (many steps) |
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Generating by averaging or random pixels | one blur / pure static | no model of the data | learn the distribution |
| Sampling a plain autoencoder | broken shapes | holes in the latent space | a VAE (KL to N(0, 1)) |
Normal(mu, sigma).sample() in a VAE |
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn |
the draw cuts the gradient | mu + sigma * eps, or .rsample() |
D step without fake.detach() |
G gets updates from D's loss | gradients flow into G | D(fake.detach()) for D's loss |
| Training G much more than D | mode collapse: 0/8 clusters, spread 0.071 | G exploits a weak D | balance the updates |
| Forgetting the KL term | a plain autoencoder again | nothing fills the holes | reconstruction + KL |
Mini project: walk between two digits in the VAE's latent space
Encode a 1 and a 7, take 6 evenly spaced points on the line between their μ codes, and decode each: in a VAE the in-between images morph smoothly. (Use vae.mu(vae.enc(x)) for the codes and vae.dec(z) to draw; print with ascii_row.) Try the same with the plain autoencoder and compare.
Practice
1. An autoencoder with a 2-number code: where does each digit sit?
Solution
import torch, torch.nn as nn
torch.manual_seed(77)
d = torch.load("/data/mnist.pt")
X = d["train_x"].float().reshape(-1, 784) / 255; Xt, Yt = d["test_x"].float().reshape(-1, 784) / 255, d["test_y"]
enc = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 2)) # only 2 numbers!
dec = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 784), nn.Sigmoid())
opt = torch.optim.Adam([*enc.parameters(), *dec.parameters()], 1e-3)
for epoch in range(5):
for i in torch.randperm(60000).split(128):
loss = ((dec(enc(X[i])) - X[i]) ** 2).mean(); opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
z = enc(Xt)
print(f"test reconstruction error with 2 numbers: {((dec(z) - Xt) ** 2).mean():.4f}")
for digit in range(10):
c = z[Yt == digit]
print(f"digit {digit}: code centre ({c[:, 0].mean():6.1f}, {c[:, 1].mean():6.1f}) spread {c.std(0).mean():.1f}")
test reconstruction error with 2 numbers: 0.0449
digit 0: code centre ( 5.7, -10.3) spread 3.0
digit 1: code centre ( 3.4, 1.0) spread 1.9
digit 2: code centre ( 5.5, -3.3) spread 2.5
digit 3: code centre ( 1.8, -2.3) spread 1.4
digit 4: code centre ( -1.6, -5.4) spread 2.3
digit 5: code centre ( 2.1, -2.2) spread 1.7
digit 6: code centre ( 3.3, -5.0) spread 2.0
digit 7: code centre ( -4.4, -2.8) spread 2.7
digit 8: code centre ( 2.5, -1.7) spread 2.0
digit 9: code centre ( -1.6, -4.3) spread 2.1
With only 2 numbers the reconstruction error is much higher than with 16 (0.0133), and the digits sit in different regions of a 2-D map; you could plot them as a scatter plot.
2. Change the VAE's KL weight to 0 and to 5. Hint
loss = rec + w * kl. With w = 0 it becomes a plain autoencoder (sharp rebuilds, bad random samples). With w = 5 the codes are pushed hard toward N(0, 1): smoother samples, blurrier rebuilds.
3. Train our GAN 2 more epochs and compare. Hint
range(1, 11). Watch the "D thinks fakes are real" percentage and the stray dots. Training longer does not always help a GAN.
4. Use 100 diffusion steps instead of 1000.
Solution
import torch, torch.nn as nn
from ring import ring, modes
def run(T):
torch.manual_seed(77)
beta = torch.linspace(1e-4, 0.02 * 1000 / T, T); abar = torch.cumprod(1 - beta, 0) # same total noise
net = nn.Sequential(nn.Linear(3, 128), nn.SiLU(), nn.Linear(128, 128), nn.SiLU(),
nn.Linear(128, 128), nn.SiLU(), nn.Linear(128, 2))
opt = torch.optim.Adam(net.parameters(), 1e-3)
for step in range(6001):
x0 = ring(512); t = torch.randint(0, T, (512,)); eps = torch.randn_like(x0)
xt = abar[t, None].sqrt() * x0 + (1 - abar[t, None]).sqrt() * eps
loss = ((net(torch.cat([xt, t[:, None] / T], 1)) - eps) ** 2).mean()
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
x = torch.randn(2000, 2)
for t in reversed(range(T)):
eps = net(torch.cat([x, torch.full((2000, 1), t / T)], 1))
x = (x - beta[t] / (1 - abar[t]).sqrt() * eps) / (1 - beta[t]).sqrt()
if t > 0: x += beta[t].sqrt() * torch.randn_like(x)
near = torch.cdist(x, torch.zeros(1, 2))[:, 0]
print(f"T = {T:4}: clusters covered {modes(x)}/8, points on the ring (radius 1.6-2.4): {((near > 1.6) & (near < 2.4)).float().mean():.0%}")
run(1000); run(100)
T = 1000: clusters covered 8/8, points on the ring (radius 1.6-2.4): 100%
T = 100: clusters covered 8/8, points on the ring (radius 1.6-2.4): 100%
β is scaled up (×10) so the total noise is similar. Here 100 steps did just as well on this easy 2-D data, and sample 10 times faster; on images, fewer steps usually costs quality.
Quick quiz
A generator makes only 3 of the 10 digits, each one perfect, and the discriminator is fooled. What is this called?
Answer
Mode collapse. The outputs look great, but most kinds of data are missing.
Interview questions
- Discriminative vs generative? p(y given x) vs p(x) (or p(x, y)); only a generative model can sample new x.
- Why can't you sample from a plain autoencoder? Its latent space has unstructured holes; random codes decode to garbage.
- What does the KL term in a VAE do? Keeps each image's code distribution close to N(0, 1), so the latent space is filled and sampleable.
- Explain the reparameterisation trick. Write z = μ + σ·ε with ε ~ N(0, I), so z is a differentiable function of μ and σ.
- How is a GAN trained? Alternate: D learns to separate real from fake; G learns to make D call fakes real.
- What is mode collapse and how do you reduce it? G covers only a few modes; balance G and D updates, and use better losses or architectures.
- What does a diffusion model learn? To predict the noise added at each step, so it can denoise step by step from pure noise.
- Why are diffusion models slow to sample? Generating takes many sequential denoising steps (1000 in DDPM).
Glossary
| Term | Meaning, with an example |
|---|---|
| Discriminative model | Maps input to label: "this is a 7" |
| Generative model | Learns what the data looks like and makes new examples: draws a new 7 |
| Distribution | How likely each possible value is |
| N(0, 1) | The standard normal: bell curve, mean 0, standard deviation 1 |
| Autoencoder | Encoder + decoder trained to rebuild its own input |
| Encoder / decoder | Input → short code / code → rebuilt input |
| Code (z) | The short vector in the middle: 16 numbers per digit |
| Bottleneck | The narrow middle layer that forces compression |
| Latent space | The space of all codes; each image is a point |
| Reconstruction loss | How different the rebuilt input is from the original |
| Denoising autoencoder | Noisy input, clean target: learns to remove noise |
| VAE | Variational autoencoder: codes are distributions (μ, σ), pulled to N(0, 1) |
| μ (mu) / σ (sigma) | Mean / spread of a code number's distribution |
| Log-variance | log σ²; predicted instead of σ so it can be any real number |
| KL divergence | How different two distributions are; here, the code area vs N(0, 1) |
| Reparameterisation trick | z = μ + σ·ε, ε ~ N(0, 1): randomness moved to a constant input |
.rsample() |
PyTorch's reparameterised sampling |
| GAN | Generative adversarial network: generator vs discriminator |
| Generator (G) | Noise → fake sample |
| Discriminator (D) | Sample → probability it is real |
| Minimax game | One player minimises what the other maximises |
detach() |
Cut a tensor from the gradient graph |
| Mode | One kind of thing in the data: a digit, a cluster |
| Mode collapse | G produces only a few kinds of output |
| Helvetica scenario | The GAN paper's name for G mapping many z to the same output |
| Diffusion model | Learns to reverse a gradual noising process |
| DDPM | Denoising Diffusion Probabilistic Models (Ho, Jain, Abbeel, 2020) |
| Forward noising | Adding a little Gaussian noise at each of T steps |
| Noise schedule (β) | How much noise each step adds: 0.0001 → 0.02 |
| ᾱ_t (a-bar) | Product of (1 − β) up to step t: how much original survives |
| Reverse denoising | Generating: from pure noise, remove predicted noise step by step |
Summary
- Random pixels give static and averages give one blur: generating needs a model of the data.
- Autoencoders compress through a bottleneck (test error 0.0133 with 16 numbers) but their latent space has holes; denoising autoencoders learn to clean (0.1188 → 0.0175).
- VAEs fill the holes with KL to N(0, 1) and train through the reparameterisation trick; samples are new but blurry.
- GANs make sharper samples by pitting G against D; rushing G caused mode collapse (8/8 → 0/8 clusters).
- Diffusion models learn to remove noise step by step and covered all 8 clusters.
Next: Episode 78, saving and serving models: pickle, joblib, ONNX, torch.export, a FastAPI prediction API, latency, batch vs online inference.