SyntaxPilot
Courses/Machine Learning/Lesson 58
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 58 of 90

PyTorch 1: tensors and autograd

tensordtypeshapereshape and viewindexingbroadcastingdevice (CPU, GPU, CUDA)NumPy interopautogradrequires_graddynamic computation graphbackward().gradtorch.no_grad()

Machine Learning Full Course · Episode 58. Every code sample below was run with Python 3.13, NumPy 2.5.3 and PyTorch 2.15.0 (the CPU build; the recording machine has no NVIDIA GPU). The output shown is the real output. Timings change from machine to machine; the numbers computed do not.

The big idea: hand-written backpropagation breaks silently. Swap sigmoid for tanh, forget to change the derivative, and the gradient comes out 0 where the truth is −4, with no error. PyTorch fixes this with two things. A tensor is a grid of numbers with a shape, a dtype and a device (CPU or GPU). Autograd records every operation done to a tensor with requires_grad=True; calling backward() on the loss walks that record backwards (backpropagation) and puts each gradient in .grad. You only write the forward pass. Three traps come with it: gradients accumulate (−2, −4, −6), the graph is freed after backward(), and updating a weight in place while recording is refused. The fixes: update inside torch.no_grad(), empty gradients with .grad.zero_(), and detach() before converting to NumPy.

Sources: PyTorch was launched by Facebook (now Meta) in 2016, and in September 2022 it moved to the PyTorch Foundation, part of the Linux Foundation (Engadget, 12 Sep 2022). NVIDIA introduced CUDA, an architecture for computing on its GPUs, on 16 November 2006 (Computer Graphics World).

Words you need first

  • Array, NumPy, shape, broadcasting (Episode 2): NumPy stores numbers in grids called arrays; broadcasting stretches smaller arrays to match bigger ones.
  • Matrix product @ (Episode 4): rows of the left matrix times columns of the right.
  • Derivative, chain rule, numerical derivative (Episode 6).
  • Computational graph (Episode 7): a diagram of operations; each node is one step of the calculation.
  • Loss, gradient descent, learning rate (Episode 8).
  • Backpropagation, local gradient, weight update rule (Episode 57): multiply local gradients from the loss backwards; w ← w − lr × dL/dw.
  • tanh (Episode 56): an activation function squashing any number into −1 … 1; its slope is 1 − a².

The problem: hand-written derivatives break silently

import math                                    # last episode: hand-written backprop for one neuron
x, y, w, b = 2.0, 1.0, 0.5, -1.0
act  = lambda z: math.tanh(z)                  # we switched sigmoid -> tanh...
dact = lambda a: a * (1 - a)                   # ...but forgot to change its derivative
def loss(w, b): return (act(w * x + b) - y) ** 2
a = act(w * x + b)
grad_w = 2 * (a - y) * dact(a) * x             # chain rule, by hand
num_w = (loss(w + 1e-5, b) - loss(w - 1e-5, b)) / 2e-5
print(f"hand-written dL/dw = {grad_w + 0:.6f}")
print(f"numerical    dL/dw = {num_w:.6f}")
print("match!" if abs(grad_w - num_w) < 1e-4 else "WRONG gradient, and no error message")

Output:

hand-written dL/dw = 0.000000
numerical    dL/dw = -4.000000
WRONG gradient, and no error message

z = 0.5 × 2 − 1 = 0, so a = tanh(0) = 0 and the wrong derivative a(1 − a) is 0. The true slope of tanh at 0 is 1 − a² = 1, giving 2 × (0 − 1) × 1 × 2 = −4. Training with the wrong value would never move w.

Why: the derivative was typed by a person, separately from the model. Change one, forget the other, and they drift apart without an error. Also, NumPy arrays only run on the CPU.

What we want: arrays that remember how they were made (so the computer can differentiate them itself) and that can run on a graphics card. That is PyTorch.

PyTorch and the tensor

  • PyTorch: a free, open-source deep learning library for Python. Install it with pip install torch; import it as torch.
  • Tensor: a grid of numbers with any number of dimensions. NumPy's array, plus a device and gradient tracking.
Real life Tensor Shape
The price of one chai: 15 0-D tensor, a scalar []
A bill: 15, 20, 40 1-D tensor, a vector [3]
A train timetable 2-D tensor, a matrix [rows, cols]
A stack of 32 colour photos 4-D tensor [32, 3, 224, 224] (photo, colour, height, width)
import torch
t = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])   # a tensor: a grid of numbers
print(t)
print("shape:", t.shape, " ndim:", t.ndim, " dtype:", t.dtype)
i = torch.tensor([1, 2, 3])                            # whole numbers -> int64
print("i.dtype:", i.dtype, "  t.int():", t.int().dtype, "  half:", t.half().dtype)
print("zeros:", torch.zeros(2, 2).tolist(), " arange:", torch.arange(5).tolist())
print("random:", torch.rand(3).shape, torch.randn(2, 3).shape)
print(torch.__version__)

Output:

tensor([[1., 2., 3.],
        [4., 5., 6.]])
shape: torch.Size([2, 3])  ndim: 2  dtype: torch.float32
i.dtype: torch.int64   t.int(): torch.int32   half: torch.float16
zeros: [[0.0, 0.0], [0.0, 0.0]]  arange: [0, 1, 2, 3, 4]
random: torch.Size([3]) torch.Size([2, 3])
2.15.0+cu132

(+cu132 is part of the version name of this PyTorch build; it does not mean a GPU is present.)

Shape and dtype

  • shape: the size along each dimension, e.g. torch.Size([2, 3]). ndim: the number of dimensions.
  • dtype (data type): the type of every number inside; one dtype per tensor.
dtype Bytes per number Default for Example
torch.float32 4 decimals weights, inputs
torch.float64 8 NumPy decimals torch.from_numpy(np.array([1.0]))
torch.float16 (.half()) 2 mixed precision (episode 64)
torch.int64 ("long") 8 whole numbers class labels
torch.bool 1 comparisons t > 2

float32 is precise enough for neural networks and fast on GPUs. Mixing dtypes in + promotes the result, but a matrix product needs the same dtype on both sides:

import torch
a = torch.tensor([1, 2, 3])                      # int64
b = torch.tensor([0.5, 0.5, 0.5])                # float32
print(a + b, (a + b).dtype)                      # mixing is fine: promoted to float
W = torch.randn(3, 3)
print(a @ W)                                     # but a matrix product needs the same dtype

Output:

tensor([1.5000, 2.5000, 3.5000]) torch.float32
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    print(a @ W)                                     # but a matrix product needs the same dtype
          ~~^~~
RuntimeError: expected m1 and m2 to have the same dtype, but got: long int != float

Fix: a.float() @ W. "long int" is PyTorch's internal name for int64.

Reshape, view and indexing

Twelve eggs can sit in a line, in a 3 × 4 tray or a 4 × 3 tray: the same eggs, arranged differently.

  • reshape: the same numbers in a new shape. -1 means "work this size out for me".
  • view: a new shape on the same memory, no copy. Change the view and the original changes.
  • Indexing: picking parts of a tensor: m[1] (row 1), m[:, 2] (column 2), m[m > 8] (a boolean mask: keep the elements where the condition is True).
import torch
t = torch.arange(12)                       # 0, 1, ..., 11: shape [12]
m = t.reshape(3, 4)                        # same 12 numbers, as 3 rows x 4 columns
print(m)
v = t.view(4, 3); v[0, 0] = 100            # view: a new shape on the SAME memory
print("t[0] after changing v:", t[0].item())
print("m[1]:", m[1].tolist(), " m[:, 2]:", m[:, 2].tolist(), " m[m > 8]:", m[m > 8].tolist())
print("reshape(2, -1):", t.reshape(2, -1).shape)
print(m.T.is_contiguous()); m.T.view(12)   # transposed: not one tidy block of memory

Output:

tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])
t[0] after changing v: 100
m[1]: [4, 5, 6, 7]  m[:, 2]: [2, 6, 10]  m[m > 8]: [100, 9, 10, 11]
reshape(2, -1): torch.Size([2, 6])
False
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    print(m.T.is_contiguous()); m.T.view(12)   # transposed: not one tidy block of memory
                                ~~~~~~~~^^^^
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

m was a reshape of a contiguous tensor, so it is also a view of the same memory: that is why 100 appears in m[m > 8].

Contiguous means the numbers are stored in memory in reading order (row after row). .T (transpose) swaps rows and columns by changing how memory is read, without moving numbers, so the result is not contiguous. view only works on contiguous memory; reshape returns a view when it can and a copy when it must. .contiguous() makes a contiguous copy.

Broadcasting

Two chai stalls (rows) sell chai, samosa and biscuits (columns). Tax is per item (one number per column); a discount is per shop (one number per row).

import torch
prices = torch.tensor([[40.0, 60.0, 25.0],      # 2 shops x 3 items (chai, samosa, biscuit)
                       [45.0, 55.0, 30.0]])
gst = torch.tensor([1.05, 1.05, 1.12])           # one tax rate per item: shape [3]
print(prices * gst)                              # [2, 3] * [3] -> stretched over every row
discount = torch.tensor([0.9, 1.0])              # one discount per shop: shape [2]
print(prices * discount)

Output:

tensor([[42.0000, 63.0000, 28.0000],
        [47.2500, 57.7500, 33.6000]])
Traceback (most recent call last):
  File "main.py", line 7, in <module>
    print(prices * discount)
          ~~~~~~~^~~~~~~~~~
RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1

The fix is a column, shape [2, 1]:

discount = torch.tensor([[0.9], [1.0]])          # one discount per shop: shape [2, 1]
print(prices * discount)                         # [2, 3] * [2, 1] -> stretched over every column

Output:

tensor([[36.0000, 54.0000, 22.5000],
        [45.0000, 55.0000, 30.0000]])

Broadcasting rules: line the shapes up from the right; each pair of sizes must be equal or one of them must be 1; a size of 1 (or a missing dimension) is stretched. [2, 3] × [3] ✓, [2, 3] × [2, 1] ✓, [2, 3] × [2] ✗ (3 vs 2). The stretch is virtual: no copies are made.

Devices: CPU, GPU and CUDA

Wedding kitchen Computer
A master chef: few hands, any recipe CPU (central processing unit): a few powerful cores
1,000 helpers chopping onions at once GPU (graphics processing unit): thousands of simple cores, ideal for the millions of multiply-adds in a network
The way the helpers receive orders CUDA: NVIDIA's platform for general computing on its GPUs (introduced 2006); in PyTorch the GPU device is called "cuda"
import torch
print("GPU available?", torch.cuda.is_available())
device = "cuda" if torch.cuda.is_available() else "cpu"   # the standard one-liner
x = torch.randn(1000, 1000, device=device)
print("x lives on:", x.device)
y = torch.randn(1000, 1000)
print((x @ y).shape, "fine: both on", device)
try:
    g = torch.randn(3, device="cuda")                      # ask for a GPU that isn't there
except RuntimeError as e:
    print("RuntimeError:", str(e).split(".")[0] + ".")

Output (this machine has no NVIDIA GPU):

GPU available? False
x lives on: cpu
torch.Size([1000, 1000]) fine: both on cpu
RuntimeError: Found no NVIDIA driver on your system.

Rules:

  • Pick the device once with the one-liner; never hard-code "cuda".
  • Move a tensor with x = x.to(device) (it returns a new tensor).
  • Both sides of an operation must be on the same device; otherwise PyTorch raises "Expected all tensors to be on the same device".
  • No NVIDIA card? Free notebook services such as Google Colab and Kaggle offer GPUs for practice.

NumPy interop

import numpy as np, torch
arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)              # shares memory with the NumPy array
c = torch.tensor(arr)                  # a copy
arr[0] = 99
print("from_numpy:", t.tolist(), " copy:", c.tolist(), " dtype:", t.dtype)
back = (t * 2).numpy()                 # tensor -> NumPy
print(type(back).__name__, back)
w = torch.tensor([1.0, 2.0], requires_grad=True)
print(w.numpy())

Output:

from_numpy: [99.0, 2.0, 3.0]  copy: [1.0, 2.0, 3.0]  dtype: torch.float64
ndarray [198.   4.   6.]
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    print(w.numpy())
          ~~~~~~~^^
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
  • torch.from_numpy(arr) shares memory (like a view); torch.tensor(arr) copies.
  • NumPy's default float is float64; convert with .float() before mixing with float32 model weights.
  • .numpy() needs a CPU tensor that does not require grad: x.detach().cpu().numpy().

Autograd

  • requires_grad=True: "record every operation done to me".
  • grad_fn: each result remembers the function that made it (PowBackward0, SigmoidBackward0). Together these records are the computational graph.
  • backward(): called on the loss, walks the graph backwards multiplying local gradients (backpropagation).
  • .grad: where each gradient lands.
  • Leaf tensor: a tensor you created yourself (like w), not the result of an operation; gradients are stored on leaves.
  • Autograd = automatic differentiation.
import torch                                   # the same neuron as episode 57, now with autograd
x, y = torch.tensor(2.0), torch.tensor(1.0)
w = torch.tensor(0.5, requires_grad=True)      # "please track every operation on me"
b = torch.tensor(-1.0, requires_grad=True)
z = w * x + b
a = torch.sigmoid(z)
L = (a - y) ** 2
print("L =", L, " made by:", type(a.grad_fn).__name__)
L.backward()                                   # backpropagation, done for us
print("dL/dw =", w.grad.item(), "  dL/db =", b.grad.item())
a = torch.tanh(w * x + b)                      # switch to tanh: no derivative to rewrite
w.grad = None; ((a - y) ** 2).backward()
print("tanh: dL/dw =", w.grad.item())

Output:

L = tensor(0.2500, grad_fn=<PowBackward0>)  made by: SigmoidBackward0
dL/dw = -0.5   dL/db = -0.25
tanh: dL/dw = -4.0

−0.5 and −0.25 are exactly episode 57's hand-written results; −4 matches the numerical check above.

The dynamic computation graph

PyTorch builds the graph while the code runs (define-by-run). Each run records only the operations that really happened, so ordinary Python if, for and while work inside a model, and you can print anything mid-way.

import torch
w = torch.tensor(3.0, requires_grad=True)
def f(w, steps):                     # ordinary Python: a loop and an if
    out = w
    for _ in range(steps):
        out = out * w if out < 50 else out + 1
    return out
for steps in (2, 5):
    w.grad = None; y = f(w, steps); y.backward()
    print(f"steps={steps}: y = {y.item():.0f}, dy/dw = {w.grad.item():.0f}")

Output:

steps=2: y = 27, dy/dw = 27
steps=5: y = 83, dy/dw = 108

Two steps: y = w³, dy/dw = 3w² = 27. Five steps: 3 → 9 → 27 → 81, then the if switches to +1 twice: y = w⁴ + 2, dy/dw = 4w³ = 108.

Autograd at scale

import numpy as np, torch, time; rng = np.random.default_rng(57)
X = rng.random((64, 784))                                  # the same network as episode 57
W1, W2 = rng.normal(0, .03, (784, 1000)), rng.normal(0, .03, (1000, 10))
Z = X @ W1; H = np.maximum(0, Z); O = H @ W2               # episode 57: backward pass by hand
dO = 2 * (O - O.mean()) / O.size; dW2 = H.T @ dO; dW1 = X.T @ (dO @ W2.T * (Z > 0))
Xt = torch.from_numpy(X)                                   # now PyTorch: forward pass only
T1 = torch.from_numpy(W1).requires_grad_(); T2 = torch.from_numpy(W2).requires_grad_()
t = time.time(); L = (torch.relu(Xt @ T1) @ T2).var(correction=0); L.backward()
print(f"autograd: {T1.numel() + T2.numel():,} gradients in {(time.time() - t) * 1000:.0f} ms")
print("same as hand backprop?", np.allclose(T1.grad.numpy(), dW1), np.allclose(T2.grad.numpy(), dW2))

Output (one run):

autograd: 794,000 gradients in 9 ms
same as hand backprop? True True

requires_grad_() (with an underscore) switches tracking on in place. var(correction=0) is the plain average of squared deviations, matching NumPy's .var().

The training-loop traps

Trap 1 and 2: gradients accumulate; the graph is freed

import torch
w = torch.tensor(0.5, requires_grad=True)
for i in range(3):
    L = (w * 2.0 - 1.5) ** 2           # the same loss every time
    L.backward()
    print(f"call {i + 1}: w.grad = {w.grad.item()}")
try:
    L.backward()                       # once more, on a graph already used
except RuntimeError as e:
    print("RuntimeError:", str(e).split(" (")[0])

Output:

call 1: w.grad = -2.0
call 2: w.grad = -4.0
call 3: w.grad = -6.0
RuntimeError: Trying to backward through the graph a second time

The true gradient is 2 × (0.5 × 2 − 1.5) × 2 = −2 each time, but .grad adds new gradients to what is already there. After backward(), PyTorch frees the values the graph saved, to save memory; running the forward pass again builds a new graph. (retain_graph=True keeps it, rarely needed.)

Trap 3: updating a leaf while recording

import torch
w = torch.tensor(0.5, requires_grad=True)
for step in range(3):
    L = (w * 2.0 - 1.5) ** 2
    L.backward()
    w -= 0.1 * w.grad                    # the update rule from episode 57

Output:

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    w -= 0.1 * w.grad                    # the update rule from episode 57
    ^^^^^^^^^^^^^^^^^
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

An in-place operation changes a tensor's own memory (-=, zero_(), anything ending in _). Doing it on a recorded leaf would make the update part of the graph, so PyTorch refuses.

The fixes

import torch                             # gradient descent on one weight, the PyTorch way
w = torch.tensor(0.5, requires_grad=True)
for step in range(1, 31):
    L = (w * 2.0 - 1.5) ** 2             # we want w * 2 = 1.5, so w = 0.75
    L.backward()
    with torch.no_grad():                # "this is an update, not part of the model"
        w -= 0.1 * w.grad
    w.grad.zero_()                       # empty the gradient for the next step
    if step in (1, 2, 10, 30): print(f"step {step:2}: w = {w.item():.4f}, loss = {L.item():.6f}")
with torch.no_grad(): p = w * 2.0
print("inside no_grad, requires_grad =", p.requires_grad)
d = (w * 2.0).detach()
print("detach: requires_grad =", d.requires_grad, " numpy:", d.numpy())

Output:

step  1: w = 0.7000, loss = 0.250000
step  2: w = 0.7400, loss = 0.010000
step 10: w = 0.7500, loss = 0.000000
step 30: w = 0.7500, loss = 0.000000
inside no_grad, requires_grad = False
detach: requires_grad = False  numpy: 1.5
  • with torch.no_grad(): switches recording off inside the block: for updates and for predictions (saves memory and time).
  • .grad.zero_() empties a gradient in place (next episode, optimizer.zero_grad() does it for every parameter).
  • Run the forward pass again every step for a fresh graph.
  • detach() returns the same numbers cut off from the graph.
  • Accumulation is a feature: it lets you sum gradients from several small batches (gradient accumulation, episode 64).

Common mistakes

Mistake Exact message / symptom Why Fix
Hand derivative not updated hand-written dL/dw = 0.000000 vs numerical -4.000000 model and maths drift apart autograd
int64 @ float32 RuntimeError: expected m1 and m2 to have the same dtype, but got: long int != float one dtype per operation a.float() @ W
m.T.view(12) RuntimeError: view size is not compatible with input tensor's size and stride ... Use .reshape(...) instead. not contiguous reshape or .contiguous().view
[2, 3] * [2] RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1 shapes align from the right shape [2, 1]
Hard-coded device="cuda" RuntimeError: Found no NVIDIA driver on your system. ... no GPU torch.cuda.is_available()
.numpy() on a tracked tensor RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead. still in the graph detach()
Calling backward() twice RuntimeError: Trying to backward through the graph a second time ... graph freed rerun the forward pass
w -= lr * w.grad while recording RuntimeError: a leaf Variable that requires grad is being used in an in-place operation. the update would be recorded with torch.no_grad():
Never zeroing .grad -2.0, -4.0, -6.0 gradients accumulate w.grad.zero_()

Mini project: XOR with tensors and autograd

Episode 57's XOR network, with no hand-written backward pass:

import torch; torch.manual_seed(58)            # mini project: XOR with tensors + autograd (episode 57, no hand backprop)
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]]); y = torch.tensor([[0.], [1.], [1.], [0.]])
W1 = torch.empty(2, 3).uniform_(-1, 1).requires_grad_(); b1 = torch.zeros(3, requires_grad=True)
W2 = torch.empty(3, 1).uniform_(-1, 1).requires_grad_(); b2 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2]
for epoch in range(1, 5001):
    out = torch.sigmoid(torch.sigmoid(X @ W1 + b1) @ W2 + b2)
    loss = ((out - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        for p in params: p -= 2.0 * p.grad; p.grad.zero_()
    if epoch in (1, 1000, 5000): print(f"epoch {epoch}: loss {loss.item():.4f}")
print("outputs:", [round(v, 3) for v in out.detach().squeeze(1).tolist()])

Output:

epoch 1: loss 0.2820
epoch 1000: loss 0.0030
epoch 5000: loss 0.0003
outputs: [0.009, 0.984, 0.981, 0.021]

Zero, one, one, zero: XOR learned, and we never wrote a derivative.

Practice

1. Make a random tensor of shape [32, 3, 28, 28] and reshape it to [32, -1].

Solution
import torch
x = torch.rand(32, 3, 28, 28)              # 32 colour images, 28 x 28 pixels
flat = x.reshape(32, -1)
print(x.shape, "->", flat.shape)           # 3 * 28 * 28 = 2352

Output:

torch.Size([32, 3, 28, 28]) -> torch.Size([32, 2352])

2. Use autograd to find dy/dx of y = x³ + 2x at x = 2.

Solution
import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3 + 2 * x
y.backward()
print("y =", y.item(), " dy/dx =", x.grad.item())   # 3x^2 + 2 = 14

Output:

y = 12.0  dy/dx = 14.0

3. Fit w and b so that w·x + b matches y = 3x + 1.

Solution
import torch; torch.manual_seed(58)
x = torch.linspace(-1, 1, 50); y = 3 * x + 1          # data from the line y = 3x + 1
w = torch.zeros((), requires_grad=True); b = torch.zeros((), requires_grad=True)
for step in range(1, 301):
    loss = ((w * x + b - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad; b -= 0.1 * b.grad
    w.grad.zero_(); b.grad.zero_()
    if step in (1, 100, 300): print(f"step {step}: w = {w.item():.4f}, b = {b.item():.4f}, loss = {loss.item():.6f}")

Output:

step 1: w = 0.2082, b = 0.2000, loss = 4.122449
step 100: w = 2.9977, b = 1.0000, loss = 0.000002
step 300: w = 3.0000, b = 1.0000, loss = 0.000000

4. Break it on purpose: remove the two zero_() calls and watch.

Solution
import torch
x = torch.linspace(-1, 1, 50); y = 3 * x + 1
w = torch.zeros((), requires_grad=True); b = torch.zeros((), requires_grad=True)
for step in range(1, 301):
    loss = ((w * x + b - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad; b -= 0.1 * b.grad
    # forgot: w.grad.zero_(); b.grad.zero_()
    if step in (1, 10, 50, 300): print(f"step {step}: w = {w.item():.4f}, b = {b.item():.4f}, loss = {loss.item():.4f}")

Output:

step 1: w = 0.2082, b = 0.2000, loss = 4.1224
step 10: w = 5.8241, b = 0.9760, loss = 2.2501
step 50: w = 0.8376, b = 1.7253, loss = 3.3575
step 300: w = 5.0037, b = 1.9260, loss = 3.2670

The accumulated gradients keep pushing long after the answer is passed: w swings around 3 and the loss never settles.

Quick quiz

w = torch.tensor(2.0, requires_grad=True)
for _ in range(2):
    L = w * w
    L.backward()
print(w.grad)
Answer

tensor(8.) dL/dw = 2w = 4 each time; each loop builds a fresh graph (no error), and .grad accumulates: 4 + 4 = 8.

More questions:

  1. What is the shape of a batch of 16 grayscale 28 × 28 images?
    Answer[16, 1, 28, 28] (one colour channel).
  2. What dtype does torch.tensor([1, 2]) have? And torch.tensor([1.0, 2.0])?
    Answerint64 and float32.
  3. Can [4, 1] and [3] broadcast? To what shape?
    AnswerYes: from the right, 1 vs 3 (stretch) and 4 vs missing (stretch): [4, 3].
  4. Why does torch.from_numpy(arr) change when arr changes?
    AnswerIt shares memory with the NumPy array instead of copying.
  5. What does torch.no_grad() do?
    AnswerTurns off recording inside its block, so results do not require grad and no graph is built.

Interview questions

  1. What is a tensor in PyTorch? An n-dimensional array with a shape, a dtype and a device, which can also record operations for automatic differentiation.
  2. view vs reshape? view always shares memory and needs a contiguous tensor; reshape returns a view when possible and copies otherwise.
  3. Explain broadcasting. Shapes are aligned from the right; dimensions must be equal or 1; size-1 or missing dimensions are virtually expanded.
  4. How does autograd work? Operations on tensors with requires_grad=True build a graph of grad_fn nodes during the forward pass; backward() traverses it in reverse applying the chain rule and accumulates gradients into leaf tensors' .grad.
  5. What does dynamic computation graph mean? The graph is built anew on every forward pass as the code runs, so Python control flow is allowed and the graph matches what actually executed.
  6. Why must you zero gradients? .grad accumulates across backward() calls; without zeroing, each step uses the sum of all past gradients.
  7. Difference between torch.no_grad() and detach()? no_grad is a context that stops recording for everything inside; detach() returns one tensor cut off from the graph.

Glossary

Term Meaning (plain words + example)
PyTorch Open-source Python library for deep learning; import torch
Tensor A grid of numbers with any number of dimensions: torch.tensor([[1., 2.], [3., 4.]])
Scalar / vector / matrix 0-D, 1-D and 2-D tensors: one chai price, a bill, a timetable
Shape Size along each dimension: torch.Size([2, 3])
ndim Number of dimensions: 2 for a matrix
dtype Type of every number in a tensor: torch.float32, torch.int64
float32 / float64 / float16 Decimal numbers using 4 / 8 / 2 bytes each
int64 (long) 8-byte whole numbers, the default for integer tensors and class labels
reshape Same numbers, new shape: arange(12).reshape(3, 4)
view A new shape on the same memory: changing it changes the original
-1 in reshape "Work this size out": reshape(2, -1) → [2, 6]
Contiguous Stored in memory in reading order; .T makes a tensor non-contiguous
Transpose (.T) Swap rows and columns
Indexing Picking parts: m[1], m[:, 2]
Boolean mask A True/False tensor used to select elements: m[m > 8]
Broadcasting Automatic stretching of size-1 or missing dimensions: [2, 3] * [3]
Device Where a tensor lives: "cpu" or "cuda"
CPU Central processing unit: a few powerful cores
GPU Graphics processing unit: thousands of simple cores working in parallel
CUDA NVIDIA's platform for computing on its GPUs; PyTorch's name for the GPU device
.to(device) Returns a copy of a tensor on another device
NumPy interop Moving data between NumPy and PyTorch: torch.from_numpy, .numpy()
torch.from_numpy Wraps a NumPy array without copying (shared memory)
Autograd Automatic differentiation: PyTorch computes gradients from the recorded forward pass
requires_grad Flag telling PyTorch to record operations on a tensor: torch.tensor(0.5, requires_grad=True)
grad_fn The recorded function that produced a tensor: SigmoidBackward0
Computational graph The record of operations from inputs to the loss
Dynamic computation graph A graph rebuilt on every run as the code executes (define-by-run)
backward() Runs backpropagation from a scalar loss and fills .grad
.grad Where a leaf tensor's gradient is stored
Leaf tensor A tensor created by you (not by an operation), such as a weight
Gradient accumulation .grad adds new gradients to old ones: −2, −4, −6
In-place operation Changes a tensor's own memory: w -= ..., zero_()
torch.no_grad() A block in which nothing is recorded: updates and predictions
zero_() Sets a tensor to zero in place: w.grad.zero_()
detach() Same numbers, cut off from the graph: x.detach().numpy()
.item() Turns a one-number tensor into a Python number

Summary

  • Hand-written derivatives break silently when the model changes; autograd removes them.
  • A tensor has a shape, a dtype and a device; view shares memory, reshape is the safe choice, and broadcasting stretches size-1 dimensions from the right.
  • Pick the device with torch.cuda.is_available(); keep both sides of an operation on the same device. from_numpy shares memory.
  • requires_grad=True + forward pass + loss.backward() → gradients in .grad, identical to hand backprop (794,000 of them in milliseconds).
  • Update inside torch.no_grad(), zero gradients every step, rerun the forward pass, and detach() before NumPy.

Next episode: PyTorch 2: nn.Module and the training loop: layers, loss functions, optimisers and the full training loop.