SyntaxPilot
Courses/AI: Zero to Agents/Lesson 11

AI: Zero to Agents · Lesson 11 of 40 · 12 min

Parameters, hyperparameters and compute

queued

These are the notes for AI episode 11. So far our models had nine knobs, a hundred thousand, and (AlexNet) sixty million. Here you separate the numbers a model learns (parameters) from the numbers you choose (hyperparameters), see why the learning rate can't be learned by training, tune a hyperparameter honestly on a validation set, and then work out the bill: how many bytes a big model needs, how many operations (FLOPs) it takes to train, why that needs GPUs working in parallel, and what scaling laws and Chinchilla taught the field. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real (Python 3.13; the speed line depends on your machine).

Two kinds of numbers: Raju's chai

Raju's chai stall A machine learning model
Sugar and ginger amounts, found by tasting and adjusting Parameters: weights and biases, learned from data by training
How much to adjust after each taste (a pinch or a spoon?), how many rounds of tasting Hyperparameters: learning rate, number of epochs, number of layers…
Decided before cooking, by Raju, not by the tasting Set before training; training does not learn them

Hyper means "above": hyperparameters sit above training and control how it runs.

Problem 1: who picks the learning rate?

Raju's milk model from episode 7: extra cups of milk = w × the drop in temperature. The data says w should be 3. Gradient descent finds w, but the learning rate lr (how big each downhill step is) comes from outside.

# Raju's milk model: extra cups = w x temperature drop. Find w.
data = [(1, 3), (2, 6), (3, 9), (4, 12)]     # (drop in °C, extra cups)

def train(lr, steps=20):
    w = 0.0
    for _ in range(steps):
        grad = sum(2 * (w * x - y) * x for x, y in data) / len(data)
        w = w - lr * grad                     # gradient descent step
    loss = sum((w * x - y) ** 2 for x, y in data) / len(data)
    return w, loss

for lr in [0.001, 0.05, 0.2]:
    w, loss = train(lr)
    print(f"learning rate {lr:<5}: w = {w:.3g}, loss = {loss:.3g}")

Output:

learning rate 0.001: w = 0.783, loss = 36.9
learning rate 0.05 : w = 3, loss = 5.58e-23
learning rate 0.2  : w = -3.15e+06, loss = 7.42e+13
  • Too small (0.001): after 20 steps w has only reached 0.78. It crawls.
  • Just right (0.05): w = 3, loss ≈ 0.
  • Too big (0.2): each step overshoots the valley, further every time: it explodes. Push it further (train(0.5, steps=200)) and Python stops with OverflowError: (34, 'Numerical result out of range').

Same data, same code; the only change was a number training can't choose.

Problem 2: picking by training error lies

Remember Rohan (episode 6), who judged himself on the practice paper he had memorised: 100% every time, then failed the real exam. The same trap appears when you pick a hyperparameter by training error.

Raju predicts chai sales by copying the k most similar days (by temperature) and averaging them. k is a hyperparameter.

# predict cups of chai from the temperature: copy the k nearest days
train = [(14, 92), (16, 85), (18, 88), (20, 71), (22, 75), (24, 62),
         (26, 66), (28, 51), (30, 55), (32, 40)]      # (°C, cups sold)
valid = [(15, 86), (19, 78), (23, 70), (27, 57), (31, 46)]

def predict(temp, k, days):
    nearest = sorted(days, key=lambda d: abs(d[0] - temp))[:k]
    return sum(cups for _, cups in nearest) / k

def error(k, check):          # mean absolute error, in cups
    return sum(abs(predict(t, k, train) - c) for t, c in check) / len(check)

for k in [1, 3, 5, 10]:
    print(f"k = {k:2}: training {error(k, train):5.1f}   validation {error(k, valid):5.1f}")

Output:

k =  1: training   0.0   validation   7.8
k =  3: training   6.0   validation   2.3
k =  5: training   5.2   validation   4.8
k = 10: training  13.7   validation  12.5

With k = 1, a training day's nearest day is itself, so training error is a perfect 0.0. By training error, k = 1 always wins. On days the model hasn't seen, it's the worst but one (7.8 cups off). The honest winner is k = 3 (2.3).

The fix: choose hyperparameters on the validation set. This is hyperparameter tuning; trying every value from a list is grid search. The test set stays locked until the very end.

try values (grid search) → train each on the training set → keep the best on the validation set → report once on the test set

Parameters vs hyperparameters

Parameters Hyperparameters
Who sets them training (gradient descent, backpropagation) you, before training
Examples weights, biases, embedding vectors learning rate, epochs, batch size, number of layers and neurons, k, number of BPE merges
How many thousands to billions usually a handful to a few dozen
Chosen using the training set the validation set

How many parameters? Counting a dense layer

A dense layer has inputs × neurons weights plus one bias per neuron.

def dense(n_in, n_out):           # every input -> every neuron, + 1 bias each
    return n_in * n_out + n_out

digit_reader = dense(784, 128) + dense(128, 10)
print(f"episode 9 digit reader: {digit_reader:,} parameters")
gpt3 = 175_000_000_000
print(f"GPT-3:                  {gpt3:,} parameters")
print(f"that's {gpt3 // digit_reader:,} digit readers")

Output:

episode 9 digit reader: 101,770 parameters
GPT-3:                  175,000,000,000 parameters
that's 1,719,563 digit readers

GPT-3 (OpenAI, 2020) has 175 billion parameters, 96 layers and 12,288 numbers per token inside (from its paper, Language Models are Few-Shot Learners).

Will it fit? Bytes, gigabytes and quantization

Every parameter is a number; a normal 32-bit number takes 4 bytes.

params = 175_000_000_000          # GPT-3: 175 billion parameters
for name, size in [("32-bit", 4), ("16-bit", 2), ("8-bit", 1)]:
    print(f"{name} numbers: {params * size / 1e9:,.0f} GB")
print("one H100 GPU holds 80 GB")
weights = [0.0] * params          # try to hold them all in Python

Output:

32-bit numbers: 700 GB
16-bit numbers: 350 GB
8-bit numbers: 175 GB
one H100 GPU holds 80 GB
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    weights = [0.0] * params          # try to hold them all in Python
              ~~~~~~^~~~~~~~
MemoryError

Even at 16 bits, 350 GB is more than four times what one NVIDIA H100 data-centre GPU holds (80 GB, from NVIDIA's H100 PCIe datasheet), so big models are split across several GPUs. Storing weights with fewer bits (16, 8, even 4) is called quantization: like rounding prices to the nearest rupee, you lose a little accuracy and save a lot of space. It's how big models run on smaller machines.

Counting the work: FLOPs and compute

  • A FLOP is one floating-point operation (one + or × on decimal numbers). Compute is the total FLOPs a job needs.
  • Reading one token, each parameter is used about once: one multiply + one add = 2 FLOPs per parameter.
  • Inference (using a trained model) ≈ 2 × parameters FLOPs per token.
  • Training also runs backwards (backpropagation, about twice the forward cost): ≈ 6 × parameters × training tokens. The Chinchilla paper uses this rule of thumb (FLOPs ≈ 6ND).

Demo: could you train GPT-3 in Python?

import time
n = 5_000_000
start = time.perf_counter()
total = 0.0
for i in range(n):
    total += 0.5 * 1.5            # one multiply + one add = 2 operations
seconds = time.perf_counter() - start
speed = 2 * n / seconds
print(f"Python on this machine: {speed / 1e6:.0f} million operations per second")

flops = 6 * 175e9 * 300e9         # training: about 6 x parameters x tokens
print(f"GPT-3 training: about {flops:.2e} operations")
year = 365 * 24 * 3600
print(f"in plain Python: {flops / speed / year:.1e} years")
print(f"one H100 GPU at 51 trillion per second: {flops / 51e12 / year:.0f} years")

Output (the first and third lines depend on your machine):

Python on this machine: 42 million operations per second
GPT-3 training: about 3.15e+23 operations
in plain Python: 2.4e+08 years
one H100 GPU at 51 trillion per second: 196 years

GPT-3 was trained on 300 billion tokens (from its paper). 6 × 175 billion × 300 billion ≈ 3 × 10²³ operations: hundreds of millions of years in plain Python, and about 200 years on one H100 at its plain 32-bit rate of 51 trillion operations per second (datasheet figure). Real training uses faster number formats and thousands of GPUs at once.

Why GPUs: one chef vs a thousand helpers

A wedding kitchen AI hardware
One master chef: can cook anything, one roti at a time CPU: a few powerful cores, step by step
A thousand helpers: each only rolls rotis, all at once GPU: thousands of simple cores in parallel
Many kitchens for a feast of a lakh guests GPU clusters: thousands of GPUs training one model

A neural network is mostly multiply-adds that don't depend on each other: perfect for parallel helpers. Doing many operations at the same time is parallelism.

Scaling laws and Chinchilla

  • Scaling laws (Jared Kaplan et al., OpenAI, 2020): a language model's loss falls as a power law with model size, dataset size and compute. Predictable, so you can train small models cheaply and forecast the big one.
  • Chinchilla (Hoffmann et al., DeepMind, 2022): for a fixed compute budget, model size and training tokens should be scaled equally.
models = [("GPT-3", 175e9, 300e9),        # (name, parameters, training tokens)
          ("Gopher", 280e9, 300e9),
          ("Chinchilla", 70e9, 1.4e12)]
for name, n, d in models:
    print(f"{name:10} {n / 1e9:3.0f}B params {d / 1e9:5.0f}B tokens "
          f"{d / n:4.1f} per param")

Output:

GPT-3      175B params   300B tokens  1.7 per param
Gopher     280B params   300B tokens  1.1 per param
Chinchilla  70B params  1400B tokens 20.0 per param

Using the 6ND rule of thumb, that's about 3.2 × 10²³ FLOPs for GPT-3, 5.0 × 10²³ for Gopher and 5.9 × 10²³ for Chinchilla. Chinchilla used about the same compute budget as Gopher (the paper says they were trained for the same number of FLOPs; the 6ND estimate is only a rough rule of thumb), but was 4× smaller and saw about 4× more data. It outperformed Gopher, GPT-3 and other larger models across a wide range of tasks. The lesson: feed the model enough data; many big models of the time were under-trained.

Training vs inference

Training Inference
When once (weeks on a GPU cluster) every question, every generated token
Cost ≈ 6 × parameters × tokens ≈ 2 × parameters per token
Why it matters a one-time bill multiplied by millions of users, forever

A smaller model trained on more data is cheaper to run for every user, which is another reason the Chinchilla lesson stuck.

Glossary

Term Meaning Example
Parameter a number learned by training a weight, a bias
Hyperparameter a setting chosen before training that controls it learning rate, epochs, k
Learning rate how big each gradient descent step is 0.05 works, 0.2 explodes
Epoch one full pass over the training data "train for 10 epochs"
Batch size how many examples per gradient step 32
Validation set data kept aside to choose hyperparameters the 5 valid days
Test set data used once at the end for an honest score locked until the end
Hyperparameter tuning choosing hyperparameters by validation score picking k = 3
Grid search trying every value from a list for k in [1, 3, 5, 10]
k-nearest neighbours predict by copying the k most similar examples predict(temp, k, days)
Mean absolute error average size of the mistakes "2.3 cups off"
Bit / byte the smallest unit of memory / 8 bits a 32-bit number = 4 bytes
GB (gigabyte) a billion bytes GPT-3 at 32-bit: 700 GB
Quantization storing weights with fewer bits 16-bit, 8-bit, 4-bit
MemoryError Python can't get enough memory [0.0] * 175_000_000_000
OverflowError a number got too big to represent an exploding learning rate
FLOP one floating-point operation (+ or ×) 0.5 * 1.5 is one
Compute total FLOPs a job needs ≈ 3 × 10²³ for GPT-3's training
Inference using a trained model ≈ 2 × parameters per token
Token a piece of text a model reads (episode 12) "chai" → "ch" + "ai"
CPU a few powerful cores, step by step your laptop's processor
GPU thousands of simple cores in parallel NVIDIA H100
Parallelism doing many operations at the same time a thousand helpers rolling rotis
GPU cluster many GPUs training one model together thousands of GPUs
Scaling laws loss falls predictably with size, data and compute Kaplan et al., 2020
Chinchilla 70B model, 1.4T tokens, beat larger models DeepMind, 2022
Tokens per parameter training tokens ÷ parameters Chinchilla: 20

Common mistakes

Mistake Code Error
Learning rate too big train(0.5, steps=200) OverflowError: (34, 'Numerical result out of range')
Holding a huge model in memory weights = [0.0] * 175_000_000_000 MemoryError
Picking hyperparameters by training error min(ks, key=lambda k: error(k, train)) no error; picks k = 1 (0.0) which is 7.8 off on new days
Tuning on the test set choosing k by test score no error; the final score is no longer honest

Fixes: try smaller learning rates (0.1, 0.01, 0.001); quantize or split across machines; tune on the validation set; touch the test set once.

Mini project: a model size calculator

Write a function that takes a list of layer sizes and prints the parameter count, the memory at 32-bit, and the FLOPs to process one input (2 × parameters).

def dense(n_in, n_out):
    return n_in * n_out + n_out

def network(layers):
    return sum(dense(a, b) for a, b in zip(layers, layers[1:]))

for layers in [[784, 128, 10], [784, 256, 128, 10], [784, 1024, 1024, 10]]:
    n = network(layers)
    print(f"{str(layers):22} {n:>10,} params  {n * 4 / 1e6:6.2f} MB (32-bit)  "
          f"{2 * n:>10,} FLOPs per image")
[784, 128, 10]            101,770 params    0.41 MB (32-bit)     203,540 FLOPs per image
[784, 256, 128, 10]       235,146 params    0.94 MB (32-bit)     470,292 FLOPs per image
[784, 1024, 1024, 10]   1,863,690 params    7.45 MB (32-bit)   3,727,380 FLOPs per image

Practice

  1. Find the largest learning rate that still lands on w = 3 in 20 steps.
  2. Add k = 2 and k = 4 (in fact, every k from 1 to 10) to the grid search. Which k wins?
  3. Size a 7-billion-parameter model at 32, 16, 8 and 4 bits, and estimate its training compute on 2 trillion tokens.
Solutions

1.

data = [(1, 3), (2, 6), (3, 9), (4, 12)]

def train(lr, steps=20):
    w = 0.0
    for _ in range(steps):
        grad = sum(2 * (w * x - y) * x for x, y in data) / len(data)
        w = w - lr * grad
    loss = sum((w * x - y) ** 2 for x, y in data) / len(data)
    return w, loss

for lr in [0.05, 0.1, 0.13, 0.14, 0.15]:
    w, loss = train(lr)
    print(f"learning rate {lr:<4}: w = {w:.3g}, loss = {loss:.3g}")
learning rate 0.05: w = 3, loss = 5.58e-23
learning rate 0.1 : w = 3, loss = 6.14e-11
learning rate 0.13: w = 1.92, loss = 8.67
learning rate 0.14: w = -17.2, loss = 3.05e+03
learning rate 0.15: w = -257, loss = 5.08e+05

0.1 still works; by 0.13 it's swinging back and forth and hasn't settled, and from 0.14 it explodes.

2.

train = [(14, 92), (16, 85), (18, 88), (20, 71), (22, 75), (24, 62),
         (26, 66), (28, 51), (30, 55), (32, 40)]
valid = [(15, 86), (19, 78), (23, 70), (27, 57), (31, 46)]

def predict(temp, k, days):
    nearest = sorted(days, key=lambda d: abs(d[0] - temp))[:k]
    return sum(cups for _, cups in nearest) / k

def error(k, check):
    return sum(abs(predict(t, k, train) - c) for t, c in check) / len(check)

best = min(range(1, 11), key=lambda k: error(k, valid))
for k in range(1, 11):
    print(f"k = {k:2}: validation error {error(k, valid):5.1f}")
print("best k:", best)
k =  1: validation error   7.8
k =  2: validation error   1.7
k =  3: validation error   2.3
k =  4: validation error   2.8
k =  5: validation error   4.8
k =  6: validation error   4.5
k =  7: validation error   5.6
k =  8: validation error   8.3
k =  9: validation error  10.2
k = 10: validation error  12.5
best k: 2

A finer grid finds k = 2. With only five validation days, though, the gap between 2 and 3 is small; with more data you'd trust it more.

3.

params = 7_000_000_000
for name, bits in [("32-bit", 32), ("16-bit", 16), ("8-bit", 8), ("4-bit", 4)]:
    print(f"{name}: {params * bits / 8 / 1e9:.1f} GB")
flops = 6 * 7e9 * 2e12
print(f"training compute: {flops:.1e} FLOPs")
print(f"one H100 at 51 trillion/s: {flops / 51e12 / (365 * 24 * 3600):.1f} years")
print(f"1,000 H100s: {flops / (1000 * 51e12) / (24 * 3600):.0f} days")
32-bit: 28.0 GB
16-bit: 14.0 GB
8-bit: 7.0 GB
4-bit: 3.5 GB
training compute: 8.4e+22 FLOPs
one H100 at 51 trillion/s: 52.2 years
1,000 H100s: 19 days

At 4 bits, a 7B model fits on a laptop-class GPU. Training is still a cluster job.

Quick quiz 1

Which of these is a parameter (learned by training)? A: the learning rate · B: a weight between two neurons · C: the number of layers · D: the number of epochs

Answer

B: a weight. The others are hyperparameters you choose.

Quick quiz 2

A dense layer has 300 inputs and 100 neurons, with one bias per neuron. How many parameters? A: 400 · B: 30,000 · C: 30,100 · D: 30,300

Answer

C: 30,100. 300 × 100 weights + 100 biases.

Interview questions

  1. What's the difference between a parameter and a hyperparameter? Parameters (weights, biases) are learned by training; hyperparameters (learning rate, epochs, batch size, architecture) are set before training and control it.
  2. Why not choose hyperparameters by training loss? Some settings let the model memorise (k = 1, huge models, many epochs), which training loss rewards. Use a validation set; keep the test set for the final score.
  3. What happens if the learning rate is too high or too low? Too low: slow progress. Too high: steps overshoot and the loss oscillates or explodes (NaN or overflow).
  4. Roughly how much memory do a model's weights need? Parameters × bytes per parameter: 7B at 16-bit ≈ 14 GB (plus extra for activations and, in training, gradients and optimizer state).
  5. Estimate training compute for a model. ≈ 6 × parameters × training tokens FLOPs; inference ≈ 2 × parameters per token.
  6. What did Chinchilla show? For a fixed compute budget, model size and training tokens should grow together (about 20 tokens per parameter for Chinchilla); a smaller, better-fed model beat larger ones.

Summary

  • Parameters are learned; hyperparameters are chosen before training.
  • The learning rate can crawl (0.001) or explode (0.2); training can't pick it.
  • Training error lies about hyperparameters (k = 1 scored 0.0); tune on the validation set with grid search.
  • GPT-3: 175 billion parameters, 700 GB at 32-bit, so models are split across GPUs or quantized.
  • Training ≈ 6 × parameters × tokens FLOPs: ≈ 3 × 10²³ for GPT-3, hence thousands of GPUs in parallel.
  • Scaling laws: loss falls predictably with size, data, compute. Chinchilla: grow data with size.
  • Next episode: text to numbers, tokens.