Machine Learning Full Course · Episode 12. Every code sample below was run with Python 3.13, NumPy 2.5.3 and SciPy 1.18.1; the output shown is the real output. All the data (coin tosses, heights, weather frequencies) is made up for teaching.
The big question: in Episode 8 we squared the errors "because it worked". Today we see that squared error, log loss and cross-entropy all come from two ideas: likelihood and information.
The magician's coin: likelihood
A street magician claims his coin is fair. Ten tosses: HHTHHHTHTH, 7 heads and 3 tails. What is the coin's chance of heads, p? This is exactly what training a model is: find the parameter that best explains the data.
Idea: for each guess of p, compute how probable these exact tosses would be. The tosses are independent, so multiply: p for each head, (1 − p) for each tail.
tosses = "HHTHHHTHTH" # the magician's 10 tosses
h, t = tosses.count("H"), tosses.count("T")
print("heads:", h, " tails:", t)
def likelihood(p): # P(these exact tosses | P(heads) = p)
return p**h * (1 - p)**t
for p in [0.3, 0.5, 0.6, 0.7, 0.8, 0.9]:
print(f"p = {p}: L = {likelihood(p):.6f}")
grid = [i / 100 for i in range(1, 100)]
print("best p:", max(grid, key=likelihood))
Output:
heads: 7 tails: 3
p = 0.3: L = 0.000075
p = 0.5: L = 0.000977
p = 0.6: L = 0.001792
p = 0.7: L = 0.002224
p = 0.8: L = 0.001678
p = 0.9: L = 0.000478
best p: 0.7
- Likelihood L(p) = P(data | p): how probable the observed data is under a parameter value. The data is fixed; we score the guesses. It is not P(p | data), and likelihoods over p don't need to add up to 1.
- Maximum likelihood estimation (MLE): choose the parameter with the highest likelihood. For a coin, the MLE is simply heads ÷ tosses = 7/10.
max(grid, key=likelihood)returns the grid value with the largest likelihood.
5,000 tosses: the obvious way fails (underflow)
import numpy as np
rng = np.random.default_rng(0)
tosses = rng.random(5000) < 0.7 # 5,000 tosses of a p = 0.7 coin
grid = np.linspace(0.01, 0.99, 99)
def likelihood(p): # multiply 5,000 probabilities
return np.prod(np.where(tosses, p, 1 - p))
L = [likelihood(p) for p in grid]
print("L at p = 0.5, 0.7:", likelihood(0.5), likelihood(0.7))
print("best p by likelihood:", grid[np.argmax(L)])
Output:
L at p = 0.5, 0.7: 0.0 0.0
best p by likelihood: 0.01
The likelihood is 0.0 for every p, and the "best" p comes out as 0.01, totally wrong.
Why: multiplying 5,000 numbers below 1 gives a number smaller than the smallest positive float a computer can store (about 10⁻³⁰⁸ for normal 64-bit floats), so it rounds to 0. This is underflow. With every value 0, np.argmax returns the first index.
np.where(tosses, p, 1 - p)gives p for each head and 1 − p for each tail.np.prodmultiplies everything in an array.
The fix: log-likelihood and negative log-likelihood
- log(a × b) = log a + log b: the logarithm turns products into sums, and tiny numbers into ordinary negative numbers (ln of one in a million ≈ −13.8).
- Log-likelihood: log L(p) = Σ log P(each data point | p). No underflow.
- The log always increases with its input, so the p that maximises L also maximises log L: same answer, safe arithmetic.
- Optimisers minimise (gradient descent, Episode 8), so flip the sign: the negative log-likelihood (NLL) = −log L is a loss (lower is better).
import numpy as np
rng = np.random.default_rng(0)
tosses = rng.random(5000) < 0.7
grid = np.linspace(0.01, 0.99, 99)
def log_likelihood(p): # add logs instead of multiplying
return np.sum(np.log(np.where(tosses, p, 1 - p)))
LL = [log_likelihood(p) for p in grid]
print("log L at p = 0.5:", round(log_likelihood(0.5), 1))
print("log L at p = 0.7:", round(log_likelihood(0.7), 1))
print("best p by log-likelihood:", round(grid[np.argmax(LL)], 2))
print("fraction of heads:", tosses.mean())
print("NLL per toss at the best p:", round(-max(LL) / 5000, 4))
Output:
log L at p = 0.5: -3465.7
log L at p = 0.7: -3060.3
best p by log-likelihood: 0.7
fraction of heads: 0.6986
NLL per toss at the best p: 0.6121
np.log is the natural logarithm (base e). Remember 0.6121 per toss: it reappears below as the log loss.
Mean squared error is a likelihood
Assume heights come from a normal distribution with unknown mean μ (and σ = 7). Minimise the normal NLL, and separately minimise the MSE:
import numpy as np
rng = np.random.default_rng(3)
heights = rng.normal(165, 7, 200)
def nll(mu, sigma=7): # negative log-likelihood of a normal
z = (heights - mu)**2 / (2 * sigma**2)
return np.sum(z + np.log(sigma * np.sqrt(2 * np.pi)))
def mse(mu):
return np.mean((heights - mu)**2)
grid = np.arange(160, 170, 0.01)
print("best mu by NLL:", round(grid[np.argmin([nll(m) for m in grid])], 2))
print("best mu by MSE:", round(grid[np.argmin([mse(m) for m in grid])], 2))
print("sample mean: ", round(heights.mean(), 2))
Output:
best mu by NLL: 165.34
best mu by MSE: 165.34
sample mean: 165.34
The normal NLL is Σ (x − μ)² / (2σ²) + a constant. With σ fixed, minimising it is minimising squared error, and both give the sample mean. Mean squared error = maximum likelihood under normal (Gaussian) noise.
Too little data: MLE overfits
import numpy as np
tosses = "HHH" # only 3 tosses so far
p_mle = tosses.count("H") / len(tosses)
print("MLE p(heads):", p_mle)
nxt = "T" # the 4th toss lands tails
p_next = p_mle if nxt == "H" else 1 - p_mle
print("probability the model gave it:", p_next)
print("log-likelihood:", np.log(p_next))
Output:
MLE p(heads): 1.0
probability the model gave it: 0.0
log-likelihood: -inf
main.py:10: RuntimeWarning: divide by zero encountered in log
After 3 heads, MLE says p = 1.0: tails is impossible. The 4th toss is tails, the model gave it probability 0, and log(0) = −∞ with RuntimeWarning: divide by zero encountered in log.
Trusting a few examples completely and failing on the next one is overfitting (Episode 20).
MAP: the prior as regularisation
You know most coins are close to fair: a prior belief (Episode 9).
- Bayes' theorem: posterior ∝ likelihood × prior.
- Maximum a posteriori (MAP) estimation: pick the parameter with the highest posterior. ("A posteriori" is Latin for "after the evidence".)
- In logs: minimise NLL − log prior = data loss + a penalty for unlikely parameter values.
- Adding a penalty to a loss to prevent overfitting is regularisation. So a prior is a regulariser: a normal prior on weights gives L2 (ridge) regularisation, a Laplace prior gives L1 (lasso) (Episode 21).
A simple coin prior (a Beta(11, 11) distribution) acts like 10 extra imaginary heads and 10 extra tails: MAP p = (heads + 10) / (n + 20).
def p_map(heads, n, a=11, b=11): # prior: like 10 extra heads + 10 tails
return (heads + a - 1) / (n + a + b - 2)
for heads, n in [(3, 3), (7, 10), (70, 100), (7000, 10000)]:
print(f"{heads:5}/{n:<5} MLE {heads / n:.3f} MAP {p_map(heads, n):.3f}")
Output:
3/3 MLE 1.000 MAP 0.565
7/10 MLE 0.700 MAP 0.567
70/100 MLE 0.700 MAP 0.667
7000/10000 MLE 0.700 MAP 0.700
With 3 tosses, MAP stays sensible (0.565). With 10,000 tosses, the prior fades and MAP = MLE = 0.700.
Entropy: weather by text message
A friend in the hills texts the weather every day. Sunny ½ of the days, cloudy ¼, rain ⅛, snow ⅛. Each bit (a 0 or 1) costs money. A fixed code needs 2 bits per day. A smarter code gives common weather shorter codes. No code is the start of another (a prefix-free code), so messages decode without separators.
import numpy as np
weather = ["sunny", "cloudy", "rain", "snow"]
p = np.array([0.5, 0.25, 0.125, 0.125]) # how often each happens
fixed = {"sunny": "00", "cloudy": "01", "rain": "10", "snow": "11"}
smart = {"sunny": "0", "cloudy": "10", "rain": "110", "snow": "111"}
for name, code in [("fixed", fixed), ("smart", smart)]:
avg = sum(pi * len(code[w]) for pi, w in zip(p, weather))
print(name, "code: average bits per day =", avg)
entropy = -np.sum(p * np.log2(p))
print("entropy:", entropy)
coin99 = -(0.99 * np.log2(0.99) + 0.01 * np.log2(0.01))
print("fair coin:", -2 * 0.5 * np.log2(0.5), " 99% coin:", round(coin99, 3))
Output:
fixed code: average bits per day = 2.0
smart code: average bits per day = 1.75
entropy: 1.75
fair coin: 1.0 99% coin: 0.081
- The surprise (information content) of an event with probability p is −log₂ p bits: rare events are more surprising (sunny: 1 bit; snow: 3 bits).
- Entropy H(p) = −Σ p log₂ p: the average surprise, and the minimum possible average code length. Here 1.75 bits, exactly the smart code.
- A fair coin has 1 bit of entropy (maximum uncertainty for two outcomes); a 99% coin only 0.081 bits.
Cross-entropy and KL divergence
What if your code was built for another town's weather, q?
import numpy as np
p = np.array([0.5, 0.25, 0.125, 0.125]) # real weather in the hills
q = np.array([0.125, 0.125, 0.25, 0.5]) # a code built for another town
def entropy(p):
return -np.sum(p * np.log2(p))
def cross_entropy(p, q): # average bits: q's code, p's weather
return -np.sum(p * np.log2(q))
def kl(p, q): # the extra bits wasted
return cross_entropy(p, q) - entropy(p)
print("entropy H(p): ", entropy(p))
print("cross-entropy H(p, q):", cross_entropy(p, q))
print("KL(p || q):", kl(p, q), " KL(p || p):", kl(p, p))
r = np.array([0.7, 0.1, 0.1, 0.1])
print("KL(p || r):", round(kl(p, r), 3), " KL(r || p):", round(kl(r, p), 3))
Output:
entropy H(p): 1.75
cross-entropy H(p, q): 2.625
KL(p || q): 0.875 KL(p || p): 0.0
KL(p || r): 0.168 KL(r || p): 0.143
| Quantity | Formula | Meaning | Here |
|---|---|---|---|
| Entropy H(p) | −Σ p log p | best average bits when you know p | 1.75 |
| Cross-entropy H(p, q) | −Σ p log q | average bits using q's code on p's data | 2.625 |
| KL divergence KL(p ‖ q) | H(p, q) − H(p) = Σ p log(p/q) | extra bits wasted by believing q | 0.875 |
- KL (Kullback–Leibler) divergence is never negative and is 0 only when q = p.
- It's not symmetric: KL(p ‖ r) = 0.168 but KL(r ‖ p) = 0.143, so it's a "divergence", not a distance.
scipy.stats.entropy(p, q, base=2)computes KL(p ‖ q) directly.
Why these are the loss functions of ML
| Weather code | Classifier |
|---|---|
| true frequencies p | true label as a one-hot vector (1 for the true class, 0 elsewhere): cat = [1, 0, 0] |
| the code you built, q | the model's predicted probabilities, e.g. [0.7, 0.2, 0.1] |
| bits you pay, H(p, q) | cross-entropy loss |
With a one-hot truth, cross-entropy = −log q[true class] = the NLL of the true label. Minimising cross-entropy = minimising KL (H(p) is fixed) = maximum likelihood.
import numpy as np
truth = np.array([1, 0, 0]) # the photo is a cat (one-hot)
for q in [[0.7, 0.2, 0.1], [0.98, 0.01, 0.01], [0.01, 0.98, 0.01]]:
q = np.array(q) # model's guess: cat, dog, bird
ce = -np.sum(truth * np.log(q)) # cross-entropy with a one-hot truth
print(q, " loss =", round(ce, 3), " -log(q[cat]) =", round(-np.log(q[0]), 3))
def bce(y, p): # binary cross-entropy (log loss)
return -(y * np.log(p) + (1 - y) * np.log(1 - p))
print("spam, model says 0.9:", round(bce(1, 0.9), 3))
print("spam, model says 0.1:", round(bce(1, 0.1), 3))
Output:
[0.7 0.2 0.1] loss = 0.357 -log(q[cat]) = 0.357
[0.98 0.01 0.01] loss = 0.02 -log(q[cat]) = 0.02
[0.01 0.98 0.01] loss = 4.605 -log(q[cat]) = 4.605
spam, model says 0.9: 0.105
spam, model says 0.1: 2.303
- Confident and right (0.98): loss 0.02. Confident and wrong (0.98 dog for a cat): loss 4.6.
- Binary cross-entropy (log loss) for yes/no labels y ∈ {0, 1}: −[y log p + (1 − y) log(1 − p)]. It is the Bernoulli NLL: the coin again. The 0.6121 per toss above was the log loss of predicting p = 0.7 for each toss.
| Loss | Likelihood it comes from | Used for |
|---|---|---|
| Mean squared error | Normal (Gaussian) noise | regression |
| Binary cross-entropy (log loss) | Bernoulli | binary classification |
| Categorical cross-entropy | Categorical (one of K classes) | multiclass classification |
Common mistakes
| Mistake | What happens | Why | Fix |
|---|---|---|---|
np.prod of thousands of probabilities |
0.0, best p = 0.01 |
underflow | sum the logs |
| predicting probability exactly 0 or 1 | RuntimeWarning: divide by zero encountered in log, loss = -inf / inf |
log(0) | clip probabilities (e.g. to [1e-15, 1 − 1e-15]) or use a prior |
| "likelihood = P(p | data)" | wrong meaning | it's P(data | p) | use Bayes for P(p | data) |
| treating KL as a distance | KL(p‖r) ≠ KL(r‖p) | not symmetric | order matters: KL(truth ‖ model) |
| mixing log₂ and ln | numbers differ by a factor 0.693 | bits vs nats | log₂ → bits; ln → nats; the best parameters are the same |
| MLE on a handful of examples | extreme estimates (p = 1.0) | overfitting | more data or a prior (MAP) |
Mini project: fit a Poisson rate by maximum likelihood
Customer counts in 50 ten-minute windows (Episode 10's Poisson). Find λ by minimising the NLL with scipy.stats.poisson.logpmf (the log of the PMF):
{{PY:/tmp/claude-0/s/m12.py}}
The MLE of a Poisson rate is the sample mean, exactly as it was for the normal mean and the coin's fraction of heads.
Practice
- MLE of p for 12 heads in 40 tosses; and the MAP with the 10 + 10 prior.
- MAP for 0 heads in 5 tosses with the 10 + 10 prior.
Solutions 1 and 2
{{PY:/tmp/claude-0/s/h12a.py}}
The MLE for 0/5 would be 0.0 (heads "impossible"); MAP keeps it at 0.4.
- Entropy of a fair 6-sided die in bits.
- Log loss when the model always says 0.5.
Solutions 3 and 4
{{PY:/tmp/claude-0/s/h12b.py}}
A model that always says 0.5 has log loss ln 2 ≈ 0.693 per example: any useful binary classifier must beat that.
Quick quiz
import numpy as np
p = np.array([0.5, 0.5])
q = np.array([0.25, 0.75])
H = -np.sum(p * np.log2(p))
X = -np.sum(p * np.log2(q))
print(H, round(X, 3), round(X - H, 3))
A) 1.0 1.208 0.208 B) 1.0 1.0 0.0 C) 0.5 1.208 0.708 D) 1.0 0.811 -0.189
Answer
A) 1.0 1.208 0.208. H(p) = 1 bit (a fair coin). X = −(0.5 log₂ 0.25 + 0.5 log₂ 0.75) = 1 + 0.208 = 1.208. KL = X − H = 0.208; KL is never negative, which rules out D.
Interview questions
- What is the likelihood, and how is it different from a probability? L(θ) = P(data | θ), a function of the parameter with the data fixed; it doesn't sum to 1 over θ and isn't P(θ | data).
- Why do we maximise the log-likelihood instead of the likelihood? Products of many small probabilities underflow; logs turn them into sums, are numerically stable, give the same maximiser, and make gradients easier.
- Show that MSE corresponds to MLE. With Gaussian noise of fixed σ, NLL = Σ(y − ŷ)²/(2σ²) + constant, so minimising NLL = minimising squared error.
- MLE vs MAP? MAP maximises likelihood × prior; in log form it adds a penalty term, i.e. regularisation. With lots of data they agree.
- What prior corresponds to L2 regularisation? A zero-mean normal (Gaussian) prior on the weights; a Laplace prior gives L1.
- Define entropy, cross-entropy and KL divergence. H(p) = −Σ p log p; H(p, q) = −Σ p log q; KL(p‖q) = H(p, q) − H(p) ≥ 0.
- Why is cross-entropy the standard classification loss? It's the NLL of a categorical model; minimising it is maximum likelihood and minimises KL to the true labels, and it punishes confident wrong answers heavily.
- Is KL divergence a distance? No: it isn't symmetric and doesn't satisfy the triangle inequality.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Parameter | A number a model learns from data | p of a coin |
| Likelihood L(θ) | P(observed data | parameter θ) | L(0.7) = 0.002224 |
| Maximum likelihood estimation (MLE) | Pick θ with the highest likelihood | p = 7/10 |
| Underflow | A result too small to store, rounded to 0 | product of 5,000 probabilities |
| Logarithm (log, ln, log₂) | Inverse of powers; log(ab) = log a + log b | ln(10⁻⁶) ≈ −13.8 |
| Natural logarithm | Log base e (≈ 2.718); np.log |
ln 2 = 0.693 |
| Log-likelihood | Sum of the logs of the data probabilities | −3060.3 |
| Negative log-likelihood (NLL) | −log-likelihood; a loss to minimise | 0.6121 per toss |
| Loss function | Number measuring how wrong a model is | MSE, log loss |
| Overfitting | Fitting the few examples too closely | p = 1.0 after 3 heads |
| Prior | Belief about a parameter before the data | coins are near fair |
| Posterior | Belief after the data; ∝ likelihood × prior | Episode 9 |
| MAP (maximum a posteriori) | Pick θ with the highest posterior | 0.565 |
| Beta distribution | A distribution over probabilities (0 to 1), used as a prior for p | Beta(11, 11) |
| Regularisation | A penalty added to the loss to prevent overfitting | the prior term |
| Bit | A binary digit, 0 or 1; unit of information with log₂ | 2 bits per day |
| Nat | Unit of information with the natural log | ML libraries use nats |
| Prefix-free code | No codeword starts another | 0, 10, 110, 111 |
| Surprise (information content) | −log₂ p | snow: 3 bits |
| Entropy H(p) | Average surprise; best average code length | 1.75 bits |
| Cross-entropy H(p, q) | Average bits using q's code on data from p | 2.625 bits |
| KL divergence KL(p ‖ q) | H(p, q) − H(p): extra bits from believing q | 0.875 bits |
| One-hot vector | 1 for the true class, 0 elsewhere | cat = [1, 0, 0] |
| Cross-entropy loss | −log q[true class] for a classifier | 0.357 |
| Binary cross-entropy (log loss) | −[y log p + (1 − y) log(1 − p)] | 0.105 vs 2.303 |
| Clipping | Keeping values inside a range | probabilities in [1e-15, 1 − 1e-15] |
Summary
- Likelihood scores parameter guesses by how probable they make the data; MLE picks the best one.
- Multiplying many probabilities underflows; use the log-likelihood, and minimise the NLL as a loss.
- MSE is the NLL of normal noise; log loss is the NLL of a Bernoulli; cross-entropy is the NLL of a categorical model.
- MAP = likelihood × prior; the prior acts as regularisation and fades as data grows.
- Entropy = average surprise in bits; cross-entropy = cost of using the wrong distribution; KL = the extra cost, never negative, not symmetric.
Next episode: Exploratory data analysis: feature types, describe, value counts, histograms, box plots, scatter and pair plots, correlation heatmaps, target distribution and a first look at outliers.