SyntaxPilot
Courses/Machine Learning/Lesson 12

Machine Learning · Lesson 12 of 90 · 13 min

Maths 10: likelihood and information theory

likelihoodmaximum likelihood estimation (MLE)log-likelihoodnegative log-likelihoodmaximum a posteriori (MAP)prior as regularisationentropycross-entropyKL divergencewhy these become loss functions

Machine Learning Full Course · Episode 12. Every code sample below was run with Python 3.13, NumPy 2.5.3 and SciPy 1.18.1; the output shown is the real output. All the data (coin tosses, heights, weather frequencies) is made up for teaching.

The big question: in Episode 8 we squared the errors "because it worked". Today we see that squared error, log loss and cross-entropy all come from two ideas: likelihood and information.

The magician's coin: likelihood

A street magician claims his coin is fair. Ten tosses: HHTHHHTHTH, 7 heads and 3 tails. What is the coin's chance of heads, p? This is exactly what training a model is: find the parameter that best explains the data.

Idea: for each guess of p, compute how probable these exact tosses would be. The tosses are independent, so multiply: p for each head, (1 − p) for each tail.

tosses = "HHTHHHTHTH"                        # the magician's 10 tosses
h, t = tosses.count("H"), tosses.count("T")
print("heads:", h, " tails:", t)

def likelihood(p):                  # P(these exact tosses | P(heads) = p)
    return p**h * (1 - p)**t

for p in [0.3, 0.5, 0.6, 0.7, 0.8, 0.9]:
    print(f"p = {p}:  L = {likelihood(p):.6f}")

grid = [i / 100 for i in range(1, 100)]
print("best p:", max(grid, key=likelihood))

Output:

heads: 7  tails: 3
p = 0.3:  L = 0.000075
p = 0.5:  L = 0.000977
p = 0.6:  L = 0.001792
p = 0.7:  L = 0.002224
p = 0.8:  L = 0.001678
p = 0.9:  L = 0.000478
best p: 0.7
  • Likelihood L(p) = P(data | p): how probable the observed data is under a parameter value. The data is fixed; we score the guesses. It is not P(p | data), and likelihoods over p don't need to add up to 1.
  • Maximum likelihood estimation (MLE): choose the parameter with the highest likelihood. For a coin, the MLE is simply heads ÷ tosses = 7/10.
  • max(grid, key=likelihood) returns the grid value with the largest likelihood.

5,000 tosses: the obvious way fails (underflow)

import numpy as np

rng = np.random.default_rng(0)
tosses = rng.random(5000) < 0.7                 # 5,000 tosses of a p = 0.7 coin
grid = np.linspace(0.01, 0.99, 99)

def likelihood(p):                              # multiply 5,000 probabilities
    return np.prod(np.where(tosses, p, 1 - p))

L = [likelihood(p) for p in grid]
print("L at p = 0.5, 0.7:", likelihood(0.5), likelihood(0.7))
print("best p by likelihood:", grid[np.argmax(L)])

Output:

L at p = 0.5, 0.7: 0.0 0.0
best p by likelihood: 0.01

The likelihood is 0.0 for every p, and the "best" p comes out as 0.01, totally wrong.

Why: multiplying 5,000 numbers below 1 gives a number smaller than the smallest positive float a computer can store (about 10⁻³⁰⁸ for normal 64-bit floats), so it rounds to 0. This is underflow. With every value 0, np.argmax returns the first index.

  • np.where(tosses, p, 1 - p) gives p for each head and 1 − p for each tail.
  • np.prod multiplies everything in an array.

The fix: log-likelihood and negative log-likelihood

  • log(a × b) = log a + log b: the logarithm turns products into sums, and tiny numbers into ordinary negative numbers (ln of one in a million ≈ −13.8).
  • Log-likelihood: log L(p) = Σ log P(each data point | p). No underflow.
  • The log always increases with its input, so the p that maximises L also maximises log L: same answer, safe arithmetic.
  • Optimisers minimise (gradient descent, Episode 8), so flip the sign: the negative log-likelihood (NLL) = −log L is a loss (lower is better).
import numpy as np

rng = np.random.default_rng(0)
tosses = rng.random(5000) < 0.7
grid = np.linspace(0.01, 0.99, 99)

def log_likelihood(p):                          # add logs instead of multiplying
    return np.sum(np.log(np.where(tosses, p, 1 - p)))

LL = [log_likelihood(p) for p in grid]
print("log L at p = 0.5:", round(log_likelihood(0.5), 1))
print("log L at p = 0.7:", round(log_likelihood(0.7), 1))
print("best p by log-likelihood:", round(grid[np.argmax(LL)], 2))
print("fraction of heads:", tosses.mean())
print("NLL per toss at the best p:", round(-max(LL) / 5000, 4))

Output:

log L at p = 0.5: -3465.7
log L at p = 0.7: -3060.3
best p by log-likelihood: 0.7
fraction of heads: 0.6986
NLL per toss at the best p: 0.6121

np.log is the natural logarithm (base e). Remember 0.6121 per toss: it reappears below as the log loss.

Mean squared error is a likelihood

Assume heights come from a normal distribution with unknown mean μ (and σ = 7). Minimise the normal NLL, and separately minimise the MSE:

import numpy as np

rng = np.random.default_rng(3)
heights = rng.normal(165, 7, 200)

def nll(mu, sigma=7):                     # negative log-likelihood of a normal
    z = (heights - mu)**2 / (2 * sigma**2)
    return np.sum(z + np.log(sigma * np.sqrt(2 * np.pi)))

def mse(mu):
    return np.mean((heights - mu)**2)

grid = np.arange(160, 170, 0.01)
print("best mu by NLL:", round(grid[np.argmin([nll(m) for m in grid])], 2))
print("best mu by MSE:", round(grid[np.argmin([mse(m) for m in grid])], 2))
print("sample mean:   ", round(heights.mean(), 2))

Output:

best mu by NLL: 165.34
best mu by MSE: 165.34
sample mean:    165.34

The normal NLL is Σ (x − μ)² / (2σ²) + a constant. With σ fixed, minimising it is minimising squared error, and both give the sample mean. Mean squared error = maximum likelihood under normal (Gaussian) noise.

Too little data: MLE overfits

import numpy as np

tosses = "HHH"                          # only 3 tosses so far
p_mle = tosses.count("H") / len(tosses)
print("MLE p(heads):", p_mle)

nxt = "T"                               # the 4th toss lands tails
p_next = p_mle if nxt == "H" else 1 - p_mle
print("probability the model gave it:", p_next)
print("log-likelihood:", np.log(p_next))

Output:

MLE p(heads): 1.0
probability the model gave it: 0.0
log-likelihood: -inf
main.py:10: RuntimeWarning: divide by zero encountered in log

After 3 heads, MLE says p = 1.0: tails is impossible. The 4th toss is tails, the model gave it probability 0, and log(0) = −∞ with RuntimeWarning: divide by zero encountered in log.

Trusting a few examples completely and failing on the next one is overfitting (Episode 20).

MAP: the prior as regularisation

You know most coins are close to fair: a prior belief (Episode 9).

  • Bayes' theorem: posterior ∝ likelihood × prior.
  • Maximum a posteriori (MAP) estimation: pick the parameter with the highest posterior. ("A posteriori" is Latin for "after the evidence".)
  • In logs: minimise NLL − log prior = data loss + a penalty for unlikely parameter values.
  • Adding a penalty to a loss to prevent overfitting is regularisation. So a prior is a regulariser: a normal prior on weights gives L2 (ridge) regularisation, a Laplace prior gives L1 (lasso) (Episode 21).

A simple coin prior (a Beta(11, 11) distribution) acts like 10 extra imaginary heads and 10 extra tails: MAP p = (heads + 10) / (n + 20).

def p_map(heads, n, a=11, b=11):     # prior: like 10 extra heads + 10 tails
    return (heads + a - 1) / (n + a + b - 2)

for heads, n in [(3, 3), (7, 10), (70, 100), (7000, 10000)]:
    print(f"{heads:5}/{n:<5}  MLE {heads / n:.3f}   MAP {p_map(heads, n):.3f}")

Output:

    3/3      MLE 1.000   MAP 0.565
    7/10     MLE 0.700   MAP 0.567
   70/100    MLE 0.700   MAP 0.667
 7000/10000  MLE 0.700   MAP 0.700

With 3 tosses, MAP stays sensible (0.565). With 10,000 tosses, the prior fades and MAP = MLE = 0.700.

Entropy: weather by text message

A friend in the hills texts the weather every day. Sunny ½ of the days, cloudy ¼, rain ⅛, snow ⅛. Each bit (a 0 or 1) costs money. A fixed code needs 2 bits per day. A smarter code gives common weather shorter codes. No code is the start of another (a prefix-free code), so messages decode without separators.

import numpy as np

weather = ["sunny", "cloudy", "rain", "snow"]
p = np.array([0.5, 0.25, 0.125, 0.125])            # how often each happens

fixed = {"sunny": "00", "cloudy": "01", "rain": "10", "snow": "11"}
smart = {"sunny": "0", "cloudy": "10", "rain": "110", "snow": "111"}
for name, code in [("fixed", fixed), ("smart", smart)]:
    avg = sum(pi * len(code[w]) for pi, w in zip(p, weather))
    print(name, "code: average bits per day =", avg)

entropy = -np.sum(p * np.log2(p))
print("entropy:", entropy)
coin99 = -(0.99 * np.log2(0.99) + 0.01 * np.log2(0.01))
print("fair coin:", -2 * 0.5 * np.log2(0.5), " 99% coin:", round(coin99, 3))

Output:

fixed code: average bits per day = 2.0
smart code: average bits per day = 1.75
entropy: 1.75
fair coin: 1.0  99% coin: 0.081
  • The surprise (information content) of an event with probability p is −log₂ p bits: rare events are more surprising (sunny: 1 bit; snow: 3 bits).
  • Entropy H(p) = −Σ p log₂ p: the average surprise, and the minimum possible average code length. Here 1.75 bits, exactly the smart code.
  • A fair coin has 1 bit of entropy (maximum uncertainty for two outcomes); a 99% coin only 0.081 bits.

Cross-entropy and KL divergence

What if your code was built for another town's weather, q?

import numpy as np

p = np.array([0.5, 0.25, 0.125, 0.125])      # real weather in the hills
q = np.array([0.125, 0.125, 0.25, 0.5])      # a code built for another town

def entropy(p):
    return -np.sum(p * np.log2(p))

def cross_entropy(p, q):                     # average bits: q's code, p's weather
    return -np.sum(p * np.log2(q))

def kl(p, q):                                # the extra bits wasted
    return cross_entropy(p, q) - entropy(p)

print("entropy H(p):         ", entropy(p))
print("cross-entropy H(p, q):", cross_entropy(p, q))
print("KL(p || q):", kl(p, q), "  KL(p || p):", kl(p, p))
r = np.array([0.7, 0.1, 0.1, 0.1])
print("KL(p || r):", round(kl(p, r), 3), "  KL(r || p):", round(kl(r, p), 3))

Output:

entropy H(p):          1.75
cross-entropy H(p, q): 2.625
KL(p || q): 0.875   KL(p || p): 0.0
KL(p || r): 0.168   KL(r || p): 0.143
Quantity Formula Meaning Here
Entropy H(p) −Σ p log p best average bits when you know p 1.75
Cross-entropy H(p, q) −Σ p log q average bits using q's code on p's data 2.625
KL divergence KL(p ‖ q) H(p, q) − H(p) = Σ p log(p/q) extra bits wasted by believing q 0.875
  • KL (Kullback–Leibler) divergence is never negative and is 0 only when q = p.
  • It's not symmetric: KL(p ‖ r) = 0.168 but KL(r ‖ p) = 0.143, so it's a "divergence", not a distance.
  • scipy.stats.entropy(p, q, base=2) computes KL(p ‖ q) directly.

Why these are the loss functions of ML

Weather code Classifier
true frequencies p true label as a one-hot vector (1 for the true class, 0 elsewhere): cat = [1, 0, 0]
the code you built, q the model's predicted probabilities, e.g. [0.7, 0.2, 0.1]
bits you pay, H(p, q) cross-entropy loss

With a one-hot truth, cross-entropy = −log q[true class] = the NLL of the true label. Minimising cross-entropy = minimising KL (H(p) is fixed) = maximum likelihood.

import numpy as np

truth = np.array([1, 0, 0])                    # the photo is a cat (one-hot)
for q in [[0.7, 0.2, 0.1], [0.98, 0.01, 0.01], [0.01, 0.98, 0.01]]:
    q = np.array(q)                            # model's guess: cat, dog, bird
    ce = -np.sum(truth * np.log(q))            # cross-entropy with a one-hot truth
    print(q, " loss =", round(ce, 3), " -log(q[cat]) =", round(-np.log(q[0]), 3))

def bce(y, p):                                 # binary cross-entropy (log loss)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))
print("spam, model says 0.9:", round(bce(1, 0.9), 3))
print("spam, model says 0.1:", round(bce(1, 0.1), 3))

Output:

[0.7 0.2 0.1]  loss = 0.357  -log(q[cat]) = 0.357
[0.98 0.01 0.01]  loss = 0.02  -log(q[cat]) = 0.02
[0.01 0.98 0.01]  loss = 4.605  -log(q[cat]) = 4.605
spam, model says 0.9: 0.105
spam, model says 0.1: 2.303
  • Confident and right (0.98): loss 0.02. Confident and wrong (0.98 dog for a cat): loss 4.6.
  • Binary cross-entropy (log loss) for yes/no labels y ∈ {0, 1}: −[y log p + (1 − y) log(1 − p)]. It is the Bernoulli NLL: the coin again. The 0.6121 per toss above was the log loss of predicting p = 0.7 for each toss.
Loss Likelihood it comes from Used for
Mean squared error Normal (Gaussian) noise regression
Binary cross-entropy (log loss) Bernoulli binary classification
Categorical cross-entropy Categorical (one of K classes) multiclass classification

Common mistakes

Mistake What happens Why Fix
np.prod of thousands of probabilities 0.0, best p = 0.01 underflow sum the logs
predicting probability exactly 0 or 1 RuntimeWarning: divide by zero encountered in log, loss = -inf / inf log(0) clip probabilities (e.g. to [1e-15, 1 − 1e-15]) or use a prior
"likelihood = P(p | data)" wrong meaning it's P(data | p) use Bayes for P(p | data)
treating KL as a distance KL(p‖r) ≠ KL(r‖p) not symmetric order matters: KL(truth ‖ model)
mixing log₂ and ln numbers differ by a factor 0.693 bits vs nats log₂ → bits; ln → nats; the best parameters are the same
MLE on a handful of examples extreme estimates (p = 1.0) overfitting more data or a prior (MAP)

Mini project: fit a Poisson rate by maximum likelihood

Customer counts in 50 ten-minute windows (Episode 10's Poisson). Find λ by minimising the NLL with scipy.stats.poisson.logpmf (the log of the PMF):

{{PY:/tmp/claude-0/s/m12.py}}

The MLE of a Poisson rate is the sample mean, exactly as it was for the normal mean and the coin's fraction of heads.

Practice

  1. MLE of p for 12 heads in 40 tosses; and the MAP with the 10 + 10 prior.
  2. MAP for 0 heads in 5 tosses with the 10 + 10 prior.
Solutions 1 and 2

{{PY:/tmp/claude-0/s/h12a.py}}

The MLE for 0/5 would be 0.0 (heads "impossible"); MAP keeps it at 0.4.

  1. Entropy of a fair 6-sided die in bits.
  2. Log loss when the model always says 0.5.
Solutions 3 and 4

{{PY:/tmp/claude-0/s/h12b.py}}

A model that always says 0.5 has log loss ln 2 ≈ 0.693 per example: any useful binary classifier must beat that.

Quick quiz

import numpy as np

p = np.array([0.5, 0.5])
q = np.array([0.25, 0.75])
H = -np.sum(p * np.log2(p))
X = -np.sum(p * np.log2(q))
print(H, round(X, 3), round(X - H, 3))

A) 1.0 1.208 0.208 B) 1.0 1.0 0.0 C) 0.5 1.208 0.708 D) 1.0 0.811 -0.189

Answer

A) 1.0 1.208 0.208. H(p) = 1 bit (a fair coin). X = −(0.5 log₂ 0.25 + 0.5 log₂ 0.75) = 1 + 0.208 = 1.208. KL = X − H = 0.208; KL is never negative, which rules out D.

Interview questions

  1. What is the likelihood, and how is it different from a probability? L(θ) = P(data | θ), a function of the parameter with the data fixed; it doesn't sum to 1 over θ and isn't P(θ | data).
  2. Why do we maximise the log-likelihood instead of the likelihood? Products of many small probabilities underflow; logs turn them into sums, are numerically stable, give the same maximiser, and make gradients easier.
  3. Show that MSE corresponds to MLE. With Gaussian noise of fixed σ, NLL = Σ(y − ŷ)²/(2σ²) + constant, so minimising NLL = minimising squared error.
  4. MLE vs MAP? MAP maximises likelihood × prior; in log form it adds a penalty term, i.e. regularisation. With lots of data they agree.
  5. What prior corresponds to L2 regularisation? A zero-mean normal (Gaussian) prior on the weights; a Laplace prior gives L1.
  6. Define entropy, cross-entropy and KL divergence. H(p) = −Σ p log p; H(p, q) = −Σ p log q; KL(p‖q) = H(p, q) − H(p) ≥ 0.
  7. Why is cross-entropy the standard classification loss? It's the NLL of a categorical model; minimising it is maximum likelihood and minimises KL to the true labels, and it punishes confident wrong answers heavily.
  8. Is KL divergence a distance? No: it isn't symmetric and doesn't satisfy the triangle inequality.

Glossary

Term Meaning Example
Parameter A number a model learns from data p of a coin
Likelihood L(θ) P(observed data | parameter θ) L(0.7) = 0.002224
Maximum likelihood estimation (MLE) Pick θ with the highest likelihood p = 7/10
Underflow A result too small to store, rounded to 0 product of 5,000 probabilities
Logarithm (log, ln, log₂) Inverse of powers; log(ab) = log a + log b ln(10⁻⁶) ≈ −13.8
Natural logarithm Log base e (≈ 2.718); np.log ln 2 = 0.693
Log-likelihood Sum of the logs of the data probabilities −3060.3
Negative log-likelihood (NLL) −log-likelihood; a loss to minimise 0.6121 per toss
Loss function Number measuring how wrong a model is MSE, log loss
Overfitting Fitting the few examples too closely p = 1.0 after 3 heads
Prior Belief about a parameter before the data coins are near fair
Posterior Belief after the data; ∝ likelihood × prior Episode 9
MAP (maximum a posteriori) Pick θ with the highest posterior 0.565
Beta distribution A distribution over probabilities (0 to 1), used as a prior for p Beta(11, 11)
Regularisation A penalty added to the loss to prevent overfitting the prior term
Bit A binary digit, 0 or 1; unit of information with log₂ 2 bits per day
Nat Unit of information with the natural log ML libraries use nats
Prefix-free code No codeword starts another 0, 10, 110, 111
Surprise (information content) −log₂ p snow: 3 bits
Entropy H(p) Average surprise; best average code length 1.75 bits
Cross-entropy H(p, q) Average bits using q's code on data from p 2.625 bits
KL divergence KL(p ‖ q) H(p, q) − H(p): extra bits from believing q 0.875 bits
One-hot vector 1 for the true class, 0 elsewhere cat = [1, 0, 0]
Cross-entropy loss −log q[true class] for a classifier 0.357
Binary cross-entropy (log loss) −[y log p + (1 − y) log(1 − p)] 0.105 vs 2.303
Clipping Keeping values inside a range probabilities in [1e-15, 1 − 1e-15]

Summary

  • Likelihood scores parameter guesses by how probable they make the data; MLE picks the best one.
  • Multiplying many probabilities underflows; use the log-likelihood, and minimise the NLL as a loss.
  • MSE is the NLL of normal noise; log loss is the NLL of a Bernoulli; cross-entropy is the NLL of a categorical model.
  • MAP = likelihood × prior; the prior acts as regularisation and fades as data grows.
  • Entropy = average surprise in bits; cross-entropy = cost of using the wrong distribution; KL = the extra cost, never negative, not symmetric.

Next episode: Exploratory data analysis: feature types, describe, value counts, histograms, box plots, scatter and pair plots, correlation heatmaps, target distribution and a first look at outliers.