SyntaxPilot
Courses/Machine Learning/Lesson 10

Machine Learning · Lesson 10 of 90 · 17 min

Maths 8: distributions

discrete vs continuous random variablesprobability mass function (PMF)probability density function (PDF)cumulative distribution function (CDF)BernoullibinomialPoissonuniformnormal (Gaussian)standard normalexpectationvariancestandard deviationsampling with NumPy

Machine Learning Full Course · Episode 10. Every code sample below was run with Python 3.13, NumPy 2.5.3 and SciPy 1.18.1; the output shown is the real output. All the data (the 0.3 buy rate, 4 customers per 10 minutes, the heights) is made up for teaching.

The problem: how many cups tomorrow?

Raju buys milk every evening for the next day. Too little and he turns customers away; too much and it spoils. So he asks: how many cups will I sell tomorrow?

There is no single answer: some days 3 cups, some days 9. Tomorrow's sales are a random variable (Episode 9: a number that depends on chance). What Raju needs is the chance of every possible value: P(0 cups), P(1 cup), …, P(20 cups).

A list (or formula) giving the probability of every value of a random variable is a probability distribution. Today we invent the most important ones.

Discrete vs continuous random variables

Kind Values Examples Tool
Discrete countable, with gaps cups sold (0, 1, 2 … never 2.5), customers per hour PMF: a probability for each value
Continuous any value in a range, no gaps height 165.37 cm, waiting time 2.81 min PDF: a density; probability = area

Bernoulli: one yes-or-no try

One person walks past: they buy (1, a success) or don't (0, a failure). A single try with two outcomes is a Bernoulli trial. If 3 in 10 people buy, then P(1) = p = 0.3 and P(0) = 1 − p = 0.7. The Bernoulli distribution is described by that one number p.

import numpy as np

rng = np.random.default_rng(10)
buys = rng.random(10) < 0.3        # 10 passers-by, each buys with chance 0.3
print(buys.astype(int))
print("cups sold:", buys.sum())
more = [int((rng.random(10) < 0.3).sum()) for _ in range(5)]
print("cups sold, 5 more days:", more)

Output:

[0 1 0 1 0 1 0 0 0 0]
cups sold: 3
cups sold, 5 more days: [2, 2, 6, 2, 1]
  • np.random.default_rng(10) creates a random number generator; the seed 10 makes the "random" numbers repeat exactly, so you get the same output.
  • rng.random(10) gives 10 numbers between 0 and 1; < 0.3 is True 30% of the time, which is one Bernoulli trial per person.
  • astype(int) turns True/False into 1/0; sum() counts the buyers.

Same p, different days: 3, 2, 2, 6, 2, 1 cups. How likely is each count?

Binomial: the obvious way fails

20 people walk past. P(exactly 6 cups)? The obvious way is to list every buy/no-buy pattern and add up the chances of the patterns with 6 buys:

import itertools, time

p, n = 0.3, 20
start = time.time()
probs = [0.0] * (n + 1)
for day in itertools.product([0, 1], repeat=n):   # every buy/no-buy pattern
    k = sum(day)
    probs[k] += p**k * (1 - p)**(n - k)
secs = time.time() - start
print("P(exactly 6 cups) =", round(probs[6], 4))
print("patterns:", 2**n, " seconds:", round(secs, 1))
years = 2**100 / (2**n / secs) / 31_557_600
print("100 people would take about", f"{years:.1e}", "years")

Output:

P(exactly 6 cups) = 0.1916
patterns: 1048576  seconds: 0.5
100 people would take about 2.1e+16 years

It works (0.1916), but needs 2²⁰ ≈ one million patterns for 20 people, and every extra person doubles the work. For 100 people it would take more than ten thousand trillion years (the exact estimate depends on your computer's speed). itertools.product([0, 1], repeat=n) produces every pattern of n zeros and ones.

Why it fails: we list patterns one by one, when all we need is how many there are.

Binomial: the fix (count the ways)

  • Every pattern with exactly k buys has the same chance: pᵏ × (1 − p)ⁿ⁻ᵏ.
  • The number of such patterns is "n choose k", the combinations of k people out of n: math.comb(20, 6) = 38,760.

Binomial PMF:

P(X = k) = comb(n, k) × pᵏ × (1 − p)ⁿ⁻ᵏ

The number of successes in n independent Bernoulli trials with chance p follows the binomial distribution, Binomial(n, p). A function giving the probability of each exact value of a discrete random variable is a probability mass function (PMF). "Mass": each value carries its own lump of probability, and all lumps add to 1.

from math import comb
import numpy as np

def binom_pmf(k, n, p):                 # ways x chance of one way
    return comb(n, k) * p**k * (1 - p)**(n - k)

n, p = 20, 0.3
print("ways to pick 6 buyers of 20:", comb(20, 6))
print("P(X = 6) =", round(binom_pmf(6, n, p), 4))
print("sum of all P:", round(sum(binom_pmf(k, n, p) for k in range(n + 1)), 6))
print("100 people, P(X = 30) =", round(binom_pmf(30, 100, p), 4))

rng = np.random.default_rng(10)
sales = rng.binomial(n, p, size=100_000)   # 100,000 simulated days
print("simulated P(X = 6) =", round(np.mean(sales == 6), 4))

Output:

ways to pick 6 buyers of 20: 38760
P(X = 6) = 0.1916
sum of all P: 1.0
100 people, P(X = 30) = 0.0868
simulated P(X = 6) = 0.1938

rng.binomial(n, p, size=100_000) simulates 100,000 days at once; np.mean(sales == 6) is the fraction of days with exactly 6 cups.

The shape of the PMF (n = 20, p = 0.3)

Cups k 2 4 6 8 10
P(X = k) 0.028 0.130 0.192 0.114 0.031

The peak is at n × p = 6.

Poisson: counting arrivals

New question: how many customers arrive in the next 10 minutes? There's no fixed number of tries n, only an average rate: λ (lambda) = 4 per 10 minutes.

Trick: slice the 10 minutes into many tiny slices, so short that at most one customer arrives in each. Each slice is a Bernoulli trial with p = λ / slices. As the slices get thinner, the binomial answer settles:

from math import comb, exp, factorial
import numpy as np

lam = 4                                  # 4 customers per 10 minutes, on average
def poisson_pmf(k):
    return lam**k * exp(-lam) / factorial(k)

for slices in [10, 100, 10_000]:         # binomial: split 10 minutes into tiny slices
    p = lam / slices
    binom6 = comb(slices, 6) * p**6 * (1 - p)**(slices - 6)
    print(slices, "slices: P(X = 6) =", round(binom6, 4))
print("Poisson:     P(X = 6) =", round(poisson_pmf(6), 4))

rng = np.random.default_rng(3)
arrivals = rng.poisson(lam, size=100_000)
print("simulated:   P(X = 6) =", round(np.mean(arrivals == 6), 4), " average:", arrivals.mean())

Output:

10 slices: P(X = 6) = 0.1115
100 slices: P(X = 6) = 0.1052
10000 slices: P(X = 6) = 0.1042
Poisson:     P(X = 6) = 0.1042
simulated:   P(X = 6) = 0.1043  average: 4.01382

The limit has its own formula, the Poisson PMF:

P(X = k) = λᵏ e^(−λ) / k!

(e ≈ 2.718 is Euler's number; k! = 1 × 2 × … × k is "k factorial".) Use the Poisson distribution for counts of events in a stretch of time or space: customers per hour, typos per page, requests per second.

Continuous variables: P(exactly 170) = 0

import numpy as np

rng = np.random.default_rng(5)
heights = rng.normal(165, 7, size=100_000)   # made-up heights in cm
print(heights[:4])
print("P(height == 170) =", np.mean(heights == 170))
print("P(169.5 < height < 170.5) =", np.mean((heights > 169.5) & (heights < 170.5)))
print("P(169.95 < height < 170.05) =", np.mean((heights > 169.95) & (heights < 170.05)))

Output:

[159.38648002 155.72948703 163.26146865 167.94311667]
P(height == 170) = 0.0
P(169.5 < height < 170.5) = 0.04586
P(169.95 < height < 170.05) = 0.00444

Not one of 100,000 people is exactly 170 cm: with infinitely many possible values, every single exact value has probability 0. So a PMF is useless here. Ask about a range instead:

Strip around 170 width probability probability ÷ width
169.5 to 170.5 1 cm 0.0459 ≈ 0.046 per cm
169.95 to 170.05 0.1 cm 0.0044 ≈ 0.044 per cm

The chance per centimetre stays steady. That is a density.

The probability density function (PDF)

Analogy: crowd density on a railway platform. You can't count people at a point, but you can say "5 people per metre near the stairs". People on a stretch = density × length.

  • The PDF f(x) is the probability per unit near x. It's a density, not a probability.
  • P(a < X < b) = area under f between a and b (an integral: adding up many thin strips).
  • Total area = 1, and P(X = exactly x) = 0.
  • A density can be greater than 1 (a packed spot on the platform); only areas must be ≤ 1.

The normal PDF with centre μ (mu, the mean) and spread σ (sigma, the standard deviation):

f(x) = e^(−(x − μ)² / (2σ²)) / (σ √(2π))

import numpy as np

def normal_pdf(x, mu, sigma):
    return np.exp(-(x - mu)**2 / (2 * sigma**2)) / (sigma * np.sqrt(2 * np.pi))

print("density at 170:", round(normal_pdf(170, 165, 7), 4))
print("area of a 1 cm strip:", round(normal_pdf(170, 165, 7) * 1, 4))

xs = np.linspace(100, 230, 100_001)                 # tiny strips
print("total area:", round(np.trapezoid(normal_pdf(xs, 165, 7), xs), 4))
print("uniform(0, 0.5) density:", 1 / 0.5)

Output:

density at 170: 0.0442
area of a 1 cm strip: 0.0442
total area: 1.0
uniform(0, 0.5) density: 2.0
  • The density at 170 (0.0442) matches the ≈ 0.045 per cm we measured.
  • np.linspace(100, 230, 100_001) makes 100,001 evenly spaced points; np.trapezoid(y, x) adds up the thin trapezoid strips under the curve: the area is 1.0.
  • A uniform distribution on 0 to 0.5 (every value equally likely) has density 1 / 0.5 = 2: bigger than 1 and perfectly valid.

The cumulative distribution function (CDF)

Most real questions are "at most": will I wait at most 3 minutes? Sell at most 6 cups?

  • CDF: F(x) = P(X ≤ x), all the probability up to x. Cumulative means piling up.
  • Discrete: add PMF values up to x. Continuous: area under the PDF up to x.
  • P(a < X ≤ b) = F(b) − F(a). F climbs from 0 to 1.
import numpy as np
from scipy.stats import norm, binom

rng = np.random.default_rng(8)
wait = rng.uniform(0, 10, size=100_000)       # bus every 10 min: wait 0 to 10
print("P(wait <= 3): simulated", round(np.mean(wait <= 3), 4), " CDF", 3 / 10)

print("P(height <= 170) =", round(norm.cdf(170, loc=165, scale=7), 4))
print("P(160 < height <= 170) =", round(norm.cdf(170, 165, 7) - norm.cdf(160, 165, 7), 4))
print("P(cups <= 6) =", round(binom.cdf(6, 20, 0.3), 4))

Output:

P(wait <= 3): simulated 0.2996  CDF 0.3
P(height <= 170) = 0.7625
P(160 < height <= 170) = 0.5249
P(cups <= 6) = 0.608
  • Bus every 10 minutes, you arrive at a random moment: the wait is Uniform(0, 10), so F(3) = 3/10.
  • SciPy (scipy.stats) has ready-made distributions: norm.cdf(x, loc=μ, scale=σ), binom.cdf(k, n, p), poisson.cdf(k, λ), and the matching .pmf, .pdf, .sf (survival function, 1 − CDF) and .ppf (inverse CDF).

The normal (Gaussian) distribution

  • A symmetric bell centred at the mean μ with spread σ; also called the Gaussian (after Gauss).
  • It appears when many small independent effects add up (height from many genes and meals; measurement errors).
  • z-score: z = (x − μ) / σ, how many σ a value is from the mean.
  • The standard normal has μ = 0, σ = 1; every normal becomes standard after converting to z-scores.
  • 68–95–99.7 rule: about 68% of values within 1σ, 95% within 2σ, 99.7% within 3σ.
import numpy as np

rng = np.random.default_rng(1)
heights = rng.normal(165, 7, size=100_000)
z = (heights - 165) / 7                     # standardise: how many sigmas away
print("z mean:", round(z.mean(), 3), " z std:", round(z.std(), 3))
for k in [1, 2, 3]:
    print(f"within {k} sigma:", round(np.mean(np.abs(z) < k), 4))

dice = rng.integers(1, 7, size=(100_000, 10)).sum(axis=1)   # add 10 dice
counts = [int(np.sum(dice == s)) for s in range(25, 50, 5)]
print("dice counts:", counts)                # sums 25, 30, 35, 40, 45

Output:

z mean: -0.005  z std: 0.997
within 1 sigma: 0.6841
within 2 sigma: 0.955
within 3 sigma: 0.9971
dice counts: [1288, 4853, 7285, 4835, 1298]

The sum of 10 flat dice is already bell-shaped: low counts at 25 and 45, the peak at 35. Episode 11 shows why (the central limit theorem).

Expectation, variance and standard deviation

  • Expectation (expected value) E[X] = Σ x · P(x): the long-run average. For a die, 3.5, a value you can never roll.
  • The obvious spread measure, the average distance from the mean, is always 0: values below the mean cancel those above (the same trap as the plain average error in Episode 8).
  • Variance Var(X) = Σ (x − E[X])² · P(x): average squared distance. Its units are squared.
  • Standard deviation σ = √Var(X): spread in the original units.
import numpy as np

faces = np.array([1, 2, 3, 4, 5, 6])
probs = np.full(6, 1 / 6)
E = np.sum(faces * probs)                       # expectation
print("expected value:", E)
print("average distance from E:", np.sum((faces - E) * probs))
var = np.sum((faces - E)**2 * probs)            # variance
print("variance:", round(var, 4), " std dev:", round(np.sqrt(var), 4))

rng = np.random.default_rng(2)
rolls = rng.integers(1, 7, size=100_000)
print("simulated mean:", rolls.mean(), " var:", round(rolls.var(), 4))

Output:

expected value: 3.5
average distance from E: 0.0
variance: 2.9167  std dev: 1.7078
simulated mean: 3.49934  var: 2.9207

Mean and variance of each distribution

Distribution Parameters Mean Variance Example
Bernoulli p p p(1 − p) one passer-by
Binomial n, p np np(1 − p) cups from 20 passers-by
Poisson λ λ λ customers per 10 min
Uniform a, b (a + b)/2 (b − a)²/12 bus wait 0 to 10
Normal μ, σ μ σ² heights

Sampling with NumPy

import numpy as np

rng = np.random.default_rng(42)
N = 100_000
samples = {
    "bernoulli p=0.3": rng.binomial(1, 0.3, N),
    "binomial 20, 0.3": rng.binomial(20, 0.3, N),
    "poisson 4": rng.poisson(4, N),
    "uniform 0 to 10": rng.uniform(0, 10, N),
    "normal 165, 7": rng.normal(165, 7, N),
}
for name, x in samples.items():
    print(f"{name:17} mean {x.mean():8.3f}   variance {x.var():7.3f}")

Output:

bernoulli p=0.3   mean    0.300   variance   0.210
binomial 20, 0.3  mean    5.997   variance   4.205
poisson 4         mean    4.001   variance   3.963
uniform 0 to 10   mean    5.007   variance   8.365
normal 165, 7     mean  164.986   variance  49.406

Every sampled mean and variance matches the table: 0.3/0.21, 6/4.2, 4/4, 5/8.33, 165/49. Sampling means drawing random values from a distribution; it checks any formula.

NumPy (rng = np.random.default_rng(seed)) Draws from
rng.random(size) Uniform(0, 1)
rng.integers(low, high, size) whole numbers low … high − 1
rng.binomial(n, p, size) Binomial (n = 1 → Bernoulli)
rng.poisson(lam, size) Poisson
rng.uniform(a, b, size) Uniform(a, b)
rng.normal(loc, scale, size) Normal(μ = loc, σ = scale)

Common mistakes

import numpy as np

rng = np.random.default_rng(0)
print(rng.normal(165, -7, size=3))

Output:

ValueError: scale < 0
Mistake What happens Why Fix
rng.normal(165, -7) ValueError: scale < 0 σ is a distance, never negative use a positive σ
rng.binomial(20, 1.3) ValueError: p < 0, p > 1 or p is NaN p is a probability keep 0 ≤ p ≤ 1
rng.poisson(-4) ValueError: lam < 0 or lam is NaN an average count can't be negative λ ≥ 0
rng.normal(165, 49) meaning variance 49 variance comes out 2,401 scale is σ, not σ² rng.normal(165, 7)
rng.integers(1, 6) for a die never rolls a 6 the upper end is excluded rng.integers(1, 7)
P(height == 170) from a PDF always 0 single points have no area ask for a range: F(b) − F(a)
"density 2.0 is impossible" false alarm density is per unit only areas must be ≤ 1
no seed in a simulation different numbers every run no fixed starting point np.random.default_rng(seed)

Mini project: how much milk should Raju plan for?

20 people pass per hour and 3 in 10 buy. Find the smallest number of cups k that covers at least 95% of hours, using the binomial CDF, then check it by simulation.

{{PY:/tmp/claude-0/s/m10.py}}

binom.ppf(0.95, n, p) is the inverse CDF (percent point function): the smallest k with F(k) ≥ 0.95. Planning for 9 cups an hour, Raju runs out in only about 5% of hours.

Practice

  1. P(at least 8 cups) for n = 20, p = 0.3.
Solution

"At least 8" = 1 − P(X ≤ 7).

{{PY:/tmp/claude-0/s/h10a.py}}

  1. With λ = 4 customers per 10 minutes, P(more than 6 customers)?
Solution

P(X > 6) = 1 − F(6), which SciPy calls the survival function sf.

{{PY:/tmp/claude-0/s/h10b.py}}

  1. Does the 68% rule hold for a uniform distribution?
Solution

No. For Uniform(0, 10), σ = 10/√12 ≈ 2.887, and the fraction within 1σ is 2 × 2.887 / 10 ≈ 0.577. The 68–95–99.7 rule is only for normal distributions.

{{PY:/tmp/claude-0/s/h10c.py}}

  1. Mean and variance of a loaded die that lands six half the time (the other faces 0.1 each).
Solution

E = 0.1 × (1 + 2 + 3 + 4 + 5) + 0.5 × 6 = 1.5 + 3 = 4.5.

{{PY:/tmp/claude-0/s/h10d.py}}

Quick quiz

from math import comb

n, p = 4, 0.5
pmf = [comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n + 1)]
print(pmf[2], sum(pmf[:2]))

A) 0.375 0.3125 B) 0.5 0.3125 C) 0.375 0.25 D) 0.25 0.375

Answer

A) 0.375 0.3125. comb(4, 2) = 6 patterns × 0.5⁴ = 6/16 = 0.375. P(X ≤ 1) = (1 + 4)/16 = 0.3125, the CDF at 1.

Interview questions

  1. PMF vs PDF? A PMF gives the probability of each exact value of a discrete variable; a PDF gives a density for a continuous variable, and probabilities are areas under it.
  2. Can a PDF be greater than 1? Yes. Density is per unit; only the total area (and every area) must be ≤ 1. Uniform(0, 0.5) has density 2.
  3. What does the CDF give, and how do you get P(a < X ≤ b)? F(x) = P(X ≤ x); F(b) − F(a).
  4. When do you use binomial vs Poisson? Binomial: a fixed number n of independent yes/no trials. Poisson: counts of events over an interval with a known average rate λ; it is the limit of the binomial with large n and small p.
  5. Mean and variance of Binomial(n, p) and Poisson(λ)? np and np(1 − p); λ and λ.
  6. Why is the normal distribution so common? Many small independent effects add up to it (central limit theorem, Episode 11).
  7. What is a z-score? (x − μ)/σ: how many standard deviations from the mean; it turns any normal into the standard normal.
  8. Why do we square deviations for variance? Plain deviations from the mean always average to zero; squaring makes every deviation count positively.

Glossary

Term Meaning Example
Random variable A number that depends on chance cups sold tomorrow
Probability distribution The probability of every value of a random variable P(0 cups), …, P(20 cups)
Discrete random variable Countable values with gaps cups: 0, 1, 2
Continuous random variable Any value in a range height 165.37 cm
Bernoulli trial One try with two outcomes: success (1) or failure (0) one passer-by buys or not
Bernoulli distribution Distribution of one Bernoulli trial; parameter p P(1) = 0.3
Success / failure The two outcomes of a trial, coded 1 / 0 buy / no buy
Combinations, math.comb(n, k) Number of ways to choose k items out of n comb(20, 6) = 38,760
Binomial distribution Number of successes in n independent Bernoulli trials cups from 20 people
Probability mass function (PMF) Probability of each exact value (discrete) P(X = 6) = 0.1916
Poisson distribution Counts of events in an interval with average rate λ customers per 10 min
λ (lambda) Average count per interval (Poisson) 4
Factorial, k! 1 × 2 × … × k 3! = 6
Euler's number e ≈ 2.718, the base of natural logarithms e^(−4)
Density Amount per unit (of length, time …) 0.044 per cm
Probability density function (PDF) Density of a continuous variable; area = probability normal PDF at 170 = 0.0442
Integral (area under a curve) Sum of many thin strips under a curve np.trapezoid
Cumulative distribution function (CDF) F(x) = P(X ≤ x) F(170) = 0.7625
Survival function (sf) 1 − CDF = P(X > x) P(X > 6) = 0.1107
Inverse CDF (ppf) Smallest x with F(x) ≥ q 9 cups for 95%
Uniform distribution Every value in [a, b] equally likely bus wait 0 to 10
Normal (Gaussian) distribution Symmetric bell with mean μ and standard deviation σ heights
μ (mu), σ (sigma) Mean and standard deviation 165, 7
Standard normal Normal with μ = 0, σ = 1 z-scores
z-score (x − μ)/σ 172 cm → z = 1
68–95–99.7 rule Share of a normal within 1, 2, 3 σ 0.6841, 0.955, 0.9971
Expectation, E[X] Σ x · P(x): long-run average die: 3.5
Variance Average squared distance from the mean die: 2.9167
Standard deviation √variance, in original units die: 1.7078
Sampling Drawing random values from a distribution rng.normal(165, 7, N)
Random number generator, seed Produces pseudo-random numbers; a seed makes them repeat default_rng(42)
SciPy, scipy.stats Python library with ready-made distributions norm.cdf

Summary

  • A distribution gives the chance of every value of a random variable.
  • Discrete variables have a PMF; continuous variables have a PDF where probability = area, and any exact value has probability 0.
  • The CDF F(x) = P(X ≤ x) answers every "at most" question; ranges are F(b) − F(a).
  • Bernoulli (one try), binomial (successes in n tries), Poisson (counts at rate λ), uniform (all equal), normal (the bell; z-scores; 68–95–99.7).
  • Expectation is the long-run average; variance is the average squared distance; standard deviation is its square root.
  • NumPy's default_rng samples every distribution; sampling checks any formula.

Next episode: Maths for ML 9, statistics for ML: mean, median, percentiles, IQR, skewness, correlation, the central limit theorem, confidence intervals, p-values, t-tests and chi-square tests.