These are the notes for Data Analytics with Python, episode 44. In episode 43 every random sample gave a slightly different answer. Here we measure how much it wobbles. You'll meet the sampling distribution, sampling variability, the law of large numbers, the standard error (of a mean and of a proportion) and the central limit theorem (CLT), all built by simulation in NumPy first. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Every program starts with these lines, which build all 100,000 order bills of TiffinBox, the made-up tiffin delivery app from episode 43:
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
from scipy import stats
rng = np.random.default_rng(44)
bill = rng.lognormal(5.5, 0.6, 100_000).round() # all 100,000 TiffinBox bills
The bills are log-normal (episode 42): most are a few hundred rupees and a few are very large, a right-skewed shape. Because we made the population, we know the true answers and can check every idea against them.
The problem: two analysts, two answers
print("parameter: true mean bill =", round(bill.mean(), 1))
a = rng.choice(bill, 500, replace=False) # analyst A's sample
b = rng.choice(bill, 500, replace=False) # analyst B's sample
print("analyst A says:", round(a.mean(), 1))
print("analyst B says:", round(b.mean(), 1))
parameter: true mean bill = 293.3
analyst A says: 287.9
analyst B says: 309.9
Both analysts took a proper random sample of 500 and both got different answers. Arguing about the two numbers can't settle it: with one sample each, you can't tell luck from skill. The real question: if you repeated the sample many times, how much would the answer move?
Real-life example: a cricketer's scores
| Cricket | Statistics |
|---|---|
| one innings: 12 runs today, 87 tomorrow | one sample's statistic (one estimate) |
| every innings on a chart | the sampling distribution |
| how consistent the batter is | the standard error: the spread of that chart |
The sampling distribution
- Sampling variability: a statistic (like a sample mean) changes from one random sample to the next.
- Sampling distribution: the distribution of a statistic over all the samples you could take. You can build it by simulation: take many samples, compute the statistic for each, and look at the results.
- Unbiased estimator: a statistic whose sampling distribution is centred on the true parameter.
means = np.array([rng.choice(bill, 500).mean() for _ in range(10_000)])
print("10,000 sample means, first five:", means[:5].round(1))
print("mean of the sample means:", round(means.mean(), 1))
print("true mean (parameter) :", round(bill.mean(), 1))
print("spread (sd) of the means:", round(means.std(), 1))
10,000 sample means, first five: [275.6 307.8 297.4 299.1 287.7]
mean of the sample means: 293.4
true mean (parameter) : 293.3
spread (sd) of the means: 8.8
(rng.choice samples with replacement by default. When the population is 200 times bigger than the sample, that makes almost no difference.)
- The centre of the sample means (293.4) sits on the true mean (293.3): the sample mean is unbiased.
- Their spread, ₹8.8, is the standard error of the mean for samples of 500. Analyst A (5.4 below) and analyst B (16.6 above) were both within two standard errors: normal wobble, not a mistake.
The law of large numbers
- Law of large numbers (LLN): as you take more and more independent random draws, their average gets closer and closer to the true mean.
rolls = rng.integers(1, 7, 100_000) # a fair die: 1 to 6
for n in [10, 100, 1_000, 10_000, 100_000]:
print(f"average of first {n:>7,} rolls: {rolls[:n].mean():.3f}")
average of first 10 rolls: 3.200
average of first 100 rolls: 3.250
average of first 1,000 rolls: 3.391
average of first 10,000 rolls: 3.486
average of first 100,000 rolls: 3.494
The long-run average of a fair die is 3.5, and the running average settles towards it. Trap (the gambler's fallacy): after a run of sixes the die does not "owe" you small numbers. The average settles because new rolls dilute the old ones, not because they balance them. The LLN says averages settle; it doesn't say how fast. The standard error does.
Standard error = σ / √n
sd = bill.std()
print("population sd:", round(sd, 1))
for n in [25, 100, 400, 1600]:
m = [rng.choice(bill, n).mean() for _ in range(5_000)]
formula = sd / np.sqrt(n)
print(f"n = {n:>4}: spread {np.std(m):5.1f} sd/sqrt(n) {formula:5.1f}")
population sd: 194.4
n = 25: spread 39.3 sd/sqrt(n) 38.9
n = 100: spread 19.5 sd/sqrt(n) 19.4
n = 400: spread 9.9 sd/sqrt(n) 9.7
n = 1600: spread 4.8 sd/sqrt(n) 4.9
Every time n grows four times, the spread halves, exactly as the formula says:
- σ (sigma): the population standard deviation. μ (mu): the population (true) mean. s: the sample standard deviation. n: the sample size.
- Standard error of the mean (SE): the standard deviation of the sampling distribution of the mean, SE = σ / √n.
- To halve the error you need 4× the data; to cut it to a tenth, 100×. Doubling n only shrinks the SE by about 29% (1 − 1/√2).
Standard deviation vs standard error. The standard deviation (₹194) is the spread of the data: how much bills differ from each other. The standard error (₹8.8 for n = 500) is the spread of the estimate: how far the sample mean typically is from the true mean.
The standard error from one sample
In real life you have one sample and you don't know σ. First, a real error:
one = rng.choice(bill, 500)
se = one.std(ddof=1) / sqrt(500)
NameError: name 'sqrt' is not defined
Python has no built-in sqrt; use np.sqrt (or math.sqrt). Then replace the unknown σ with the sample's own standard deviation s (with ddof=1, dividing by n − 1: Bessel's correction from episode 35). scipy.stats.sem (standard error of the mean) does exactly this.
one = rng.choice(bill, 500) # the ONLY sample we have
se = one.std(ddof=1) / np.sqrt(len(one))
print("sample mean:", round(one.mean(), 1))
print("sample sd :", round(one.std(ddof=1), 1))
print("estimated standard error:", round(se, 1))
print("scipy stats.sem :", round(stats.sem(one), 1))
sample mean: 275.6
sample sd : 173.2
estimated standard error: 7.7
scipy stats.sem : 7.7
Estimated SE = s / √n. From a single sample you get 7.7 (the true SE is 8.8): not exact, but it tells you roughly how far off your mean can be.
The central limit theorem
import matplotlib.pyplot as plt
fig, axs = plt.subplots(1, 4, figsize=(16, 4))
axs[0].hist(bill, bins=np.arange(0, 1001, 10), color="#e4572e"); axs[0].set_title("the bills (population): skewed")
for ax, n in zip(axs[1:], [2, 10, 50]):
m = [rng.choice(bill, n).mean() for _ in range(20_000)]
ax.hist(m, bins=np.arange(0, 1001, 10)); ax.set_title(f"means of samples of n = {n}")
for ax in axs: ax.set_yticks([]); ax.set_xlim(0, 1000)
fig.tight_layout(); fig.savefig("clt.png", dpi=110)

The bills are skewed. Means of 2 bills are still skewed, means of 10 are nearly symmetric, means of 50 form a narrow bell.
- Central limit theorem (CLT): for a large enough sample of independent values, the sample mean follows an approximately normal distribution, centred on the true mean μ with standard deviation σ/√n, whatever the shape of the data (as long as the data has a finite standard deviation).
Measured with skewness (episode 36):
print("skew of the bills themselves:", round(stats.skew(bill), 2))
for n in [2, 10, 50, 500]:
m = np.array([rng.choice(bill, n).mean() for _ in range(20_000)])
print(f"skew of means, n = {n:>3}: {stats.skew(m):.2f}")
one = rng.choice(bill, 500)
print("skew of ONE sample of 500:", round(stats.skew(one), 2))
skew of the bills themselves: 2.3
skew of means, n = 2: 1.66
skew of means, n = 10: 0.75
skew of means, n = 50: 0.31
skew of means, n = 500: 0.11
skew of ONE sample of 500: 3.31
Two traps: 1. The CLT makes the mean normal, not your data: one sample of 500 bills is still very skewed (3.31). 2. "n ≥ 30" is only a rough rule of thumb. The more skewed the data, the bigger n must be before the bell appears.
Why it matters: many statistics are averages or sums, so the normal curve appears everywhere, and confidence intervals (episode 45) and hypothesis tests (episodes 46-49) rely on it.
Standard error of a proportion
A proportion (share) is just the mean of 0s and 1s, so the CLT covers it too. Its standard error is √(p(1 − p) / n).
five = rng.random(100_000) < 0.20 # True = gave 5 stars (20%)
shares = [rng.choice(five, 500).mean() for _ in range(10_000)]
p = five.mean()
print("true share p:", round(p, 3))
print("spread of sample shares:", round(np.std(shares), 4))
print("sqrt(p * (1 - p) / n) :", round(np.sqrt(p * (1 - p) / 500), 4))
true share p: 0.2
spread of sample shares: 0.0177
sqrt(p * (1 - p) / n) : 0.0179
A sample of 500 measures a 20% share to within about 1.8 percentage points (one standard error).
95% of sample means within 1.96 SE
Sample means are approximately normal with spread SE, and 95% of a normal lies within 1.96 standard deviations (episode 42):
means = np.array([rng.choice(bill, 500).mean() for _ in range(10_000)])
se = bill.std() / np.sqrt(500)
inside = np.mean(abs(means - bill.mean()) < 1.96 * se)
print("SE =", round(se, 1), " 1.96 x SE =", round(1.96 * se, 1))
print("sample means within 1.96 SE of the truth:", round(inside, 3))
SE = 8.7 1.96 x SE = 17.0
sample means within 1.96 SE of the truth: 0.948
94.8% of sample means land within ₹17 of the truth. Turned around: your one sample mean ± 1.96 SE catches the truth about 95% of the time. That range is a confidence interval, the topic of episode 45.
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
s / sqrt(500) |
NameError: name 'sqrt' is not defined |
no built-in sqrt |
np.sqrt(500) or math.sqrt |
| "the CLT made my data normal" | no error; one sample's skew is 3.31 | only the mean becomes normal | check the data's shape separately |
| using the sd where the SE is needed | no error; ₹194 vs ₹8.8 | sd = data spread, SE = estimate spread | SE = s / √n |
| doubling n to halve the error | no error; SE shrinks only ~29% | SE falls with √n | 4× n halves the SE |
np.std(x) for the sample sd |
no error; slightly too small | NumPy's default is ddof=0 |
x.std(ddof=1) or stats.sem |
| trusting the CLT with n = 5 on skewed data | no error; means still skewed | n too small for that skew | larger n, or the bootstrap (episode 45) |
| "after 5 heads, tails is due" | no error; wrong belief | each draw is independent | LLN works by dilution |
Mini project: how many bills to sample?
TiffinBox wants its estimate of the mean bill to have a standard error of at most ₹5. Since SE = σ/√n, n = (σ / SE)².
target = 5 # SE we can live with, rupees
n = int(np.ceil((bill.std() / target) ** 2))
print("sample size needed:", n)
means = [rng.choice(bill, n).mean() for _ in range(5_000)]
print("simulated SE with that n:", round(np.std(means), 2))
sample size needed: 1512
simulated SE with that n: 5.03
About 1,512 bills; the simulation confirms an SE of about ₹5. (In real life you'd use a guess of σ from a pilot sample or past data.)
Practice
1. Waiting gaps follow an exponential distribution with mean 5 minutes (very skewed). Simulate 10,000 means of samples of 30. How skewed are the data and the means? Does SE = σ/√n hold?
Solution
import numpy as np
from scipy import stats
rng = np.random.default_rng(1)
pop = rng.exponential(5, 100_000) # waiting gaps, mean 5 min
means = np.array([rng.choice(pop, 30).mean() for _ in range(10_000)])
print("skew of the data :", round(stats.skew(pop), 2))
print("skew of the means:", round(stats.skew(means), 2))
print("SE simulated:", round(means.std(), 3), " formula:", round(pop.std() / np.sqrt(30), 3))
skew of the data : 1.99
skew of the means: 0.35
SE simulated: 0.904 formula: 0.905
The means are far less skewed than the data, and the formula matches the simulation.
2. What is the standard error of a 50% share measured on 1,000 people?
Solution
import numpy as np
p, n = 0.5, 1000
print(round(np.sqrt(p * (1 - p) / n), 4))
0.0158
About 1.6 percentage points.
3. Compute a die's running average after 1, 10, 100 and 1,000 rolls.
Solution
import numpy as np
rng = np.random.default_rng(2)
rolls = rng.integers(1, 7, 1000)
running = np.cumsum(rolls) / np.arange(1, 1001)
print(running[[0, 9, 99, 999]].round(3))
[6. 3. 3.61 3.51]
Quick quiz
# SE of the mean with n = 100 is 6.0
# How many people do you need for SE = 3.0?
A. 200 B. 300 C. 400 D. 600
Answer
C. 400. SE = σ/√n; halving the SE means doubling √n, so n grows four times.
Interview questions
- What is a sampling distribution? The distribution of a statistic across all possible samples of the same size; its spread is the standard error.
- Standard deviation vs standard error? SD measures the spread of individual values; SE measures the spread of an estimate (SE = σ/√n).
- State the central limit theorem. For large n, the sample mean of independent values with finite variance is approximately normal with mean μ and standard deviation σ/√n, whatever the data's shape.
- Does the CLT make the data normal? No, only the distribution of the sample mean (or sum).
- Law of large numbers vs CLT? LLN: the sample mean converges to μ. CLT: how it is distributed around μ on the way (normal, spread σ/√n).
- How much data to halve the error? Four times as much.
- SE of a proportion? √(p(1 − p)/n), largest at p = 0.5.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Sampling variability | A statistic changes from sample to sample | 287.9 vs 309.9 |
| Sampling distribution | The distribution of a statistic over many samples | 10,000 sample means |
| Unbiased estimator | Its sampling distribution is centred on the true value | mean of means 293.4 ≈ 293.3 |
| Standard error (SE) | The standard deviation of a sampling distribution | ₹8.8 for n = 500 |
| Standard error of the mean | σ / √n, estimated by s / √n | 194.4 / √500 = 8.7 |
| σ (sigma) | Population standard deviation | 194.4 |
| μ (mu) | Population mean | 293.3 |
| s | Sample standard deviation (ddof=1) |
173.2 |
| n | Sample size | 500 |
stats.sem |
SciPy's standard error of the mean | 7.7 |
| Law of large numbers (LLN) | Averages of more draws get closer to the true mean | die → 3.5 |
| Gambler's fallacy | Believing past draws change future ones | "tails is due" |
| Central limit theorem (CLT) | Sample means are approximately normal for large n | skew 2.3 → 0.11 |
| Proportion (share) | Mean of 0/1 values | 20% gave 5 stars |
| SE of a proportion | √(p(1 − p)/n) | 0.0179 |
| Sampling with replacement | A unit can be picked more than once | rng.choice(bill, 500) |
| Confidence interval (preview) | Estimate ± about 2 SE; catches the truth ~95% of the time | episode 45 |
Summary
- A statistic varies from sample to sample; its sampling distribution shows how.
- Random sampling is unbiased: the sampling distribution is centred on the truth.
- Law of large numbers: averages settle towards the true mean as n grows.
- Standard error = σ/√n, estimated by s/√n (
stats.sem); 4× data halves it. - CLT: sample means are approximately normal whatever the data's shape; the data itself stays as it is.
- SE of a proportion = √(p(1 − p)/n).
- About 95% of sample means fall within 1.96 SE of the truth, which leads to confidence intervals in episode 45.