These are the notes for Data Analytics with Python, episode 35. Last episode gave you seven averages. But two stalls can have exactly the same average and completely different days. Today you invent the numbers that measure spread: the range, variance and standard deviation, the difference between population and sample variance (Bessel's correction, ddof), percentiles, quartiles and quantile(), the interquartile range (IQR), the median absolute deviation (MAD), the coefficient of variation, the five-number summary and how to read a box plot. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs are copied from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its last line, as in the video.
The problem: same average, different days
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
a = pd.Series([95, 100, 105, 98, 102, 100, 97, 103, 99, 101]) # Pune
b = pd.Series([20, 180, 60, 140, 100, 40, 160, 100, 80, 120]) # Nashik
print("mean :", a.mean(), b.mean())
print("median:", a.median(), b.median())
mean : 100.0 100.0
median: 100.0 100.0
Identical centres. But Pune's bills are all between 95 and 105, and Nashik's run from 20 to 180.
Real-life example: two buses. Bus A arrives at 9:00 every day, give or take a minute. Bus B also arrives at 9:00 on average, but anywhere from 8:40 to 9:20. Same average, very different life. What you need is a second number: how spread out the values are around the centre. That is dispersion (or spread), and a number that measures it is a measure of dispersion.
Range
The range is the largest value minus the smallest.
print("range a:", a.max() - a.min())
print("range b:", b.max() - b.min())
a2 = pd.concat([a, pd.Series([300])], ignore_index=True)
print("range a + one big order:", a2.max() - a2.min())
range a: 10
range b: 160
range a + one big order: 205
The range sees the difference (10 vs 160), but one 300-rupee order makes steady Pune's range 205, bigger than Nashik's. The range uses only two values, the most extreme ones, so one outlier (a value far from the rest) controls it completely.
Variance and standard deviation, step by step
Step 1: deviations. A deviation is a value minus the mean. Try averaging them:
d = b - b.mean() # deviation from the mean
print(d.tolist())
print("mean deviation:", d.mean())
[-80.0, 80.0, -40.0, 40.0, 0.0, -60.0, 60.0, 0.0, -20.0, 20.0]
mean deviation: 0.0
Always zero, for every dataset: negative deviations exactly cancel positive ones, because the mean is the balance point of the data.
Step 2: square them, so minus signs disappear. Step 3: the mean of the squared deviations is the variance. Step 4: its square root is the standard deviation.
d = b - b.mean()
var = (d ** 2).mean() # mean squared deviation
print("variance:", var, "rupees squared")
print("std dev :", round(var ** 0.5, 1), "rupees")
variance: 2400.0 rupees squared
std dev : 49.0 rupees
Variance is in squared units (rupees², which means nothing to a chai seller), so the standard deviation, back in rupees, is what you report: roughly how far a typical value is from the mean. Small standard deviation: values huddle near the mean. Large: they are scattered.
Why squares and not absolute values? Squares have nice mathematical properties (variances of independent things add up, and they lead to the normal distribution's formulas). The absolute-value version exists too: the mean absolute deviation, and the more robust MAD below.
Population vs sample: ddof and Bessel's correction
print("pandas b.std() :", round(b.std(), 1))
print("numpy np.std(b) :", round(np.std(b.to_numpy()), 1))
print("pandas ddof=0 :", round(b.std(ddof=0), 1))
print("Pune vs Nashik :", round(a.std(), 1), round(b.std(), 1))
pandas b.std() : 51.6
numpy np.std(b) : 49.0
pandas ddof=0 : 49.0
Pune vs Nashik : 2.9 51.6
Same data, two answers, and both are right:
| Divide squared deviations by | Called | Default in | |
|---|---|---|---|
| The data is the whole population | n | population variance / std | NumPy np.var, np.std; scipy.stats.zscore |
| The data is a sample, used to estimate the population | n − 1 | sample variance / std | pandas .var(), .std(), describe() |
ddof means delta degrees of freedom: the functions divide by n − ddof. pandas uses ddof=1, NumPy ddof=0. Pass it explicitly when it matters.
Why n − 1? Proof by simulation:
import sys; sys.tracebacklimit = 0
import numpy as np
rng = np.random.default_rng(35)
pop = rng.normal(100, 50, 100_000) # every bill ever: the population
samples = rng.choice(pop, size=(20_000, 5)) # 20,000 samples of 5 bills
print("true variance :", round(pop.var()))
print("divide by n :", round(samples.var(axis=1, ddof=0).mean()))
print("divide by n - 1 :", round(samples.var(axis=1, ddof=1).mean()))
true variance : 2484
divide by n : 1988
divide by n - 1 : 2486
Dividing by n underestimates the true variance (by a factor of (n − 1)/n = 4/5 for samples of 5). A sample's own mean sits in the middle of its values, so distances to it are smaller than distances to the true population mean. Dividing by n − 1 corrects for that: Bessel's correction. On average the n − 1 version hits the target (it is unbiased). With large samples, n and n − 1 are nearly equal.
(axis=1 means "along each row", so each of the 20,000 rows, one sample, gets its own variance.)
Percentiles, quartiles and quantile()
Real-life example: exam rank. "You scored better than 90% of students" means you are at the 90th percentile. A percentile is the value below which that percentage of the data falls. Cut the sorted data into four equal groups and the three cut points are the quartiles: Q1 = 25th percentile, Q2 = 50th = the median, Q3 = 75th percentile. In code the percentage is written as a fraction between 0 and 1 and called a quantile: the 0.9 quantile is the 90th percentile.
print(b.sort_values().tolist())
print(b.quantile([0.25, 0.5, 0.75]))
print("90th percentile:", b.quantile(0.9))
[20, 40, 60, 80, 100, 100, 120, 140, 160, 180]
0.25 65.0
0.50 100.0
0.75 135.0
dtype: float64
90th percentile: 162.0
65 is not in the data: with 10 values, the 25th percentile's position is 0.25 × (10 − 1) = 2.25, between the 3rd value (60) and the 4th (80), so pandas interpolates a quarter of the way: 65. That is the default interpolation="linear"; other methods ("lower", "nearest", "midpoint") exist and give slightly different answers on small data.
Delivery apps state promises as percentiles ("90% of orders within 30 minutes") because a percentile tells customers what they can rely on, which an average cannot.
print(b.quantile(25))
ValueError: percentiles should all be in the interval [0, 1]
pandas quantile wants a fraction (0.25). NumPy's np.percentile(b, 25) is the one that takes 0–100.
Robust spread: IQR and MAD
- Interquartile range (IQR) = Q3 − Q1: the width of the middle 50% of the data.
- Median absolute deviation (MAD) = the median of |value − median|.
Both ignore how extreme the extremes are, just like the median.
from scipy import stats
b2 = pd.concat([b, pd.Series([5000])], ignore_index=True)
for s in (b, b2):
q1, q3 = s.quantile(0.25), s.quantile(0.75)
print("std", round(s.std(), 1), "| IQR", q3 - q1,
"| MAD", stats.median_abs_deviation(s))
std 51.6 | IQR 70.0 | MAD 40.0
std 1478.2 | IQR 80.0 | MAD 40.0
One 5,000-rupee order multiplies the standard deviation by almost 30, because squaring makes big deviations enormous. The IQR moves a little and the MAD not at all: they are robust measures of spread. Report them (with the median) when data has outliers or is skewed. (scipy.stats.median_abs_deviation has a scale="normal" option that multiplies by about 1.4826 so it estimates the standard deviation for bell-shaped data.)
Coefficient of variation
chai = pd.Series([20, 180, 60, 140, 100, 40, 160, 100, 80, 120])
catering = pd.Series([4500, 5600, 4800, 5200, 5000, 4400, 5500, 5000])
for name, s in [("chai", chai), ("catering", catering)]:
cv = s.std() / s.mean()
print(f"{name:9} mean {s.mean():6.0f} std {s.std():5.0f} CV {cv:.0%}")
chai mean 100 std 52 CV 52%
catering mean 5000 std 431 CV 9%
Catering's standard deviation (431) is bigger, but relative to its size it varies far less. The coefficient of variation (CV) = std ÷ mean, the spread as a share of the mean. Use it to compare spread between things on different scales (prices of phones vs chai, weights of elephants vs mice). It only makes sense for positive data measured from a true zero (not for temperatures in °C).
f"{cv:.0%}" formats a fraction as a percentage with no decimals.
The five-number summary and describe()
import matplotlib.pyplot as plt
rng = np.random.default_rng(35)
bills = pd.Series(np.r_[rng.normal(110, 25, 196).round(), [260, 275, 300, 15]])
print(bills.describe().round(1))
count 200.0
mean 114.6
std 33.8
min 15.0
25% 94.8
50% 111.5
75% 131.2
max 300.0
dtype: float64
std in describe() is the sample standard deviation (n − 1). The last five rows, minimum, Q1, median, Q3, maximum, are the five-number summary: centre, spread and extremes in five numbers. (np.r_[...] joins arrays and lists into one array.)
Reading a box plot
q1, med, q3 = bills.quantile([0.25, 0.5, 0.75])
iqr = q3 - q1
print("Q1", q1, "median", med, "Q3", q3, "IQR", iqr)
print("fences:", q1 - 1.5 * iqr, q3 + 1.5 * iqr)
fig, ax = plt.subplots(figsize=(10, 4.2))
ax.boxplot(bills, orientation="horizontal", widths=0.5)
for x, t, y in [(q1, "Q1", 1.36), (med, "median", 1.47), (q3, "Q3", 1.36)]:
ax.annotate(t, (x, 1.26), (x, y), ha="center", fontsize=13,
arrowprops={"arrowstyle": "->"})
ax.set(xlabel="bill (rupees)", yticks=[], ylim=(0.6, 1.56))
fig.tight_layout(); fig.savefig("box.png")
Q1 94.75 median 111.5 Q3 131.25 IQR 36.5
fences: 40.0 186.0

| Part | What it shows |
|---|---|
| Left edge of the box | Q1 (25th percentile) |
| Line inside the box | the median |
| Right edge of the box | Q3 (75th percentile) |
| Box width | the IQR: the middle 50% of values |
| Fences (not drawn) | Q1 − 1.5 × IQR and Q3 + 1.5 × IQR (here 40 and 186) |
| Whiskers | reach the furthest data point still inside the fences (here 40 and 185) |
| Dots | points beyond the fences: possible outliers (15, 192, 260, 275, 300) |
This is the IQR outlier rule from episode 23, drawn as a picture. A long whisker or box half on one side signals skew. (ax.annotate(text, xy, xytext, arrowprops=...) writes text at xytext with an arrow pointing to xy; in matplotlib 3.11 orientation="horizontal" replaces the older vert=False.)
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
b.quantile(25) |
ValueError: percentiles should all be in the interval [0, 1] |
quantile wants a fraction |
b.quantile(0.25) or np.percentile(b, 25) |
b.std() vs np.std(b) |
no error; 51.6 vs 49.0 | pandas ddof=1, NumPy ddof=0 |
pass ddof= explicitly |
(b - b.mean()).mean() as spread |
no error; always 0 | deviations cancel | square them: variance, std |
| std of data with outliers | no error; 51.6 → 1478.2 | squaring magnifies big values | IQR or MAD |
| reporting variance to a manager | no error; "2400 rupees squared" | squared units | report the standard deviation |
| comparing std across different scales | no error; misleading | bigger values have bigger std | coefficient of variation |
| range as the only spread measure | no error; one outlier decides it | uses only min and max | IQR / std |
Mini project: delivery reliability report
Three stalls deliver in about 12 minutes on average. Which one can promise a delivery time?
import numpy as np, pandas as pd
from scipy import stats
rng = np.random.default_rng(11)
df = pd.DataFrame({"stall": np.repeat(["Pune", "Nashik", "Mumbai"], 100),
"minutes": np.r_[rng.normal(12, 2, 100), rng.normal(12, 6, 100),
rng.gamma(2, 6, 100)].round(1)})
def iqr(s): return s.quantile(0.75) - s.quantile(0.25)
def p90(s): return s.quantile(0.9)
def mad(s): return stats.median_abs_deviation(s)
def cv(s): return s.std() / s.mean()
report = df.groupby("stall")["minutes"].agg(["mean", "median", "std", iqr, mad, p90, cv]).round(2)
print(report)
mean median std iqr mad p90 cv
stall
Mumbai 12.15 12.35 6.89 8.57 4.65 20.50 0.57
Nashik 12.05 12.10 5.59 7.33 3.50 19.27 0.46
Pune 12.05 12.10 1.84 2.48 1.25 14.41 0.15
All three average about 12 minutes, but Pune can honestly promise "90% within 15 minutes"; Mumbai would need to promise 21. Named functions passed to agg become column names (iqr, mad, p90, cv). (np.repeat repeats each label 100 times.)
Practice
1. Your daily screen time (hours) for a week: 3.5, 4.0, 2.5, 6.0, 5.0, 3.0, 4.5. Compute the standard deviation both ways. Which one estimates your usual variability?
Solution
import numpy as np, pandas as pd
screen = pd.Series([3.5, 4.0, 2.5, 6.0, 5.0, 3.0, 4.5]) # hours per day
print("pandas std (ddof=1):", round(screen.std(), 3))
print("numpy std (ddof=0):", round(np.std(screen.to_numpy()), 3))
print("pandas ddof=0 :", round(screen.std(ddof=0), 3))
pandas std (ddof=1): 1.205
numpy std (ddof=0): 1.116
pandas ddof=0 : 1.116
One week is a sample of your life, so the ddof=1 value (1.205 hours) is the better estimate.
2. Commute times (minutes): 32, 35, 31, 40, 33, 34, 58, 36, 33, 37. Give the median, the 90th percentile, the IQR and the box-plot fences. Is 58 an outlier?
Solution
import pandas as pd
commute = pd.Series([32, 35, 31, 40, 33, 34, 58, 36, 33, 37]) # minutes
print("median:", commute.median())
print("90th percentile:", commute.quantile(0.9))
q1, q3 = commute.quantile([0.25, 0.75])
print("IQR:", q3 - q1, "fences:", q1 - 1.5 * (q3 - q1), q3 + 1.5 * (q3 - q1))
median: 34.5
90th percentile: 41.8
IQR: 3.75 fences: 27.375 42.375
Yes: 58 is above the upper fence (42.375).
3. Phone prices vs chai prices: which is relatively more variable?
Solution
import pandas as pd
phone = pd.Series([15000, 18000, 22000, 16000, 19000])
chai = pd.Series([10, 15, 12, 20, 8])
for name, s in [("phone", phone), ("chai", chai)]:
print(name, round(s.std(), 1), f"CV {s.std() / s.mean():.0%}")
phone 2738.6 CV 15%
chai 4.7 CV 36%
Chai: its standard deviation is tiny in rupees but 36% of its mean.
4. Compute the sample variance of 4, 8, 6, 5, 3, 7, 9, 5 by hand in code and check against pandas.
Solution
import pandas as pd
s = pd.Series([4, 8, 6, 5, 3, 7, 9, 5])
d = s - s.mean()
print("mean:", s.mean(), "sum of deviations:", d.sum())
print("variance (n-1):", (d ** 2).sum() / (len(s) - 1), "| pandas:", s.var())
print("std:", round(s.std(), 3))
mean: 5.875 sum of deviations: 0.0
variance (n-1): 4.125 | pandas: 4.125
std: 2.031
Quick quiz
x = pd.Series([50, 50, 50, 50])
y = pd.Series([10, 30, 70, 90])
print(x.mean(), y.mean()) # 50.0 50.0
Which has the larger standard deviation? A. x B. y C. Equal, same mean D. Can't tell without a chart
Answer
B. y. All of x's deviations are 0, so its std is 0. y's deviations are ±20 and ±40: sample variance = (1600 + 400 + 400 + 1600) / 3 ≈ 1333.3, std ≈ 36.5.
Interview questions
- Why not use the mean of the deviations as spread? It is always zero; positive and negative deviations cancel.
- Variance vs standard deviation? Variance is the mean squared deviation (squared units); std is its square root (original units).
- Why divide by n − 1 for a sample? The sample mean is fitted to the sample, which makes deviations too small; n − 1 (Bessel's correction) makes the variance estimate unbiased.
- What is
ddof? Delta degrees of freedom: divide by n − ddof. pandas defaults to 1, NumPy to 0. - Percentile vs quantile? Same idea; percentiles use 0–100, quantiles use 0–1.
- What is the IQR and why use it? Q3 − Q1, the spread of the middle 50%; robust to outliers.
- What is MAD? Median of absolute deviations from the median; a very robust spread measure.
- When is the coefficient of variation useful? Comparing variability across variables with different scales or units.
- What are the five numbers in the five-number summary? Min, Q1, median, Q3, max.
- How are box plot whiskers and outlier dots defined (default)? Whiskers reach the furthest point within 1.5 × IQR of the box; points beyond are drawn individually.
- Why do companies quote the 90th or 99th percentile of latency or delivery time? Averages hide the slow cases that customers actually notice.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Spread / dispersion | How far values sit from the centre | Pune low, Nashik high |
| Measure of dispersion | A number that measures spread | range, std, IQR |
| Range | Max − min | Nashik: 160 |
| Outlier | A value far from the rest | the 300-rupee order |
| Deviation | Value − mean | 20 − 100 = −80 |
| Variance | Mean of squared deviations (units²) | 2400 rupees² |
| Standard deviation (std) | √variance, in the data's units | 49 rupees |
| Population variance | Divide by n | np.var(x) |
| Sample variance | Divide by n − 1 | s.var() |
| Bessel's correction | Using n − 1 for a sample to avoid underestimating | 1988 → 2486 |
| Unbiased | Correct on average over many samples | ddof=1 variance |
ddof |
Delta degrees of freedom: divide by n − ddof | s.std(ddof=0) |
| Percentile | Value below which p% of the data falls | 90th percentile = 162 |
| Quantile | Percentile written as a fraction 0–1 | quantile(0.9) |
| Quartiles (Q1, Q2, Q3) | 25th, 50th, 75th percentiles | 65, 100, 135 |
| Interpolation | Estimating a value between two data points | Q1 = 65 between 60 and 80 |
| Interquartile range (IQR) | Q3 − Q1 | 70 |
| Median absolute deviation (MAD) | Median of |x − median| | 40 |
| Robust | Barely affected by outliers | IQR, MAD, median |
| Coefficient of variation (CV) | std ÷ mean | chai 52%, catering 9% |
| Five-number summary | Min, Q1, median, Q3, max | describe() rows |
| Box plot | Chart of the five-number summary plus outliers | ax.boxplot(...) |
| Whisker | Line from the box to the furthest point inside the fences | 40 and 185 |
| Fence | Q1 − 1.5·IQR and Q3 + 1.5·IQR | 40 and 186 |
Summary
- The same average can hide very different data; you need a measure of dispersion.
- Range uses only two values. Variance averages squared deviations; the standard deviation brings it back to real units.
- Population: divide by n; sample: by n − 1 (Bessel's correction). pandas
ddof=1, NumPyddof=0. - Percentiles/quantiles and quartiles describe positions; pandas
quantiletakes 0–1. - IQR and MAD are robust spread; the CV compares spread across scales.
- A box plot draws the five-number summary and flags points beyond 1.5 × IQR.
- Next episode: the shape of data: skewness, kurtosis, z-scores and transformations.