SyntaxPilot
Courses/Data Analytics with Python/Lesson 34
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 34 of 90

Descriptive statistics 1: averages

descriptive vs inferential statisticspopulationsamplemeasures of central tendencymeanmedianmodeweighted meantrimmed meangeometric meanharmonic meaneffect of outlierschoosing the right average

These are the notes for Data Analytics with Python, episode 34. A chai stall owner wants one number for "a typical bill". You will invent that number yourself, watch it lie, and fix it: the mean, the median, the mode, the weighted mean, the trimmed mean, the geometric mean and the harmonic mean. Before that you meet the first words of statistics: population, sample, descriptive and inferential statistics, and measures of central tendency. Every term is defined the first time it appears and collected in the Glossary at the end.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). The outputs are copied from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its last line, exactly as in the video.

The problem: one number for "typical"

Ramesh runs a chai stall. He wants to write in his diary: "a typical bill at my stall is ___ rupees." He does not want a list of 100 numbers. He wants one number that sits in the middle of the data and represents it.

That number is called a measure of central tendency: central because it is the centre of the data, tendency because values tend to gather around it. In everyday language, an average. There are at least seven useful averages, and each is right for a different kind of data.

Statistics, population and sample

Statistics is the science of collecting data, summarising it, and drawing conclusions from it.

Real-life example: tasting chai from the pot.

At the chai stall In statistics Meaning
The whole pot of chai Population every item you care about (every order the stall will ever serve)
One spoonful to taste Sample the part of the population you actually measured (today's 10 bills)
"This spoon is sweet" Descriptive statistics summarising the data you have with numbers and charts
"So the whole pot is sweet" Inferential statistics concluding something about the population from a sample

Today's averages are descriptive statistics. Inference (confidence intervals, hypothesis tests) comes later in the course, because one spoon can mislead.

A number computed from a population is often called a parameter; a number computed from a sample is a statistic. You will meet both words properly in episode 43.

Mean and median

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
bills = pd.Series([80, 95, 100, 110, 120, 90, 105, 100, 85, 115])
print("sum   :", bills.sum(), "count:", bills.count())
print("mean  :", bills.mean())
print("sorted:", sorted(bills))
print("median:", bills.median())
sum   : 1000 count: 10
mean  : 100.0
sorted: [80, 85, 90, 95, 100, 100, 105, 110, 115, 120]
median: 100.0
  • Mean (arithmetic mean): add every value and divide by how many there are. 1000 / 10 = 100. Formula: mean = (x₁ + x₂ + … + xₙ) / n.
  • Median: sort the values and take the middle one. With an odd count there is one middle value. With an even count (like 10) there are two middle values (the 5th and 6th, both 100 here) and the median is the mean of those two.

Here mean and median agree: a typical bill is 100 rupees.

One outlier breaks the mean

An office orders chai for a conference: one bill of 5,000 rupees.

bills = pd.concat([bills, pd.Series([5000])], ignore_index=True)
print("orders:", len(bills))
print("mean  :", round(bills.mean(), 1))
print("median:", bills.median())
orders: 11
mean  : 545.5
median: 100.0

The mean jumps to 545.5, a bill nobody paid. Why? The mean uses the size of every value: the 5,000 adds 4,900 extra rupees to the sum, and that is shared across all 11 orders.

A value far away from the rest is an outlier. The median stays at 100 because it only uses the order of the values; the 5,000 is simply the last item in the sorted list. Making it 5 lakh would not move the median. A statistic that outliers barely affect is called robust: the median is robust, the mean is not.

Real-life example: a billionaire walks into the chai stall. Ten customers earn about 30,000 rupees a month each. A billionaire earning 100 crore a month sits down. The mean income of the 11 people becomes about 9 crore; the median (the 6th person in sorted order) is still about 30,000. A headline saying "the average customer here earns 9 crore a month" is true for the mean and useless. That is why incomes, house prices and delivery times are usually reported with the median.

Skewed data: mean vs median

import matplotlib.pyplot as plt
rng = np.random.default_rng(34)
salary = pd.Series(rng.lognormal(10.3, 0.6, 1000).round(-2))
print("mean  :", round(salary.mean()))
print("median:", salary.median())
print("above mean:", (salary > salary.mean()).mean())
fig, ax = plt.subplots(figsize=(10, 4.5))
ax.hist(salary, bins=60, color="#9aa5b8")
ax.axvline(salary.mean(), color="#e4572e", lw=3, label="mean")
ax.axvline(salary.median(), color="#2a9d8f", lw=3, label="median")
ax.set(xlabel="monthly salary (rupees)", ylabel="people")
ax.legend(); fig.savefig("skew.png")
mean  : 34244
median: 29300.0
above mean: 0.4

skew.png

Most salaries are modest and a few are very high, so the histogram has a long thin tail to the right. A distribution whose tail is longer on one side is skewed (episode 36 measures it). The tail pulls the mean (red) toward it; the median (green) stays with the middle person. Only 40% of people earn more than the "average" salary.

(salary > salary.mean()).mean() works because comparing gives a Series of True/False, and the mean of booleans is the share of True values.

Rule of thumb: for symmetric data the mean and median are almost equal; for skewed data or data with outliers, the median is the better "typical value".

Mode: the average for categories

size = pd.Series(["small", "large", "small", "medium",
                  "small", "large", "small"])
print(size.mean())
TypeError: Cannot perform reduction 'mean' with string dtype

A reduction is any operation that boils many values down to one (sum, mean, max). You cannot add "small" + "large". Cup size is categorical data (episode 2), and for categories the average is the mode: the most frequent value.

print(size.value_counts())
print(size.mode())
small     4
large     2
medium    1
Name: count, dtype: int64
0    small
dtype: str

mode() returns a Series, not a single value, because there can be a tie:

sugar = pd.Series([1, 1, 2, 2, 3])
print(sugar.mode().tolist())
[1, 2]

Two modes means the data is bimodal (two peaks), which often means two different groups are mixed together. Use mode()[0] when you need just one value. The mode also works for numbers, but it is most useful for categories and for counts with few distinct values.

Weighted mean

Three stalls report their own average rating and their number of orders:

r = pd.DataFrame({"stall": ["Pune", "Nashik", "Mumbai"],
                  "avg_rating": [4.8, 4.0, 3.5],
                  "orders": [10, 200, 90]})
print(r)
print("mean of averages:", round(r["avg_rating"].mean(), 2))
    stall  avg_rating  orders
0    Pune         4.8      10
1  Nashik         4.0     200
2  Mumbai         3.5      90
mean of averages: 4.1

4.1 is wrong: Pune's 4.8 came from only 10 orders but counts as much as Nashik's 200. The fix is a weight, a number saying how much each value should count. The weighted mean = Σ(value × weight) ÷ Σ weight.

total = (r["avg_rating"] * r["orders"]).sum()
print("by hand :", round(total / r["orders"].sum(), 2))
w = np.average(r["avg_rating"], weights=r["orders"])
print("weighted:", round(w, 2))
by hand : 3.88
weighted: 3.88

3.88 is exactly the mean rating of all 300 individual orders. (The plain mean, np.mean, is just a weighted mean where every weight is 1.) Exam percentages with "theory 70%, practical 30%" are weighted means too.

Weights must line up one-to-one with the values:

np.average(r["avg_rating"], weights=[10, 200])
TypeError: Axis must be specified when shapes of a and weights differ.

Trimmed mean

Real-life example: diving judges. Several judges score a dive; the highest and lowest scores are thrown away and the rest are averaged, so no single judge can swing the result.

A trimmed mean sorts the values, cuts a fixed share from each end, and averages the middle. SciPy (the scientific Python library) has it in scipy.stats:

from scipy import stats
bills = pd.Series([80, 95, 100, 110, 120, 90, 105, 100, 85, 115, 5000])
print("mean        :", round(bills.mean(), 1))
print("trimmed 10% :", round(stats.trim_mean(bills, 0.1), 1))
print("median      :", bills.median())
mean        : 545.5
trimmed 10% : 102.2
median      : 100.0

0.1 means 10% from each end. 10% of 11 values is 1.1, which SciPy rounds down to 1 value per end, so it drops 80 and 5,000 and averages the other nine. The trimmed mean ignores the outlier but still uses nine values, not just one middle value. (The median is the most extreme trimmed mean: trim almost 50% from each side.)

Geometric mean: for growth

Sales grow 50% in year 1 (×1.5) and fall 50% in year 2 (×0.5).

import numpy as np
from scipy import stats
growth = [1.5, 0.5]          # year 1: +50%, year 2: -50%
avg = np.mean(growth)
print("arithmetic mean:", avg)
print("start 100, after 2 years:", 100 * 1.5 * 0.5)
print("what the mean predicts  :", 100 * avg * avg)
arithmetic mean: 1.0
start 100, after 2 years: 75.0
what the mean predicts  : 100.0

The arithmetic mean says "no change", but 100 became 75. Growth factors multiply, they do not add. The right average is the factor that, applied every year, gives the same final result: multiply all values and take the n-th root. That is the geometric mean: (x₁ × x₂ × … × xₙ)^(1/n).

g = stats.gmean(growth)
print("geometric mean:", round(g, 4))
print("check: 100 * g * g =", round(100 * g * g, 1))
print("by hand:", round((1.5 * 0.5) ** (1 / 2), 4))
geometric mean: 0.866
check: 100 * g * g = 75.0
by hand: 0.866

So the honest average is a fall of about 13.4% a year. Use the geometric mean for growth rates, investment returns, inflation and anything that compounds. It needs positive values: stats.gmean([1.5, -0.5]) prints RuntimeWarning: invalid value encountered in log and returns nan, and any zero makes it 0.

Harmonic mean: for rates

Ramesh drives 120 km from Pune to Nashik at 60 km/h and back at 40 km/h.

from scipy import stats
km = 120                       # Pune to Nashik, one way
hours = km / 60 + km / 40       # there at 60, back at 40
print("hours :", hours)
print("speed :", 2 * km / hours, "km/h")
print("mean  :", (60 + 40) / 2)
print("hmean :", round(stats.hmean([60, 40]), 1))
hours : 5.0
speed : 48.0 km/h
mean  : 50.0
hmean : 48.0

Average speed is total distance ÷ total time = 240 / 5 = 48, not 50, because he spent more time driving slowly. The harmonic mean is n ÷ Σ(1/x): flip each value, take the ordinary mean, flip back. Use it to average rates over the same distance or the same amount (speeds, price per item when you spend the same money each time, the F1 score in machine learning).

For positive numbers that are not all equal: harmonic mean < geometric mean < arithmetic mean.

Missing values: pandas vs NumPy

s = pd.Series([100, np.nan, 120])      # one bill was never typed in
print("pandas mean :", s.mean())
print("numpy mean  :", np.mean(s.to_numpy()))
print("np.nanmean  :", np.nanmean(s.to_numpy()))
pandas mean : 110.0
numpy mean  : nan
np.nanmean  : 110.0

pandas skips NaN by default (skipna=True); plain NumPy lets one NaN poison the result (episode 10); np.nanmean skips it. Always know which one you are calling, and check how many values were missing (s.isna().sum()).

Choosing the right average

Data Use Example
Categories (text labels) Mode most popular cup size, city
Numbers, roughly symmetric, no outliers Mean heights, test scores of a normal class
Skewed numbers or outliers Median salaries, house prices, delivery times
Mean-like, but ignore a few wild values Trimmed mean judges' scores, noisy sensor readings
Combining groups of different sizes Weighted mean overall rating from stall averages, exam weightage
Growth factors, returns, ratios that compound Geometric mean yearly sales growth, investment returns
Rates over the same distance or amount Harmonic mean average speed of a round trip

When unsure, report both mean and median. A big gap between them is itself a signal of skew or outliers: draw a histogram.

Common mistakes

Code / habit Real error / symptom Why Fix
size.mean() on a text column TypeError: Cannot perform reduction 'mean' with string dtype words have no size to add size.mode() or value_counts()
np.average(r["avg_rating"], weights=[10, 200]) TypeError: Axis must be specified when shapes of a and weights differ. 3 values, 2 weights one weight per value
r["avg_rating"].mean() (averaging averages) no error; 4.1 instead of 3.88 small groups count as much as big ones np.average(..., weights=orders)
np.mean([1.5, 0.5]) for growth no error; "no change" but 100 → 75 growth multiplies stats.gmean(growth)
stats.gmean([1.5, -0.5]) RuntimeWarning: invalid value encountered in log, result nan log of a negative number use growth factors (1 + rate), all positive
Mean of salaries as "typical salary" no error; most people earn less than it skew pulls the mean report the median
np.mean on data with NaN no error; result nan NaN propagates in NumPy np.nanmean or pandas .mean()
size.mode() used as a single value a Series is printed there can be several modes size.mode()[0] or .tolist()

Mini project: a stall summary

Build one table with the mean, median, 10% trimmed mean and order count per stall, the most common cup size, and check that the weighted mean of the stall means equals the overall mean.

import numpy as np, pandas as pd
from scipy import stats
rng = np.random.default_rng(7)
orders = pd.DataFrame({
    "stall": rng.choice(["Pune", "Nashik", "Mumbai"], 300),
    "bill": rng.gamma(3, 40, 300).round(),
    "size": rng.choice(["small", "medium", "large"], 300, p=[0.5, 0.3, 0.2])})
orders.loc[[5, 77], "bill"] = [4500, 3800]          # two catering orders
summary = orders.groupby("stall")["bill"].agg(
    mean="mean", median="median",
    trimmed=lambda s: stats.trim_mean(s, 0.1), orders="count").round(1)
print(summary)
print("most common size:", orders["size"].mode()[0])
overall = np.average(summary["mean"], weights=summary["orders"])
print("weighted overall mean:", round(overall, 1), "| direct:", round(orders["bill"].mean(), 1))
         mean  median  trimmed  orders
stall                                 
Mumbai  185.3   103.0    106.0     105
Nashik  105.4    94.0     96.2     109
Pune    108.8    88.5     98.5      86
most common size: small
weighted overall mean: 134.3 | direct: 134.3

Mumbai's mean (185.3) is far above its median (103) and trimmed mean (106): the two catering orders landed there. The weighted mean of the stall means equals the mean of all 300 orders, exactly as the weighted-mean section promised.

Practice

1. Monthly rents in a building: 12,000, 15,000, 11,000, 14,000, 13,000 and one penthouse at 95,000. Which average describes a typical flat? Compute both.

Solution
import pandas as pd
rent = pd.Series([12000, 15000, 11000, 14000, 13000, 95000])
print("mean  :", round(rent.mean(), 1))
print("median:", rent.median())
mean  : 26666.7
median: 13500.0

The median, 13,500. The penthouse drags the mean to 26,667, which is more than every other flat pays.

2. Your theory mark is 78 and practical is 92. Theory counts 70% and practical 30%. What is your final mark?

Solution
import numpy as np
marks = {"theory": 78, "practical": 92}
weights = {"theory": 70, "practical": 30}
print(round(np.average(list(marks.values()), weights=list(weights.values())), 1))
82.2

(78 × 70 + 92 × 30) / 100 = 82.2. Weights do not have to add up to 1 or 100; np.average divides by their sum.

3. Sales grew 10%, then 20%, then fell 10%. What is the average yearly growth? Check it reproduces the final value.

Solution
from scipy import stats
growth = [1.10, 1.20, 0.90]
g = stats.gmean(growth)
print(round(g, 4), f"{g - 1:.2%} per year")
print(round(100 * 1.10 * 1.20 * 0.90, 2), round(100 * g ** 3, 2))
1.0591 5.91% per year
118.8 118.8

About 5.91% a year. (The arithmetic mean of the rates would say 6.67%, which overstates the growth.)

4. Customers' cities: Pune, Delhi, Pune, Delhi, Goa. What is the mode?

Solution
import pandas as pd
city = pd.Series(["Pune", "Delhi", "Pune", "Delhi", "Goa"])
print(city.mode().tolist())
['Delhi', 'Pune']

Two modes (a tie), returned in sorted order. The data is bimodal.

5. A runner does a 30 km route at 30 km/h and returns at 60 km/h. Average speed?

Solution
from scipy import stats
print(round(stats.hmean([30, 60]), 1))
print(round(2 * 30 / (30 / 30 + 30 / 60), 1))
40.0
40.0

40 km/h, not 45: 60 km in 1.5 hours.

Quick quiz

delivery_minutes = pd.Series(
    [22, 25, 24, 27, 23, 26, 180])
# 180 = the rider got lost

Which average best describes a typical delivery? A. The mean, about 46.7 B. The median, 25 C. The mode D. The geometric mean

Answer

B. The median, 25 minutes. The single 180 pulls the mean to 327 / 7 ≈ 46.7, a time no delivery took. Every value appears once, so the mode is not useful, and the geometric mean is for growth factors.

Interview questions

  1. Mean vs median: when do you use which? Mean for symmetric data without outliers; median for skewed data or data with outliers, because it is robust.
  2. What does it mean when the mean is much larger than the median? The data is right-skewed or has large outliers.
  3. Can you take the mean of categorical data? No; use the mode (or proportions per category).
  4. Why can a dataset have more than one mode? Several values can tie for the highest frequency; two modes = bimodal, often two mixed groups.
  5. What is a weighted mean and when is it needed? Σ(value × weight) / Σ weight; whenever values represent groups of different sizes or have different importance.
  6. Why is averaging averages dangerous? Small groups get the same influence as big ones (closely related to Simpson's paradox).
  7. What is a trimmed mean? The mean after removing a fixed percentage from each end of the sorted data; a compromise between mean and median.
  8. When would you use a geometric mean? For multiplicative quantities: growth rates, returns, ratios.
  9. When would you use a harmonic mean? For averaging rates over a fixed quantity, e.g. speeds over the same distance; also the F1 score.
  10. Population vs sample? The population is everything you care about; a sample is the subset you measured. Descriptive statistics summarise a sample; inferential statistics generalise from it.
  11. How do pandas and NumPy treat NaN in a mean? pandas skips it by default; NumPy returns nan unless you use np.nanmean.

Glossary

Term Meaning Example
Statistics The science of collecting, summarising and drawing conclusions from data describing today's bills
Population Every item you care about every order the stall will ever serve
Sample The part of the population you actually measured today's 10 bills
Parameter / statistic A number describing a population / a sample true mean bill / today's mean bill
Descriptive statistics Summarising data you have with numbers and charts "today's median bill is 100"
Inferential statistics Concluding about a population from a sample "all our bills average about 100"
Measure of central tendency One number representing the centre ("typical value") of data mean, median, mode
Average Everyday word for a measure of central tendency "average bill"
Mean (arithmetic mean) Sum of values ÷ number of values 1000 / 10 = 100
Median Middle value of the sorted data (mean of the two middle values for an even count) 100
Mode Most frequent value; may be several "small"
Outlier A value far away from the rest the 5,000 rupee order
Robust Barely affected by outliers the median
Skewed A distribution with a longer tail on one side salaries
Reduction An operation that turns many values into one sum, mean, max
Categorical data Values that are labels, not quantities cup size
Bimodal Having two modes / two peaks sugar spoons 1 and 2
Weight A number saying how much a value counts number of orders
Weighted mean Σ(value × weight) ÷ Σ weight 3.88
Trimmed mean Mean after cutting a % from each end stats.trim_mean(x, 0.1) → 102.2
SciPy Scientific Python library; scipy.stats has statistics functions from scipy import stats
Growth factor 1 + growth rate +50% → 1.5
Geometric mean n-th root of the product of n values stats.gmean([1.5, 0.5]) → 0.866
Harmonic mean n ÷ Σ(1/x) stats.hmean([60, 40]) → 48
np.average NumPy mean with optional weights= np.average(x, weights=w)
NaN "Not a Number", NumPy/pandas marker for a missing value np.nan
np.nanmean Mean that ignores NaN 110.0

Summary

  • Statistics starts with a population (everything) and a sample (what you measured); averages are descriptive statistics.
  • A measure of central tendency is one number for "typical". The mean uses every value's size, so outliers drag it; the median uses only order, so it is robust.
  • In skewed data the mean moves toward the tail; report the median for salaries, prices and times.
  • The mode is the average for categories; mode() returns a Series because of ties.
  • Weighted mean for groups of different sizes, trimmed mean to ignore extremes, geometric mean for growth, harmonic mean for rates.
  • pandas skips NaN by default, NumPy does not.
  • Next episode: one number for the centre isn't enough. Two stalls can have the same average and completely different days: spread and percentiles.