These are the notes for Data Analytics with Python, episode 38. Episode 37 explored a dataset one column at a time (univariate analysis) and ended with two hypotheses: long distances make deliveries slow and slow deliveries get fewer ratings. Today you test them by looking at columns together: bivariate analysis (two variables) and multivariate analysis (three or more). You will meet the three kinds of pairs (numeric vs numeric with scatter plots, numeric vs categorical with grouped box plots, categorical vs categorical with crosstabs and stacked bars), segment comparison, interactions, and how to document EDA findings. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its message, as in the video.
The dataset
A fresh made-up export of 600 delivery orders from Ramesh's three cities, already cleaned with episode 37's checklist, with one new column: weather (dry or rain) when the order was placed. Every program starts with these lines:
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
rng = np.random.default_rng(38); n = 600 # made-up, cleaned
city = rng.choice(["Pune", "Nashik", "Nagpur"], n, p=[0.5, 0.3, 0.2])
km = (rng.gamma(2, np.where(city == "Nagpur", 2.4, 1.4)) + 0.5).round(1)
rain = rng.random(n) < 0.25
minutes = (15 + 4 * km + rain * (1 + 2.5 * km) + rng.gamma(2, 3, n)).round()
rating = (4.6 - 0.05 * (minutes - 30) + rng.normal(0, 0.6, n)).clip(1, 5).round()
rating[rng.random(n) < np.where(minutes > 45, 0.4, 0.1)] = np.nan
df = pd.DataFrame({"city": city, "km": km,
"weather": np.where(rain, "rain", "dry"),
"minutes": minutes, "rating": rating})
Why one column can't answer
print(df[["km", "minutes"]].describe().round(1))
km minutes
count 600.0 600.0
mean 3.7 38.9
std 2.3 12.8
min 0.5 20.0
25% 2.0 29.0
50% 3.2 37.0
75% 5.0 45.0
max 18.9 109.0
Distance has a median of 3.2 km and minutes a median of 37; both are right-skewed (skew 1.66 and 1.41). But nothing in these two summaries says whether longer distances take longer. A one-column summary throws away which value came from which row. The relationship lives in pairs: order 2 is 11.2 km and 67 minutes. To see a relationship, keep the pairs together.
Real-life example: a report card. The class average in Maths is univariate (one subject). "Do students who study more hours score more?" needs each student's hours and marks side by side: bivariate ("bi" = two, like a bicycle's two wheels). "…and does tuition change that?" adds a third variable: multivariate ("multi" = many).
| Pair of variables | Question it answers | Numbers | Chart |
|---|---|---|---|
| numeric vs numeric | does one rise with the other? | bands with pd.cut + groupby; correlation (episode 50) |
scatter plot |
| numeric vs categorical | does the number differ by group? | groupby(cat)[num].describe() |
grouped box plot, violin plot |
| categorical vs categorical | do the categories go together? | pd.crosstab(..., normalize="index") |
100% stacked bar, heatmap |
Numeric vs numeric: the scatter plot
print(df[["km", "minutes"]].head(4))
fig, ax = plt.subplots(figsize=(10, 4.5))
ax.scatter(df["km"], df["minutes"], alpha=0.4)
ax.set_xlabel("km"); ax.set_ylabel("minutes")
fig.tight_layout(); fig.savefig("scatter.png")
km minutes
0 1.5 27.0
1 4.4 38.0
2 11.2 67.0
3 3.2 47.0

Each dot is one order (one row = one pair). alpha=0.4 makes dots see-through, so dense areas look darker; without it, 600 overlapping dots hide each other (overplotting). The cloud rises to the right: the farther the order, the longer it takes. When one variable tends to go up as the other goes up, that is a positive relationship (a negative relationship is one going down as the other goes up). Episode 50 measures it with one number, the correlation coefficient.
A scatter plot also shows outliers in a pair: 67 minutes for 3 km is odd, even though neither 67 minutes nor 3 km is odd on its own.
To put numbers on the picture, cut the x variable into bands (bins) and summarise y per band:
df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
print(df.groupby("dist", observed=True)["minutes"].agg(["count", "median"]))
count median
dist
(0, 2] 154 27.0
(2, 4] 222 33.0
(4, 8] 199 46.0
(8, 30] 25 66.0
Up to 2 km: median 27 minutes; 2-4 km: 33; 4-8 km: 46; over 8 km: 66, but from only 25 orders, so trust that band less. Always print the count next to a group's average. Hypothesis 1 holds.
observed=True tells groupby to show only bands that actually occur (the bands are a categorical column, episode 20).
Numeric vs categorical: grouped box plots
print(df.groupby("city")["minutes"].describe().round(1))
fig, ax = plt.subplots(figsize=(10, 4.5))
df.boxplot(column="minutes", by="city", ax=ax, grid=False)
fig.suptitle(""); fig.tight_layout(); fig.savefig("box.png")
count mean std min 25% 50% 75% max
city
Nagpur 117.0 46.2 16.5 21.0 35.0 44.0 52.0 109.0
Nashik 178.0 37.7 10.6 21.0 30.0 35.5 42.0 71.0
Pune 305.0 36.8 11.3 20.0 28.0 35.0 43.0 72.0

groupby("city")["minutes"].describe() gives one row of summary statistics per city; df.boxplot(column=..., by=...) draws one box per group side by side: a grouped box plot. Nagpur's median is 44 minutes against about 35 for Nashik and Pune, and its box sits higher with more slow outliers. (fig.suptitle("") removes the automatic "Boxplot grouped by city" title.)
Segment comparison
The tempting conclusion: Nagpur's riders are slow. Each city is a segment: a group of rows that share a trait (city, plan, age group, device). Comparing segments is fair only if they are alike in everything else. Check the obvious other difference:
print(df.groupby("city")["km"].median())
city
Nagpur 4.7
Nashik 3.1
Pune 2.7
Name: km, dtype: float64
Nagpur's median order is 4.7 km away against 2.7 km in Pune: Nagpur's orders are farther, and farther means slower. Compare within the same distance band:
import numpy as np, pandas as pd
rng = np.random.default_rng(38); n = 600
city = rng.choice(["Pune", "Nashik", "Nagpur"], n, p=[0.5, 0.3, 0.2])
km = (rng.gamma(2, np.where(city == "Nagpur", 2.4, 1.4)) + 0.5).round(1)
rain = rng.random(n) < 0.25
minutes = (15 + 4 * km + rain * (1 + 2.5 * km) + rng.gamma(2, 3, n)).round()
df = pd.DataFrame({"city": city, "km": km, "minutes": minutes})
df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
print(df.pivot_table(index="dist", columns="city", values="minutes",
aggfunc="median", observed=True))
city Nagpur Nashik Pune
dist
(0, 2] 26.0 30.0 26.0
(2, 4] 35.0 32.5 33.0
(4, 8] 46.0 43.0 48.0
(8, 30] 76.0 61.5 59.0
Within each band up to 8 km, Nagpur is no slower than the others (26 vs 26-30, 35 vs 32.5-33, 46 vs 43-48). Most of the gap was distance, not the riders. A variable that hides behind a comparison like this is a confounder (confounding variable); episode 50 covers it properly. Rule: before blaming a segment, check what else differs between segments.
Categorical vs categorical: crosstab and stacked bars
Hypothesis 2: slow deliveries get fewer ratings. Make two yes/no columns: late (more than 45 minutes) and rated (rating not missing). Count every combination with pd.crosstab (episode 27); margins=True adds totals:
df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
print(pd.crosstab(df["late"], df["rated"], margins=True))
rated False True All
late
False 34 423 457
True 55 88 143
All 89 511 600
34 on-time orders and 55 late orders were not rated, but there are 457 on-time orders and only 143 late ones. Raw counts of groups this different in size are unfair (episode 36). You need shares within each row. A natural guess:
df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
share = pd.crosstab(df["late"], df["rated"], normalize="row")
KeyError: 'row'
The above exception was the direct cause of the following exception:
ValueError: Not a valid normalize argument
ValueError: Not a valid normalize argument. crosstab's normalize accepts "index" (each row sums to 1), "columns" (each column sums to 1), "all" (the whole table sums to 1), or True (same as "all"); there is no "row".
df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
share = pd.crosstab(df["late"], df["rated"], normalize="index")
print(share.round(2))
ax = share.plot.bar(stacked=True, figsize=(9, 4.5), rot=0)
ax.set_ylabel("share of orders")
ax.figure.tight_layout(); ax.figure.savefig("stacked.png")
rated False True
late
False 0.07 0.93
True 0.38 0.62

7% of on-time orders were not rated against 38% of late orders: more than five times as many. Hypothesis 2 holds. The chart is a 100% stacked bar (episode 32): each bar is one row of the table stretched to 100%. It is the standard chart for category vs category; a heatmap of the table works too for bigger tables.
Why it matters: if unhappy (late) customers skip the rating, the average rating hides your worst experiences and looks better than reality. This is non-response bias, which returns in episode 43.
Multivariate analysis and interactions
print(df.groupby("weather")["minutes"].mean().round(1))
weather
dry 36.0
rain 47.1
Name: minutes, dtype: float64
Dry: 36.0 minutes; rain: 47.1. "Rain adds about 11 minutes." But is that true for every order? Look at three variables at once (distance, minutes and weather) with a scatter plot coloured by weather:
fig, ax = plt.subplots(figsize=(10, 4.5))
for w, part in df.groupby("weather"):
ax.scatter(part["km"], part["minutes"], alpha=0.5, label=w)
ax.legend(title="weather"); ax.set_xlabel("km"); ax.set_ylabel("minutes")
fig.tight_layout(); fig.savefig("hue.png")

df.groupby("weather") gives one (name, part) pair per group, and each part is drawn in its own colour. At 1-2 km, orange (rain) and blue (dry) overlap; farther out, the orange cloud climbs faster. Measure it with a pivot_table (episode 27):
df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
t = df.pivot_table(index="dist", columns="weather", values="minutes",
aggfunc="median", observed=True)
t["rain adds"] = t["rain"] - t["dry"]
print(t)
ax = t[["dry", "rain"]].plot(marker="o", figsize=(9, 4.5))
ax.set_ylabel("median minutes")
ax.figure.tight_layout(); ax.figure.savefig("interaction.png")
weather dry rain rain adds
dist
(0, 2] 26.0 31.0 5.0
(2, 4] 32.0 41.0 9.0
(4, 8] 42.0 57.0 15.0
(8, 30] 59.0 91.0 32.0

Rain adds 5 minutes up to 2 km, 9 minutes at 2-4 km, 15 at 4-8 km and 32 beyond 8 km. There is no single "rain effect". If rain added a fixed time, the two lines would be parallel; they spread apart. When the effect of one variable depends on the value of another, that is an interaction: here, weather interacts with distance. The "11 minutes" average was right on average and wrong for almost every actual order.
Everyday interactions: a discount works better on weekends; a medicine's effect depends on age; an ad works on mobile but not on desktop. To find them, split one variable's effect by another (pivot tables, coloured scatter plots, small multiples from episode 32, seaborn hue from episode 33).
Documenting EDA findings
A finding nobody can find tomorrow is lost. Keep a findings log: one line per finding, each with the finding, the evidence, a caveat (what could make it wrong) and the next step.
| # | Finding | Evidence | Caveat | Next step |
|---|---|---|---|---|
| 1 | Longer distance → slower delivery | scatter; median 27 → 33 → 46 → 66 min by band | only 25 orders over 8 km | promise times by distance |
| 2 | Late orders go unrated far more often | crosstab: 38% vs 7% not rated | rating may be missing for other reasons too | treat average rating as optimistic; send a follow-up for late orders |
| 3 | Rain adds 5 to 32 minutes depending on distance | pivot table, coloured scatter | 25% of orders in rain; long-distance rain orders are few | promise times by distance and weather |
| 4 | Nagpur looks slower, but its orders are farther | city box plots; km medians; within-band medians | small numbers in the 8+ km band | don't blame riders; check zones |
In a Jupyter notebook (episode 3) this lives in a markdown cell at the top or the end; in a team it goes into the report or ticket. The log is what turns EDA into decisions.
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
pd.crosstab(a, b, normalize="row") |
ValueError: Not a valid normalize argument |
rows are called "index" |
normalize="index" (or "columns", "all") |
| comparing raw crosstab counts | no error; unfair comparison | groups differ in size | normalise by row |
scatter of many points without alpha |
no error; dense areas look the same as sparse ones | overplotting | alpha=0.3, smaller dots, or bands |
| averages per group without counts | no error; trusting tiny groups | a group of 5 is noise | print count next to every mean/median |
| "Nagpur riders are slow" | no error; wrong blame | Nagpur orders are farther | compare within distance bands |
| reporting one average effect | no error; hides an interaction | effect depends on another variable | split by the other variable |
two separate describe() calls for a relationship |
no error; no answer | summaries forget the pairing | scatter, groupby, crosstab |
Mini project: is weekend demand different?
import numpy as np, pandas as pd
rng = np.random.default_rng(5)
days = pd.date_range("2026-01-01", periods=120, freq="D")
weekend = days.dayofweek >= 5
promo = rng.random(120) < 0.3
orders = (200 + 60 * weekend + promo * np.where(weekend, 80, 20) + rng.normal(0, 15, 120)).round()
df = pd.DataFrame({"day": days, "weekend": weekend, "promo": promo, "orders": orders})
print(df.groupby("weekend")["orders"].agg(["count", "median"]))
t = df.pivot_table(index="weekend", columns="promo", values="orders", aggfunc="median")
t["promo adds"] = t[True] - t[False]
print(t)
count median
weekend
False 86 206.5
True 34 264.5
promo False True promo adds
weekend
False 200.0 219.0 19.0
True 261.5 338.0 76.5
Weekends have more orders, and the promotion adds far more on weekends than on weekdays: an interaction between weekend and promo. A findings-log line: promo adds about 4× more orders on weekends than weekdays (pivot table, 120 days, 33 of them promo days); next step: run promos on weekends.
Practice
1. Ten students' study hours and marks. Draw (or just print) the pairs sorted by hours. Is the relationship positive or negative?
Solution
import pandas as pd
s = pd.DataFrame({"hours": [1, 3, 2, 5, 4, 6, 2, 7, 3, 5], "marks": [45, 60, 52, 75, 66, 80, 50, 88, 58, 71]})
print(s.sort_values("hours").to_string(index=False))
hours marks
1 45
2 52
2 50
3 60
3 58
4 66
5 71
5 75
6 80
7 88
Marks rise with hours: a positive relationship. s.plot.scatter(x="hours", y="marks") draws it.
2. Plan (gold/basic) vs renewed (yes/no): gold 80 customers, 64 renewed; basic 420 customers, 210 renewed. Build the crosstab with normalize="index".
Solution
import pandas as pd
plan = ["gold"] * 80 + ["basic"] * 420
renewed = ["yes"] * 64 + ["no"] * 16 + ["yes"] * 210 + ["no"] * 210
print(pd.crosstab(pd.Series(plan, name="plan"), pd.Series(renewed, name="renewed"), normalize="index"))
renewed no yes
plan
basic 0.5 0.5
gold 0.2 0.8
80% of gold customers renewed vs 50% of basic: compare shares, not the raw 64 vs 210.
3. Why is "Nagpur's median delivery is 44 minutes, so Nagpur riders are slow" a weak conclusion? What would you check?
Solution
Nagpur may differ in other ways (distance, traffic, weather, order size). Check those variables per city, then compare cities within the same distance band (as above): the gap mostly disappears.
Quick quiz
# Question: do customers on the "gold" plan
# (plan: gold / basic) rate us higher
# (rated: yes / no, then rating 1-5)?
# Step 1: plan vs rated, both categories
Which tool first? A. A scatter plot B. pd.crosstab(..., normalize="index") C. df["plan"].describe() D. A histogram of rating
Answer
B. Two categories → a crosstab, normalised by row because the plans have different numbers of customers. A scatter plot needs two numbers; describe() and a histogram look at one column only.
Interview questions
- Univariate vs bivariate vs multivariate analysis? One variable; two variables paired by row; three or more together.
- Which chart for each pair of variable types? Numeric-numeric: scatter; numeric-categorical: grouped box/violin plot; categorical-categorical: crosstab with a 100% stacked bar or heatmap.
- What does
normalize="index"do inpd.crosstab? Divides each row by its total so rows sum to 1, giving shares within each row. - What is overplotting and how do you fix it? Overlapping points hide density; use transparency (
alpha), smaller markers, sampling, hexbin plots or bands. - What is an interaction? When the effect of one variable on the outcome depends on another variable; e.g. rain adds more delay on longer trips.
- How do you compare segments fairly? Check what else differs between them, and compare within groups that are alike on those other variables.
- How do you document EDA findings? A findings log: finding, evidence, caveat, next step.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Univariate analysis | One variable at a time | describe() of minutes |
| Bivariate analysis | Two variables together, paired by row | km vs minutes |
| Multivariate analysis | Three or more variables together | km, minutes and weather |
| Scatter plot | One dot per row at (x, y) | km across, minutes up |
| Overplotting | Points drawn on top of each other hide how many there are | 600 dots |
alpha |
Transparency of a marker (0 invisible, 1 solid) | alpha=0.4 |
| Positive / negative relationship | One tends to rise / fall as the other rises | distance and minutes: positive |
| Band (bin) | A range of values grouped together | 2-4 km |
| Grouped box plot | One box plot per category, side by side | minutes per city |
| Segment | A group of rows sharing a trait | the Nagpur orders |
| Segment comparison | Comparing a measure across segments | median minutes per city |
| Confounder | A hidden variable that affects both sides of a comparison (episode 50) | distance behind "Nagpur is slow" |
| Crosstab (contingency table) | Counts of every combination of two categories | late × rated |
normalize |
Turn crosstab counts into shares: "index", "columns" or "all" |
normalize="index" |
| 100% stacked bar | Each bar is one group split into shares that add to 100% | rated share, on time vs late |
| Pivot table | A table with one variable down the side, one across, and a summary in each cell | distance band × weather |
| Interaction | The effect of one variable depends on the value of another | rain +5 min short, +32 min long |
| Findings log | Written list of findings with evidence, caveat and next step | the table above |
| Caveat | A warning about what could make a finding wrong | only 25 orders over 8 km |
Summary
- One-column summaries forget the pairing; relationships need the rows kept together.
- Bivariate: numeric-numeric → scatter + bands; numeric-categorical →
groupby().describe()+ grouped box plot; categorical-categorical →crosstab(normalize="index")+ 100% stacked bar. - Segments: before blaming one, check what else differs; compare within like groups.
- Multivariate: colour, split or pivot by a third variable; when an effect changes across groups, it's an interaction.
- Write every finding in a findings log: finding, evidence, caveat, next step.
- Next episode: Probability 1: events, the rules, and conditional probability, the maths behind "38% of late orders".