SyntaxPilot
Courses/Data Analytics with Python/Lesson 38
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 38 of 90

EDA 2: relationships between variables

bivariate analysismultivariate analysisnumeric vs numeric (scatter)numeric vs categorical (grouped box plots)categorical vs categorical (crosstab, stacked bar)segment comparisoninteractiondocumenting EDA findings

These are the notes for Data Analytics with Python, episode 38. Episode 37 explored a dataset one column at a time (univariate analysis) and ended with two hypotheses: long distances make deliveries slow and slow deliveries get fewer ratings. Today you test them by looking at columns together: bivariate analysis (two variables) and multivariate analysis (three or more). You will meet the three kinds of pairs (numeric vs numeric with scatter plots, numeric vs categorical with grouped box plots, categorical vs categorical with crosstabs and stacked bars), segment comparison, interactions, and how to document EDA findings. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its message, as in the video.

The dataset

A fresh made-up export of 600 delivery orders from Ramesh's three cities, already cleaned with episode 37's checklist, with one new column: weather (dry or rain) when the order was placed. Every program starts with these lines:

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
rng = np.random.default_rng(38); n = 600             # made-up, cleaned
city = rng.choice(["Pune", "Nashik", "Nagpur"], n, p=[0.5, 0.3, 0.2])
km = (rng.gamma(2, np.where(city == "Nagpur", 2.4, 1.4)) + 0.5).round(1)
rain = rng.random(n) < 0.25
minutes = (15 + 4 * km + rain * (1 + 2.5 * km) + rng.gamma(2, 3, n)).round()
rating = (4.6 - 0.05 * (minutes - 30) + rng.normal(0, 0.6, n)).clip(1, 5).round()
rating[rng.random(n) < np.where(minutes > 45, 0.4, 0.1)] = np.nan
df = pd.DataFrame({"city": city, "km": km,
                   "weather": np.where(rain, "rain", "dry"),
                   "minutes": minutes, "rating": rating})

Why one column can't answer

print(df[["km", "minutes"]].describe().round(1))
          km  minutes
count  600.0    600.0
mean     3.7     38.9
std      2.3     12.8
min      0.5     20.0
25%      2.0     29.0
50%      3.2     37.0
75%      5.0     45.0
max     18.9    109.0

Distance has a median of 3.2 km and minutes a median of 37; both are right-skewed (skew 1.66 and 1.41). But nothing in these two summaries says whether longer distances take longer. A one-column summary throws away which value came from which row. The relationship lives in pairs: order 2 is 11.2 km and 67 minutes. To see a relationship, keep the pairs together.

Real-life example: a report card. The class average in Maths is univariate (one subject). "Do students who study more hours score more?" needs each student's hours and marks side by side: bivariate ("bi" = two, like a bicycle's two wheels). "…and does tuition change that?" adds a third variable: multivariate ("multi" = many).

Pair of variables Question it answers Numbers Chart
numeric vs numeric does one rise with the other? bands with pd.cut + groupby; correlation (episode 50) scatter plot
numeric vs categorical does the number differ by group? groupby(cat)[num].describe() grouped box plot, violin plot
categorical vs categorical do the categories go together? pd.crosstab(..., normalize="index") 100% stacked bar, heatmap

Numeric vs numeric: the scatter plot

print(df[["km", "minutes"]].head(4))
fig, ax = plt.subplots(figsize=(10, 4.5))
ax.scatter(df["km"], df["minutes"], alpha=0.4)
ax.set_xlabel("km"); ax.set_ylabel("minutes")
fig.tight_layout(); fig.savefig("scatter.png")
     km  minutes
0   1.5     27.0
1   4.4     38.0
2  11.2     67.0
3   3.2     47.0

scatter.png

Each dot is one order (one row = one pair). alpha=0.4 makes dots see-through, so dense areas look darker; without it, 600 overlapping dots hide each other (overplotting). The cloud rises to the right: the farther the order, the longer it takes. When one variable tends to go up as the other goes up, that is a positive relationship (a negative relationship is one going down as the other goes up). Episode 50 measures it with one number, the correlation coefficient.

A scatter plot also shows outliers in a pair: 67 minutes for 3 km is odd, even though neither 67 minutes nor 3 km is odd on its own.

To put numbers on the picture, cut the x variable into bands (bins) and summarise y per band:

df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
print(df.groupby("dist", observed=True)["minutes"].agg(["count", "median"]))
         count  median
dist                  
(0, 2]     154    27.0
(2, 4]     222    33.0
(4, 8]     199    46.0
(8, 30]     25    66.0

Up to 2 km: median 27 minutes; 2-4 km: 33; 4-8 km: 46; over 8 km: 66, but from only 25 orders, so trust that band less. Always print the count next to a group's average. Hypothesis 1 holds.

observed=True tells groupby to show only bands that actually occur (the bands are a categorical column, episode 20).

Numeric vs categorical: grouped box plots

print(df.groupby("city")["minutes"].describe().round(1))
fig, ax = plt.subplots(figsize=(10, 4.5))
df.boxplot(column="minutes", by="city", ax=ax, grid=False)
fig.suptitle(""); fig.tight_layout(); fig.savefig("box.png")
        count  mean   std   min   25%   50%   75%    max
city                                                    
Nagpur  117.0  46.2  16.5  21.0  35.0  44.0  52.0  109.0
Nashik  178.0  37.7  10.6  21.0  30.0  35.5  42.0   71.0
Pune    305.0  36.8  11.3  20.0  28.0  35.0  43.0   72.0

box.png

groupby("city")["minutes"].describe() gives one row of summary statistics per city; df.boxplot(column=..., by=...) draws one box per group side by side: a grouped box plot. Nagpur's median is 44 minutes against about 35 for Nashik and Pune, and its box sits higher with more slow outliers. (fig.suptitle("") removes the automatic "Boxplot grouped by city" title.)

Segment comparison

The tempting conclusion: Nagpur's riders are slow. Each city is a segment: a group of rows that share a trait (city, plan, age group, device). Comparing segments is fair only if they are alike in everything else. Check the obvious other difference:

print(df.groupby("city")["km"].median())
city
Nagpur    4.7
Nashik    3.1
Pune      2.7
Name: km, dtype: float64

Nagpur's median order is 4.7 km away against 2.7 km in Pune: Nagpur's orders are farther, and farther means slower. Compare within the same distance band:

import numpy as np, pandas as pd
rng = np.random.default_rng(38); n = 600
city = rng.choice(["Pune", "Nashik", "Nagpur"], n, p=[0.5, 0.3, 0.2])
km = (rng.gamma(2, np.where(city == "Nagpur", 2.4, 1.4)) + 0.5).round(1)
rain = rng.random(n) < 0.25
minutes = (15 + 4 * km + rain * (1 + 2.5 * km) + rng.gamma(2, 3, n)).round()
df = pd.DataFrame({"city": city, "km": km, "minutes": minutes})
df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
print(df.pivot_table(index="dist", columns="city", values="minutes",
                     aggfunc="median", observed=True))
city     Nagpur  Nashik  Pune
dist                         
(0, 2]     26.0    30.0  26.0
(2, 4]     35.0    32.5  33.0
(4, 8]     46.0    43.0  48.0
(8, 30]    76.0    61.5  59.0

Within each band up to 8 km, Nagpur is no slower than the others (26 vs 26-30, 35 vs 32.5-33, 46 vs 43-48). Most of the gap was distance, not the riders. A variable that hides behind a comparison like this is a confounder (confounding variable); episode 50 covers it properly. Rule: before blaming a segment, check what else differs between segments.

Categorical vs categorical: crosstab and stacked bars

Hypothesis 2: slow deliveries get fewer ratings. Make two yes/no columns: late (more than 45 minutes) and rated (rating not missing). Count every combination with pd.crosstab (episode 27); margins=True adds totals:

df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
print(pd.crosstab(df["late"], df["rated"], margins=True))
rated  False  True  All
late                   
False     34   423  457
True      55    88  143
All       89   511  600

34 on-time orders and 55 late orders were not rated, but there are 457 on-time orders and only 143 late ones. Raw counts of groups this different in size are unfair (episode 36). You need shares within each row. A natural guess:

df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
share = pd.crosstab(df["late"], df["rated"], normalize="row")
KeyError: 'row'

The above exception was the direct cause of the following exception:

ValueError: Not a valid normalize argument

ValueError: Not a valid normalize argument. crosstab's normalize accepts "index" (each row sums to 1), "columns" (each column sums to 1), "all" (the whole table sums to 1), or True (same as "all"); there is no "row".

df["late"] = df["minutes"] > 45
df["rated"] = df["rating"].notna()
share = pd.crosstab(df["late"], df["rated"], normalize="index")
print(share.round(2))
ax = share.plot.bar(stacked=True, figsize=(9, 4.5), rot=0)
ax.set_ylabel("share of orders")
ax.figure.tight_layout(); ax.figure.savefig("stacked.png")
rated  False  True 
late               
False   0.07   0.93
True    0.38   0.62

stacked.png

7% of on-time orders were not rated against 38% of late orders: more than five times as many. Hypothesis 2 holds. The chart is a 100% stacked bar (episode 32): each bar is one row of the table stretched to 100%. It is the standard chart for category vs category; a heatmap of the table works too for bigger tables.

Why it matters: if unhappy (late) customers skip the rating, the average rating hides your worst experiences and looks better than reality. This is non-response bias, which returns in episode 43.

Multivariate analysis and interactions

print(df.groupby("weather")["minutes"].mean().round(1))
weather
dry     36.0
rain    47.1
Name: minutes, dtype: float64

Dry: 36.0 minutes; rain: 47.1. "Rain adds about 11 minutes." But is that true for every order? Look at three variables at once (distance, minutes and weather) with a scatter plot coloured by weather:

fig, ax = plt.subplots(figsize=(10, 4.5))
for w, part in df.groupby("weather"):
    ax.scatter(part["km"], part["minutes"], alpha=0.5, label=w)
ax.legend(title="weather"); ax.set_xlabel("km"); ax.set_ylabel("minutes")
fig.tight_layout(); fig.savefig("hue.png")

hue.png

df.groupby("weather") gives one (name, part) pair per group, and each part is drawn in its own colour. At 1-2 km, orange (rain) and blue (dry) overlap; farther out, the orange cloud climbs faster. Measure it with a pivot_table (episode 27):

df["dist"] = pd.cut(df["km"], [0, 2, 4, 8, 30])
t = df.pivot_table(index="dist", columns="weather", values="minutes",
                   aggfunc="median", observed=True)
t["rain adds"] = t["rain"] - t["dry"]
print(t)
ax = t[["dry", "rain"]].plot(marker="o", figsize=(9, 4.5))
ax.set_ylabel("median minutes")
ax.figure.tight_layout(); ax.figure.savefig("interaction.png")
weather   dry  rain  rain adds
dist                          
(0, 2]   26.0  31.0        5.0
(2, 4]   32.0  41.0        9.0
(4, 8]   42.0  57.0       15.0
(8, 30]  59.0  91.0       32.0

interaction.png

Rain adds 5 minutes up to 2 km, 9 minutes at 2-4 km, 15 at 4-8 km and 32 beyond 8 km. There is no single "rain effect". If rain added a fixed time, the two lines would be parallel; they spread apart. When the effect of one variable depends on the value of another, that is an interaction: here, weather interacts with distance. The "11 minutes" average was right on average and wrong for almost every actual order.

Everyday interactions: a discount works better on weekends; a medicine's effect depends on age; an ad works on mobile but not on desktop. To find them, split one variable's effect by another (pivot tables, coloured scatter plots, small multiples from episode 32, seaborn hue from episode 33).

Documenting EDA findings

A finding nobody can find tomorrow is lost. Keep a findings log: one line per finding, each with the finding, the evidence, a caveat (what could make it wrong) and the next step.

# Finding Evidence Caveat Next step
1 Longer distance → slower delivery scatter; median 27 → 33 → 46 → 66 min by band only 25 orders over 8 km promise times by distance
2 Late orders go unrated far more often crosstab: 38% vs 7% not rated rating may be missing for other reasons too treat average rating as optimistic; send a follow-up for late orders
3 Rain adds 5 to 32 minutes depending on distance pivot table, coloured scatter 25% of orders in rain; long-distance rain orders are few promise times by distance and weather
4 Nagpur looks slower, but its orders are farther city box plots; km medians; within-band medians small numbers in the 8+ km band don't blame riders; check zones

In a Jupyter notebook (episode 3) this lives in a markdown cell at the top or the end; in a team it goes into the report or ticket. The log is what turns EDA into decisions.

Common mistakes

Code / habit Real error / symptom Why Fix
pd.crosstab(a, b, normalize="row") ValueError: Not a valid normalize argument rows are called "index" normalize="index" (or "columns", "all")
comparing raw crosstab counts no error; unfair comparison groups differ in size normalise by row
scatter of many points without alpha no error; dense areas look the same as sparse ones overplotting alpha=0.3, smaller dots, or bands
averages per group without counts no error; trusting tiny groups a group of 5 is noise print count next to every mean/median
"Nagpur riders are slow" no error; wrong blame Nagpur orders are farther compare within distance bands
reporting one average effect no error; hides an interaction effect depends on another variable split by the other variable
two separate describe() calls for a relationship no error; no answer summaries forget the pairing scatter, groupby, crosstab

Mini project: is weekend demand different?

import numpy as np, pandas as pd
rng = np.random.default_rng(5)
days = pd.date_range("2026-01-01", periods=120, freq="D")
weekend = days.dayofweek >= 5
promo = rng.random(120) < 0.3
orders = (200 + 60 * weekend + promo * np.where(weekend, 80, 20) + rng.normal(0, 15, 120)).round()
df = pd.DataFrame({"day": days, "weekend": weekend, "promo": promo, "orders": orders})
print(df.groupby("weekend")["orders"].agg(["count", "median"]))
t = df.pivot_table(index="weekend", columns="promo", values="orders", aggfunc="median")
t["promo adds"] = t[True] - t[False]
print(t)
         count  median
weekend               
False       86   206.5
True        34   264.5
promo    False   True  promo adds
weekend                          
False    200.0  219.0        19.0
True     261.5  338.0        76.5

Weekends have more orders, and the promotion adds far more on weekends than on weekdays: an interaction between weekend and promo. A findings-log line: promo adds about 4× more orders on weekends than weekdays (pivot table, 120 days, 33 of them promo days); next step: run promos on weekends.

Practice

1. Ten students' study hours and marks. Draw (or just print) the pairs sorted by hours. Is the relationship positive or negative?

Solution
import pandas as pd
s = pd.DataFrame({"hours": [1, 3, 2, 5, 4, 6, 2, 7, 3, 5], "marks": [45, 60, 52, 75, 66, 80, 50, 88, 58, 71]})
print(s.sort_values("hours").to_string(index=False))
 hours  marks
     1     45
     2     52
     2     50
     3     60
     3     58
     4     66
     5     71
     5     75
     6     80
     7     88

Marks rise with hours: a positive relationship. s.plot.scatter(x="hours", y="marks") draws it.

2. Plan (gold/basic) vs renewed (yes/no): gold 80 customers, 64 renewed; basic 420 customers, 210 renewed. Build the crosstab with normalize="index".

Solution
import pandas as pd
plan = ["gold"] * 80 + ["basic"] * 420
renewed = ["yes"] * 64 + ["no"] * 16 + ["yes"] * 210 + ["no"] * 210
print(pd.crosstab(pd.Series(plan, name="plan"), pd.Series(renewed, name="renewed"), normalize="index"))
renewed   no  yes
plan             
basic    0.5  0.5
gold     0.2  0.8

80% of gold customers renewed vs 50% of basic: compare shares, not the raw 64 vs 210.

3. Why is "Nagpur's median delivery is 44 minutes, so Nagpur riders are slow" a weak conclusion? What would you check?

Solution

Nagpur may differ in other ways (distance, traffic, weather, order size). Check those variables per city, then compare cities within the same distance band (as above): the gap mostly disappears.

Quick quiz

# Question: do customers on the "gold" plan
# (plan: gold / basic) rate us higher
# (rated: yes / no, then rating 1-5)?
# Step 1: plan vs rated, both categories

Which tool first? A. A scatter plot B. pd.crosstab(..., normalize="index") C. df["plan"].describe() D. A histogram of rating

Answer

B. Two categories → a crosstab, normalised by row because the plans have different numbers of customers. A scatter plot needs two numbers; describe() and a histogram look at one column only.

Interview questions

  1. Univariate vs bivariate vs multivariate analysis? One variable; two variables paired by row; three or more together.
  2. Which chart for each pair of variable types? Numeric-numeric: scatter; numeric-categorical: grouped box/violin plot; categorical-categorical: crosstab with a 100% stacked bar or heatmap.
  3. What does normalize="index" do in pd.crosstab? Divides each row by its total so rows sum to 1, giving shares within each row.
  4. What is overplotting and how do you fix it? Overlapping points hide density; use transparency (alpha), smaller markers, sampling, hexbin plots or bands.
  5. What is an interaction? When the effect of one variable on the outcome depends on another variable; e.g. rain adds more delay on longer trips.
  6. How do you compare segments fairly? Check what else differs between them, and compare within groups that are alike on those other variables.
  7. How do you document EDA findings? A findings log: finding, evidence, caveat, next step.

Glossary

Term Meaning Example
Univariate analysis One variable at a time describe() of minutes
Bivariate analysis Two variables together, paired by row km vs minutes
Multivariate analysis Three or more variables together km, minutes and weather
Scatter plot One dot per row at (x, y) km across, minutes up
Overplotting Points drawn on top of each other hide how many there are 600 dots
alpha Transparency of a marker (0 invisible, 1 solid) alpha=0.4
Positive / negative relationship One tends to rise / fall as the other rises distance and minutes: positive
Band (bin) A range of values grouped together 2-4 km
Grouped box plot One box plot per category, side by side minutes per city
Segment A group of rows sharing a trait the Nagpur orders
Segment comparison Comparing a measure across segments median minutes per city
Confounder A hidden variable that affects both sides of a comparison (episode 50) distance behind "Nagpur is slow"
Crosstab (contingency table) Counts of every combination of two categories late × rated
normalize Turn crosstab counts into shares: "index", "columns" or "all" normalize="index"
100% stacked bar Each bar is one group split into shares that add to 100% rated share, on time vs late
Pivot table A table with one variable down the side, one across, and a summary in each cell distance band × weather
Interaction The effect of one variable depends on the value of another rain +5 min short, +32 min long
Findings log Written list of findings with evidence, caveat and next step the table above
Caveat A warning about what could make a finding wrong only 25 orders over 8 km

Summary

  • One-column summaries forget the pairing; relationships need the rows kept together.
  • Bivariate: numeric-numeric → scatter + bands; numeric-categorical → groupby().describe() + grouped box plot; categorical-categorical → crosstab(normalize="index") + 100% stacked bar.
  • Segments: before blaming one, check what else differs; compare within like groups.
  • Multivariate: colour, split or pivot by a third variable; when an effect changes across groups, it's an interaction.
  • Write every finding in a findings log: finding, evidence, caveat, next step.
  • Next episode: Probability 1: events, the rules, and conditional probability, the maths behind "38% of late orders".