Machine Learning Full Course · Episode 13. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2 and scikit-learn 1.9.1; the output shown is the real output, and the pictures are the PNG files those exact programs saved. All 300 flat listings are generated by the code (made up for teaching; the locality names are only labels).
The big idea: before you train any model, look at the data. Looking at, summarising and plotting a dataset to understand it is called exploratory data analysis (EDA). "Exploratory" means you don't know yet what you'll find. EDA turns surprises into a to-do list for cleaning (Episode 14) and feature engineering.
The problem: a fair rent for your cousin
Your cousin is moving to Pune and a broker quotes ₹25,000 a month. Is that fair? You collect 300 listings with five columns:
| Column | Meaning |
|---|---|
area |
size of the flat in square feet |
bhk |
number of bedrooms (BHK = bedroom, hall, kitchen) |
furnishing |
none, semi or full |
locality |
the area of the city |
rent |
monthly rent in rupees: the label (also called the target), the value we want to predict |
Each row is one example (also called a sample); each column except the label is a feature. Predicting a number is regression (Episode 1).
This code builds the table. One typo is planted on purpose (row 17's area gets an extra zero), because every real dataset has a few:
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area,
"bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15,
"Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
rng.normal(950, 300, n)draws 300 areas from a normal (bell-shaped) distribution with mean 950 and standard deviation 300 (Episode 10);.clip(350, 2500)keeps them between 350 and 2500.rng.choice([...], n, p=[...])picks a value for each row with the given probabilities.liftmakes some localities pricier;rng.lognormal(0, .3, n)multiplies each rent by random noise that is always positive and skewed to the right.
The obvious attempt: straight to a model (it crashes)
import sys
sys.tracebacklimit = 0 # show only the error message
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(df.drop(columns="rent"), df["rent"]) # straight to a model
Output:
ValueError: could not convert string to float: 'full'
sys.tracebacklimit = 0 only hides the long traceback so the error message is easy to read.
Why it failed: a model does arithmetic, so every input must be a number. furnishing and locality hold text such as 'full'. We handed the model a table we had never looked at.
Real-life picture: a cook doesn't tip the vegetable bag straight into the pot. She looks at what came, weighs and counts it, and throws out the rotten tomato first. EDA is that check for data.
Step 1: the first look (head, shape, dtypes)
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
print(df.head())
print("shape:", df.shape)
print(df.dtypes)
Output:
area bhk furnishing locality rent
0 1498.0 3 full Wakad 37800.0
1 350.0 1 semi Wakad 4600.0
2 1237.0 2 none Baner 34900.0
3 971.0 2 full Hadapsar 10600.0
4 1345.0 2 full Wakad 24100.0
shape: (300, 5)
area float64
bhk int64
furnishing str
locality str
rent float64
dtype: object
df.head()shows the first 5 rows (df.head(10)shows 10).df.shapeis(rows, columns): 300 examples, 5 columns.df.dtypesgives each column's data type (dtype):float64= numbers with decimals,int64= whole numbers,str= text (pandas 3 shows text columns asstr; older versions showobject).
Feature types: numerical, categorical, ordinal
A school report card has all three kinds:
| Report card | Feature type | Meaning | In our data |
|---|---|---|---|
| Marks: 78, 91 | Numerical | amounts: adding and averaging make sense | area, rent (and bhk) |
| House: Red, Blue | Categorical (nominal) | names with no order; an "average house" means nothing | locality |
| Grade: C < B < A | Ordinal | names with a natural order | furnishing: none < semi < full |
bhk holds the numbers 1, 2, 3: it is numerical, with only three values, and its order matters.
Why it matters: the type decides which summaries and plots fit (means and histograms for numbers, counts and group comparisons for categories), and how a column will later be encoded, turned into numbers (Episode 16). For an ordinal column, the encoding must keep the order (none = 0, semi = 1, full = 2).
Step 2: summary statistics with describe()
Summary statistics are a few numbers that describe a whole column.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
print(df.describe().round(1))
Output:
area bhk rent
count 300.0 300.0 300.0
mean 975.3 1.7 19543.7
std 355.2 0.7 9503.5
min 350.0 1.0 3500.0
25% 765.0 1.0 12700.0
50% 974.0 2.0 17650.0
75% 1168.0 2.0 24250.0
max 4080.0 3.0 52400.0
| Row | Meaning |
|---|---|
count |
how many values are present (not missing) |
mean |
the average |
std |
standard deviation: the typical distance from the mean (Episode 10) |
min, max |
smallest and largest value |
25%, 50%, 75% |
the quartiles (Episode 11): 25% of values lie below the first; the 50% line is the median |
Reading it like a detective:
- The middle half of areas is 765 to 1168 sq ft, but the max is 4080 while the max
bhkis 3. A 4080 sq ft flat with 1 bedroom? Suspicious. - Rent's mean (19,543.7) is above its median (17,650). A mean pulled above the median means a few very high values: a long right tail.
Value counts for categorical columns
describe() skips text columns. For those we count how often each value appears.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
print(df["locality"].value_counts())
print(df["furnishing"].value_counts(normalize=True).round(2))
print("bhk values:", sorted(df["bhk"].unique().tolist()))
Output:
locality
Kothrud 90
Wakad 80
Hadapsar 67
Baner 63
Name: count, dtype: int64
furnishing
none 0.43
semi 0.40
full 0.18
Name: proportion, dtype: float64
bhk values: [1, 2, 3]
value_counts()gives counts;normalize=Truegives fractions.- Every locality has 60+ examples: enough to learn from. A category seen only 5 times is hard for any model.
- Value counts also expose spelling problems (
Banervsbaner): that's Episode 14.
The target distribution: histograms and skew
A histogram splits the range into equal intervals called bins and draws a bar for how many values fall in each bin. The distribution of a column is how its values are spread out; the target distribution is the first plot to draw in any project.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
fig, ax = plt.subplots(1, 2, figsize=(13, 5))
ax[0].hist(df["rent"], bins=30); ax[0].set_title("rent (Rs per month)")
ax[1].hist(np.log(df["rent"]), bins=30, color="tab:green"); ax[1].set_title("log(rent)")
plt.savefig("rent_hist.png", dpi=110, bbox_inches="tight")
print("mean:", round(df["rent"].mean()), " median:", df["rent"].median())
print("skew of rent:", round(df["rent"].skew(), 2))
print("skew of log(rent):", round(np.log(df["rent"]).skew(), 2))
Output:
mean: 19544 median: 17650.0
skew of rent: 0.87
skew of log(rent): -0.4
Our compiler shows text, not windows, so plt.savefig saves the picture to a file. In a Jupyter notebook the plot appears under the cell.

- Most rents are ₹10k to ₹25k, with a long tail to ₹52k: right skew (skewness 0.87; positive = right tail, Episode 11).
- The log of rent is much more balanced (skew −0.4). Taking the log of a skewed target is a common trick; we'll use it in the feature engineering episodes.
Box plots: comparing groups, and a first look at outliers
A box plot summarises a column with five numbers and is ideal for comparing groups side by side.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
df.boxplot(column="rent", by="locality", figsize=(12, 5.5))
plt.suptitle("")
plt.savefig("rent_box.png", dpi=110, bbox_inches="tight")
print(df.groupby("locality")["rent"].median())
q1, q3 = df["rent"].quantile([0.25, 0.75]); iqr = q3 - q1
print("Q1:", q1, " Q3:", q3, " IQR:", iqr)
print("above Q3 + 1.5 IQR:", (df["rent"] > q3 + 1.5 * iqr).sum(), "flats")
Output:
locality
Baner 21600.0
Hadapsar 14500.0
Kothrud 19450.0
Wakad 17300.0
Name: rent, dtype: float64
Q1: 12700.0 Q3: 24250.0 IQR: 11550.0
above Q3 + 1.5 IQR: 11 flats

How to read it:
- The box runs from Q1 (25th percentile) to Q3 (75th percentile): the middle half of the data. Its height is the interquartile range (IQR) = Q3 − Q1 = 11,550 here.
- The line inside is the median.
- The whiskers reach the furthest point within 1.5 × IQR of the box.
- Points beyond the whiskers are drawn as dots: outliers, values far from the rest. Here 11 flats lie above Q3 + 1.5 × IQR = 41,575.
An outlier is not automatically a mistake. A big furnished flat in Baner can truly cost ₹50,000. An outlier is a question to investigate, not a row to delete. Locality clearly matters: median rent ₹21,600 in Baner vs ₹14,500 in Hadapsar.
Scatter plots: two numbers together
A scatter plot draws one dot per row, one feature across (x axis, horizontal) and another up (y axis, vertical). It shows how two numbers move together.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
df.plot.scatter(x="area", y="rent", figsize=(12, 5.5))
plt.savefig("area_rent.png", dpi=110, bbox_inches="tight")
print(df[df["area"] > 3000]) # who is that dot?
Output:
area bhk furnishing locality rent
17 4080.0 1 none Kothrud 8000.0

Bigger flats cost more: the cloud rises from bottom left to top right. And one dot sits far to the right on its own: row 17, 4080 sq ft, 1 BHK, ₹8,000. A one-bedroom flat of 4080 sq ft doesn't exist: someone typed an extra zero (it should be 408). The box plot of rent couldn't show it because its rent is ordinary; only area against rent reveals it.
Correlation and the correlation heatmap
Pearson correlation (Episode 11) measures how close the points are to a straight line, from −1 (perfect downward line) through 0 (no straight-line relationship) to +1 (perfect upward line).
The obvious attempt fails for the same reason the model did:
import sys
sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
print(df.corr())
Output:
ValueError: could not convert string to float: 'full'
The fix is numeric_only=True, which keeps only the numerical columns:
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
corr = df.corr(numeric_only=True)
print(corr.round(2))
print("area vs rent without row 17:", round(df.drop(17)[["area", "rent"]].corr().iloc[0, 1], 2))
plt.imshow(corr, cmap="coolwarm", vmin=-1, vmax=1); plt.colorbar()
plt.xticks(range(3), corr.columns); plt.yticks(range(3), corr.columns)
for (i, j), v in np.ndenumerate(corr.round(2).values):
plt.text(j, i, v, ha="center")
plt.savefig("heatmap.png", dpi=110, bbox_inches="tight")
Output:
area bhk rent
area 1.00 0.73 0.58
bhk 0.73 1.00 0.62
rent 0.58 0.62 1.00
area vs rent without row 17: 0.71

- A correlation matrix lists every column against every other; the diagonal is always 1.
- A correlation heatmap paints that matrix as colours, so with 10 or 50 features you can spot the strong pairs at a glance.
- Look for (1) features strongly tied to the target and (2) features strongly tied to each other (area and bhk, 0.73): those carry overlapping information.
- One typo changed the area–rent correlation from 0.71 to 0.58. Summaries can be fooled by a single bad row; plots show you why.
Pair plots (scatter matrix)
A pair plot draws a scatter plot for every pair of numerical columns in a grid. pandas calls it scatter_matrix; seaborn calls it pairplot.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 14
pd.plotting.scatter_matrix(df[["area", "bhk", "rent"]], figsize=(9, 7))
plt.savefig("pairplot.png", dpi=110, bbox_inches="tight")
print("columns:", df.select_dtypes("number").columns.tolist())
Output:
columns: ['area', 'bhk', 'rent']

Off the diagonal: scatter plots of two columns. On the diagonal: each column's histogram (a column against itself would just be a straight line). The area histogram (top left) is squashed into a spike because its axis must stretch to 4080 for one typo.
The EDA checklist
head,shape,dtypes: what did we get?- Name each feature's type: numerical, categorical, ordinal.
describe()andvalue_counts(): counts, ranges, odd minimums and maximums, spellings.- Plot the target distribution: histogram, skew.
- Box plots for groups, scatter and pair plots for pairs, a correlation heatmap.
- Write down every surprise: it becomes a cleaning or feature task.
| Surprise found today | Becomes |
|---|---|
| text columns crash the model | encoding (Episode 16) |
| right-skewed target | log transform (feature engineering) |
| row 17: 4080 sq ft, 1 BHK | fix the typo (Episode 14) |
| 11 very expensive flats | investigate; maybe cap (Episode 14) |
| area and bhk correlated 0.73 | overlapping features (feature selection) |
Common mistakes
| Mistake | Exact error / symptom | Why | Fix |
|---|---|---|---|
model.fit on the raw table |
ValueError: could not convert string to float: 'full' |
text columns | look first; encode (Ep 16) |
df.corr() with text columns |
ValueError: could not convert string to float: 'full' |
text columns | df.corr(numeric_only=True) |
| trusting the mean of a skewed column | mean 19,543.7 vs median 17,650 | long right tail | report the median too |
| trusting numbers without plots | correlation 0.58 instead of 0.71 | one typo row | plot it (scatter, pair plot) |
| deleting every outlier | real expensive flats disappear | outlier ≠ error | investigate each one |
plt.show() in a script without a display |
no picture appears | no window available | plt.savefig("file.png") |
Mini project: a one-call EDA report
Write a function that prints the whole checklist for any DataFrame:
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # one typo: an extra zero
# Mini project: a one-call EDA report for any DataFrame
def eda_report(df, target):
print("shape:", df.shape)
num = df.select_dtypes("number").columns.tolist()
cat = df.select_dtypes(exclude="number").columns.tolist()
print("numerical:", num, " categorical:", cat)
print("missing values per column:", df.isna().sum().to_dict())
print("duplicate rows:", df.duplicated().sum())
for c in cat:
print(c, "->", df[c].value_counts().to_dict())
print("target skew:", round(df[target].skew(), 2))
print("correlation with target:")
print(df[num].corr()[target].drop(target).round(2).sort_values(ascending=False))
for c in num:
q1, q3 = df[c].quantile([0.25, 0.75]); iqr = q3 - q1
out = ((df[c] < q1 - 1.5 * iqr) | (df[c] > q3 + 1.5 * iqr)).sum()
print(f"{c}: {out} value(s) outside the IQR fence")
eda_report(df, "rent")
Output:
shape: (300, 5)
numerical: ['area', 'bhk', 'rent'] categorical: ['furnishing', 'locality']
missing values per column: {'area': 0, 'bhk': 0, 'furnishing': 0, 'locality': 0, 'rent': 0}
duplicate rows: 0
furnishing -> {'none': 128, 'semi': 119, 'full': 53}
locality -> {'Kothrud': 90, 'Wakad': 80, 'Hadapsar': 67, 'Baner': 63}
target skew: 0.87
correlation with target:
bhk 0.62
area 0.58
Name: rent, dtype: float64
area: 2 value(s) outside the IQR fence
bhk: 0 value(s) outside the IQR fence
rent: 11 value(s) outside the IQR fence
Practice
- Histogram of
area: what does the typo do to the bins? - Box plot of rent by
furnishing: does furnishing matter?
Solutions 1 and 2
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # one typo: an extra zero
# Homework 1: histogram of area, with and without the typo
print("max area:", df["area"].max())
counts, edges = np.histogram(df["area"], bins=10)
print("bins with the typo: ", counts.tolist())
counts, edges = np.histogram(df.drop(17)["area"], bins=10)
print("bins without the typo:", counts.tolist())
Output:
max area: 4080.0
bins with the typo: [64, 143, 77, 15, 0, 0, 0, 0, 0, 1]
bins without the typo: [21, 27, 34, 53, 68, 41, 28, 17, 5, 5]
With the typo, 10 equal bins must stretch from 350 to 4080, so almost every flat lands in the first four bins and the last bin holds one flat. Without it, the shape of the data is visible.
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # one typo: an extra zero
# Homework 2: rent by furnishing (box-plot numbers)
print(df.groupby("furnishing")["rent"].describe()[["count", "25%", "50%", "75%"]])
Output:
count 25% 50% 75%
furnishing
full 53.0 14400.0 19300.0 24800.0
none 128.0 11450.0 17300.0 24625.0
semi 119.0 12800.0 17500.0 22550.0
Fully furnished flats have a higher median (₹19,300 vs ₹17,300 to ₹17,500), but the boxes overlap a lot: furnishing matters much less than locality in this data. (To draw it: df.boxplot(column="rent", by="furnishing").)
- Recompute
describe()without row 17. - Name the type of each column in a dataset you know.
Solutions 3 and 4
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300 # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
"furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
"locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10 # one typo: an extra zero
# Homework 3: describe() without the typo row
print(df.drop(17)[["area", "rent"]].describe().round(1))
Output:
area rent
count 299.0 299.0
mean 965.0 19582.3
std 306.8 9495.9
min 350.0 3500.0
25% 765.0 12750.0
50% 973.0 17800.0
75% 1167.0 24300.0
max 1826.0 52400.0
The max area drops from 4080 to 1826 and the std from 355.2 to 306.8: one row was inflating the spread.
Example for a cricket dataset: runs, balls faced, strike rate = numerical; team, venue = categorical; batting position (opener < middle order < tail) or match result (loss < tie < win) = ordinal.
Quick quiz
import pandas as pd
s = pd.Series([10, 12, 11, 13, 200])
print(s.mean(), s.median())
A) 49.2 12.0 B) 12.0 49.2 C) 49.2 49.2 D) 11.5 12.0
Answer
A) 49.2 12.0. Mean = 246 / 5 = 49.2, pulled up by the outlier 200. Median = the middle of 10, 11, 12, 13, 200 = 12. A big gap between mean and median signals skew or outliers.
Interview questions
- What is EDA and why do it before modelling? Looking at, summarising and plotting data to understand it; it finds wrong types, missing values, outliers, skew, leakage candidates and relationships that decide cleaning, features and model choice.
- Numerical vs categorical vs ordinal? Numerical = amounts; categorical = labels with no order; ordinal = labels with an order. They need different plots and encodings.
- What does
describe()show? count, mean, std, min, 25%, 50% (median), 75%, max for numerical columns (include="all"adds text columns). - How do you spot skew? Histogram shape, mean far from median, and
skew(); right skew → mean > median. - How is a box plot built? Box Q1 to Q3 (the IQR), median line, whiskers to the furthest point within 1.5 × IQR, dots beyond = outliers.
- Does a high correlation mean a feature is useful? Does zero correlation mean useless? Pearson only measures straight-line relationships; a curved relationship can have low Pearson correlation, and correlation is not causation (Episode 11). It's a first hint, not a verdict.
- What would you do with an outlier? Investigate: fix if it's a typo, keep if it's real, cap or use robust methods if it distorts the model (Episode 14).
- Why plot instead of only computing statistics? Different data can share the same summary numbers; one bad row changed our correlation from 0.71 to 0.58.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Exploratory data analysis (EDA) | Looking at, summarising and plotting data before modelling | today's checklist |
| Dataset | A table of examples | 300 flat listings |
| Example (sample, row) | One record in the table | one flat |
| Feature | An input column | area |
| Label (target) | The column we want to predict | rent |
| Regression | Predicting a number | predicting rent |
| Data type (dtype) | The kind of values in a column | float64, int64, str |
float64 / int64 |
Decimal numbers / whole numbers (64 bits each) | 1498.0 / 3 |
| Numerical feature | A column of amounts | area, rent |
| Categorical (nominal) feature | Labels with no order | locality |
| Ordinal feature | Labels with an order | furnishing: none < semi < full |
| Encoding | Turning text categories into numbers | none → 0, semi → 1, full → 2 |
| Summary statistics | A few numbers that summarise a column | mean, median, std |
describe() |
pandas summary of every numerical column | count, mean, std, quartiles |
| Mean | The average | 19,543.7 |
| Median | The middle value; the 50% quartile | 17,650 |
| Standard deviation (std) | Typical distance from the mean | 9,503.5 |
| Quartiles (Q1, Q3) | Values with 25% / 75% of the data below | 12,700 / 24,250 |
| Interquartile range (IQR) | Q3 − Q1: spread of the middle half | 11,550 |
value_counts() |
Count of each distinct value | Kothrud 90 |
| Distribution | How a column's values are spread | most rents 10k to 25k |
| Target distribution | The distribution of the label | rent histogram |
| Histogram | Bars counting values per bin | plt.hist(df["rent"], bins=30) |
| Bin | One interval of a histogram | one bar of the rent histogram |
| Skewness (skew) | Lopsidedness; positive = long right tail | 0.87 |
| Log transform | Replacing x by log(x) to tame a right tail | skew 0.87 → −0.4 |
| Box plot | Box (Q1 to Q3), median line, whiskers, outlier dots | rent by locality |
| Whiskers | Lines reaching the furthest point within 1.5 × IQR | the lines above and below each box |
| Outlier | A value far from the rest; not always an error | row 17 |
| Scatter plot | One dot per row, two numerical features | area vs rent |
| x axis / y axis | Horizontal / vertical axis | area / rent |
| Pearson correlation | Straight-line relationship, −1 to +1 | area–rent 0.71 |
| Correlation matrix | Correlation of every column with every other | df.corr(numeric_only=True) |
| Correlation heatmap | The matrix drawn as colours | red = +1, blue = −1 |
| Pair plot (scatter matrix) | Grid of scatter plots for every pair | pd.plotting.scatter_matrix |
numeric_only=True |
Use only the numerical columns | df.corr(numeric_only=True) |
savefig |
Save a matplotlib figure to a file | plt.savefig("rent_hist.png") |
| Traceback | Python's list of calls that led to an error | hidden with sys.tracebacklimit = 0 |
Summary
- EDA: look at, summarise and plot data before modelling. A model trained on an unexamined table crashed on the first text column.
- Name each feature's type: numerical, categorical, ordinal.
describe()andvalue_counts()reveal odd values; a mean far above the median means a right-skewed column.- Plot the target distribution, then box plots (groups and outliers), scatter and pair plots (pairs), and a correlation heatmap (
numeric_only=True). - One typo changed a correlation from 0.71 to 0.58: plots catch what summaries hide. Every surprise is a task for the next step.
Next episode: Data cleaning: missing values (MCAR, MAR, MNAR), dropping vs imputing, mean, median, mode and KNN imputation, the missing-indicator feature, duplicates, outliers with IQR and z-scores, capping, and fixing types and categories.