SyntaxPilot
Courses/Machine Learning/Lesson 13

Machine Learning · Lesson 13 of 90 · 14 min

Exploratory data analysis

exploratory data analysis (EDA)numerical vs categorical vs ordinal featuressummary statistics (describe)value countshistogramsbox plotsscatter plotspair plotscorrelation heatmaptarget distributionoutliers (first look)

Machine Learning Full Course · Episode 13. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2 and scikit-learn 1.9.1; the output shown is the real output, and the pictures are the PNG files those exact programs saved. All 300 flat listings are generated by the code (made up for teaching; the locality names are only labels).

The big idea: before you train any model, look at the data. Looking at, summarising and plotting a dataset to understand it is called exploratory data analysis (EDA). "Exploratory" means you don't know yet what you'll find. EDA turns surprises into a to-do list for cleaning (Episode 14) and feature engineering.

The problem: a fair rent for your cousin

Your cousin is moving to Pune and a broker quotes ₹25,000 a month. Is that fair? You collect 300 listings with five columns:

Column Meaning
area size of the flat in square feet
bhk number of bedrooms (BHK = bedroom, hall, kitchen)
furnishing none, semi or full
locality the area of the city
rent monthly rent in rupees: the label (also called the target), the value we want to predict

Each row is one example (also called a sample); each column except the label is a feature. Predicting a number is regression (Episode 1).

This code builds the table. One typo is planted on purpose (row 17's area gets an extra zero), because every real dataset has a few:

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area,
    "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15,
                           "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
  • rng.normal(950, 300, n) draws 300 areas from a normal (bell-shaped) distribution with mean 950 and standard deviation 300 (Episode 10); .clip(350, 2500) keeps them between 350 and 2500.
  • rng.choice([...], n, p=[...]) picks a value for each row with the given probabilities.
  • lift makes some localities pricier; rng.lognormal(0, .3, n) multiplies each rent by random noise that is always positive and skewed to the right.

The obvious attempt: straight to a model (it crashes)

import sys
sys.tracebacklimit = 0     # show only the error message
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(df.drop(columns="rent"), df["rent"])    # straight to a model

Output:

ValueError: could not convert string to float: 'full'

sys.tracebacklimit = 0 only hides the long traceback so the error message is easy to read.

Why it failed: a model does arithmetic, so every input must be a number. furnishing and locality hold text such as 'full'. We handed the model a table we had never looked at.

Real-life picture: a cook doesn't tip the vegetable bag straight into the pot. She looks at what came, weighs and counts it, and throws out the rotten tomato first. EDA is that check for data.

Step 1: the first look (head, shape, dtypes)

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0

print(df.head())
print("shape:", df.shape)
print(df.dtypes)

Output:

     area  bhk furnishing  locality     rent
0  1498.0    3       full     Wakad  37800.0
1   350.0    1       semi     Wakad   4600.0
2  1237.0    2       none     Baner  34900.0
3   971.0    2       full  Hadapsar  10600.0
4  1345.0    2       full     Wakad  24100.0
shape: (300, 5)
area          float64
bhk             int64
furnishing        str
locality          str
rent          float64
dtype: object
  • df.head() shows the first 5 rows (df.head(10) shows 10).
  • df.shape is (rows, columns): 300 examples, 5 columns.
  • df.dtypes gives each column's data type (dtype): float64 = numbers with decimals, int64 = whole numbers, str = text (pandas 3 shows text columns as str; older versions show object).

Feature types: numerical, categorical, ordinal

A school report card has all three kinds:

Report card Feature type Meaning In our data
Marks: 78, 91 Numerical amounts: adding and averaging make sense area, rent (and bhk)
House: Red, Blue Categorical (nominal) names with no order; an "average house" means nothing locality
Grade: C < B < A Ordinal names with a natural order furnishing: none < semi < full

bhk holds the numbers 1, 2, 3: it is numerical, with only three values, and its order matters.

Why it matters: the type decides which summaries and plots fit (means and histograms for numbers, counts and group comparisons for categories), and how a column will later be encoded, turned into numbers (Episode 16). For an ordinal column, the encoding must keep the order (none = 0, semi = 1, full = 2).

Step 2: summary statistics with describe()

Summary statistics are a few numbers that describe a whole column.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0

print(df.describe().round(1))

Output:

         area    bhk     rent
count   300.0  300.0    300.0
mean    975.3    1.7  19543.7
std     355.2    0.7   9503.5
min     350.0    1.0   3500.0
25%     765.0    1.0  12700.0
50%     974.0    2.0  17650.0
75%    1168.0    2.0  24250.0
max    4080.0    3.0  52400.0
Row Meaning
count how many values are present (not missing)
mean the average
std standard deviation: the typical distance from the mean (Episode 10)
min, max smallest and largest value
25%, 50%, 75% the quartiles (Episode 11): 25% of values lie below the first; the 50% line is the median

Reading it like a detective:

  1. The middle half of areas is 765 to 1168 sq ft, but the max is 4080 while the max bhk is 3. A 4080 sq ft flat with 1 bedroom? Suspicious.
  2. Rent's mean (19,543.7) is above its median (17,650). A mean pulled above the median means a few very high values: a long right tail.

Value counts for categorical columns

describe() skips text columns. For those we count how often each value appears.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0

print(df["locality"].value_counts())
print(df["furnishing"].value_counts(normalize=True).round(2))
print("bhk values:", sorted(df["bhk"].unique().tolist()))

Output:

locality
Kothrud     90
Wakad       80
Hadapsar    67
Baner       63
Name: count, dtype: int64
furnishing
none    0.43
semi    0.40
full    0.18
Name: proportion, dtype: float64
bhk values: [1, 2, 3]
  • value_counts() gives counts; normalize=True gives fractions.
  • Every locality has 60+ examples: enough to learn from. A category seen only 5 times is hard for any model.
  • Value counts also expose spelling problems (Baner vs baner): that's Episode 14.

The target distribution: histograms and skew

A histogram splits the range into equal intervals called bins and draws a bar for how many values fall in each bin. The distribution of a column is how its values are spread out; the target distribution is the first plot to draw in any project.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
fig, ax = plt.subplots(1, 2, figsize=(13, 5))
ax[0].hist(df["rent"], bins=30); ax[0].set_title("rent (Rs per month)")
ax[1].hist(np.log(df["rent"]), bins=30, color="tab:green"); ax[1].set_title("log(rent)")
plt.savefig("rent_hist.png", dpi=110, bbox_inches="tight")
print("mean:", round(df["rent"].mean()), "  median:", df["rent"].median())
print("skew of rent:", round(df["rent"].skew(), 2))
print("skew of log(rent):", round(np.log(df["rent"]).skew(), 2))

Output:

mean: 19544   median: 17650.0
skew of rent: 0.87
skew of log(rent): -0.4

Our compiler shows text, not windows, so plt.savefig saves the picture to a file. In a Jupyter notebook the plot appears under the cell.

Histograms of rent and log(rent)

  • Most rents are ₹10k to ₹25k, with a long tail to ₹52k: right skew (skewness 0.87; positive = right tail, Episode 11).
  • The log of rent is much more balanced (skew −0.4). Taking the log of a skewed target is a common trick; we'll use it in the feature engineering episodes.

Box plots: comparing groups, and a first look at outliers

A box plot summarises a column with five numbers and is ideal for comparing groups side by side.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
df.boxplot(column="rent", by="locality", figsize=(12, 5.5))
plt.suptitle("")
plt.savefig("rent_box.png", dpi=110, bbox_inches="tight")
print(df.groupby("locality")["rent"].median())
q1, q3 = df["rent"].quantile([0.25, 0.75]); iqr = q3 - q1
print("Q1:", q1, " Q3:", q3, " IQR:", iqr)
print("above Q3 + 1.5 IQR:", (df["rent"] > q3 + 1.5 * iqr).sum(), "flats")

Output:

locality
Baner       21600.0
Hadapsar    14500.0
Kothrud     19450.0
Wakad       17300.0
Name: rent, dtype: float64
Q1: 12700.0  Q3: 24250.0  IQR: 11550.0
above Q3 + 1.5 IQR: 11 flats

Box plots of rent by locality

How to read it:

  • The box runs from Q1 (25th percentile) to Q3 (75th percentile): the middle half of the data. Its height is the interquartile range (IQR) = Q3 − Q1 = 11,550 here.
  • The line inside is the median.
  • The whiskers reach the furthest point within 1.5 × IQR of the box.
  • Points beyond the whiskers are drawn as dots: outliers, values far from the rest. Here 11 flats lie above Q3 + 1.5 × IQR = 41,575.

An outlier is not automatically a mistake. A big furnished flat in Baner can truly cost ₹50,000. An outlier is a question to investigate, not a row to delete. Locality clearly matters: median rent ₹21,600 in Baner vs ₹14,500 in Hadapsar.

Scatter plots: two numbers together

A scatter plot draws one dot per row, one feature across (x axis, horizontal) and another up (y axis, vertical). It shows how two numbers move together.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
df.plot.scatter(x="area", y="rent", figsize=(12, 5.5))
plt.savefig("area_rent.png", dpi=110, bbox_inches="tight")
print(df[df["area"] > 3000])          # who is that dot?

Output:

      area  bhk furnishing locality    rent
17  4080.0    1       none  Kothrud  8000.0

Scatter plot of area vs rent

Bigger flats cost more: the cloud rises from bottom left to top right. And one dot sits far to the right on its own: row 17, 4080 sq ft, 1 BHK, ₹8,000. A one-bedroom flat of 4080 sq ft doesn't exist: someone typed an extra zero (it should be 408). The box plot of rent couldn't show it because its rent is ordinary; only area against rent reveals it.

Correlation and the correlation heatmap

Pearson correlation (Episode 11) measures how close the points are to a straight line, from −1 (perfect downward line) through 0 (no straight-line relationship) to +1 (perfect upward line).

The obvious attempt fails for the same reason the model did:

import sys
sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0

print(df.corr())

Output:

ValueError: could not convert string to float: 'full'

The fix is numeric_only=True, which keeps only the numerical columns:

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 16
corr = df.corr(numeric_only=True)
print(corr.round(2))
print("area vs rent without row 17:", round(df.drop(17)[["area", "rent"]].corr().iloc[0, 1], 2))
plt.imshow(corr, cmap="coolwarm", vmin=-1, vmax=1); plt.colorbar()
plt.xticks(range(3), corr.columns); plt.yticks(range(3), corr.columns)
for (i, j), v in np.ndenumerate(corr.round(2).values):
    plt.text(j, i, v, ha="center")
plt.savefig("heatmap.png", dpi=110, bbox_inches="tight")

Output:

      area   bhk  rent
area  1.00  0.73  0.58
bhk   0.73  1.00  0.62
rent  0.58  0.62  1.00
area vs rent without row 17: 0.71

Correlation heatmap

  • A correlation matrix lists every column against every other; the diagonal is always 1.
  • A correlation heatmap paints that matrix as colours, so with 10 or 50 features you can spot the strong pairs at a glance.
  • Look for (1) features strongly tied to the target and (2) features strongly tied to each other (area and bhk, 0.73): those carry overlapping information.
  • One typo changed the area–rent correlation from 0.71 to 0.58. Summaries can be fooled by a single bad row; plots show you why.

Pair plots (scatter matrix)

A pair plot draws a scatter plot for every pair of numerical columns in a grid. pandas calls it scatter_matrix; seaborn calls it pairplot.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300     # 300 made-up flats
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df["locality"].map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": 0.9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # a typo: an extra 0
import matplotlib.pyplot as plt
plt.rcParams["font.size"] = 14
pd.plotting.scatter_matrix(df[["area", "bhk", "rent"]], figsize=(9, 7))
plt.savefig("pairplot.png", dpi=110, bbox_inches="tight")
print("columns:", df.select_dtypes("number").columns.tolist())

Output:

columns: ['area', 'bhk', 'rent']

Pair plot of area, bhk and rent

Off the diagonal: scatter plots of two columns. On the diagonal: each column's histogram (a column against itself would just be a straight line). The area histogram (top left) is squashed into a spike because its axis must stretch to 4080 for one typo.

The EDA checklist

  1. head, shape, dtypes: what did we get?
  2. Name each feature's type: numerical, categorical, ordinal.
  3. describe() and value_counts(): counts, ranges, odd minimums and maximums, spellings.
  4. Plot the target distribution: histogram, skew.
  5. Box plots for groups, scatter and pair plots for pairs, a correlation heatmap.
  6. Write down every surprise: it becomes a cleaning or feature task.
Surprise found today Becomes
text columns crash the model encoding (Episode 16)
right-skewed target log transform (feature engineering)
row 17: 4080 sq ft, 1 BHK fix the typo (Episode 14)
11 very expensive flats investigate; maybe cap (Episode 14)
area and bhk correlated 0.73 overlapping features (feature selection)

Common mistakes

Mistake Exact error / symptom Why Fix
model.fit on the raw table ValueError: could not convert string to float: 'full' text columns look first; encode (Ep 16)
df.corr() with text columns ValueError: could not convert string to float: 'full' text columns df.corr(numeric_only=True)
trusting the mean of a skewed column mean 19,543.7 vs median 17,650 long right tail report the median too
trusting numbers without plots correlation 0.58 instead of 0.71 one typo row plot it (scatter, pair plot)
deleting every outlier real expensive flats disappear outlier ≠ error investigate each one
plt.show() in a script without a display no picture appears no window available plt.savefig("file.png")

Mini project: a one-call EDA report

Write a function that prints the whole checklist for any DataFrame:

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300          # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # one typo: an extra zero
# Mini project: a one-call EDA report for any DataFrame
def eda_report(df, target):
    print("shape:", df.shape)
    num = df.select_dtypes("number").columns.tolist()
    cat = df.select_dtypes(exclude="number").columns.tolist()
    print("numerical:", num, " categorical:", cat)
    print("missing values per column:", df.isna().sum().to_dict())
    print("duplicate rows:", df.duplicated().sum())
    for c in cat:
        print(c, "->", df[c].value_counts().to_dict())
    print("target skew:", round(df[target].skew(), 2))
    print("correlation with target:")
    print(df[num].corr()[target].drop(target).round(2).sort_values(ascending=False))
    for c in num:
        q1, q3 = df[c].quantile([0.25, 0.75]); iqr = q3 - q1
        out = ((df[c] < q1 - 1.5 * iqr) | (df[c] > q3 + 1.5 * iqr)).sum()
        print(f"{c}: {out} value(s) outside the IQR fence")

eda_report(df, "rent")

Output:

shape: (300, 5)
numerical: ['area', 'bhk', 'rent']  categorical: ['furnishing', 'locality']
missing values per column: {'area': 0, 'bhk': 0, 'furnishing': 0, 'locality': 0, 'rent': 0}
duplicate rows: 0
furnishing -> {'none': 128, 'semi': 119, 'full': 53}
locality -> {'Kothrud': 90, 'Wakad': 80, 'Hadapsar': 67, 'Baner': 63}
target skew: 0.87
correlation with target:
bhk     0.62
area    0.58
Name: rent, dtype: float64
area: 2 value(s) outside the IQR fence
bhk: 0 value(s) outside the IQR fence
rent: 11 value(s) outside the IQR fence

Practice

  1. Histogram of area: what does the typo do to the bins?
  2. Box plot of rent by furnishing: does furnishing matter?
Solutions 1 and 2
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300          # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # one typo: an extra zero
# Homework 1: histogram of area, with and without the typo
print("max area:", df["area"].max())
counts, edges = np.histogram(df["area"], bins=10)
print("bins with the typo:   ", counts.tolist())
counts, edges = np.histogram(df.drop(17)["area"], bins=10)
print("bins without the typo:", counts.tolist())

Output:

max area: 4080.0
bins with the typo:    [64, 143, 77, 15, 0, 0, 0, 0, 0, 1]
bins without the typo: [21, 27, 34, 53, 68, 41, 28, 17, 5, 5]

With the typo, 10 equal bins must stretch from 350 to 4080, so almost every flat lands in the first four bins and the last bin holds one flat. Without it, the shape of the data is visible.

import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300          # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # one typo: an extra zero
# Homework 2: rent by furnishing (box-plot numbers)
print(df.groupby("furnishing")["rent"].describe()[["count", "25%", "50%", "75%"]])

Output:

            count      25%      50%      75%
furnishing                                  
full         53.0  14400.0  19300.0  24800.0
none        128.0  11450.0  17300.0  24625.0
semi        119.0  12800.0  17500.0  22550.0

Fully furnished flats have a higher median (₹19,300 vs ₹17,300 to ₹17,500), but the boxes overlap a lot: furnishing matters much less than locality in this data. (To draw it: df.boxplot(column="rent", by="furnishing").)

  1. Recompute describe() without row 17.
  2. Name the type of each column in a dataset you know.
Solutions 3 and 4
import numpy as np, pandas as pd
rng = np.random.default_rng(13); n = 300          # 300 made-up flat listings
area = rng.normal(950, 300, n).clip(350, 2500).round()
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3).astype(int),
    "furnishing": rng.choice(["none", "semi", "full"], n, p=[.4, .4, .2]),
    "locality": rng.choice(["Baner", "Wakad", "Kothrud", "Hadapsar"], n)})
lift = df.locality.map({"Baner": 1.25, "Kothrud": 1.15, "Wakad": 1, "Hadapsar": .9})
df["rent"] = (area * 18 * lift * rng.lognormal(0, .3, n)).round(-2)
df.loc[17, "area"] = df.loc[17, "area"] * 10     # one typo: an extra zero
# Homework 3: describe() without the typo row
print(df.drop(17)[["area", "rent"]].describe().round(1))

Output:

         area     rent
count   299.0    299.0
mean    965.0  19582.3
std     306.8   9495.9
min     350.0   3500.0
25%     765.0  12750.0
50%     973.0  17800.0
75%    1167.0  24300.0
max    1826.0  52400.0

The max area drops from 4080 to 1826 and the std from 355.2 to 306.8: one row was inflating the spread.

Example for a cricket dataset: runs, balls faced, strike rate = numerical; team, venue = categorical; batting position (opener < middle order < tail) or match result (loss < tie < win) = ordinal.

Quick quiz

import pandas as pd

s = pd.Series([10, 12, 11, 13, 200])
print(s.mean(), s.median())

A) 49.2 12.0 B) 12.0 49.2 C) 49.2 49.2 D) 11.5 12.0

Answer

A) 49.2 12.0. Mean = 246 / 5 = 49.2, pulled up by the outlier 200. Median = the middle of 10, 11, 12, 13, 200 = 12. A big gap between mean and median signals skew or outliers.

Interview questions

  1. What is EDA and why do it before modelling? Looking at, summarising and plotting data to understand it; it finds wrong types, missing values, outliers, skew, leakage candidates and relationships that decide cleaning, features and model choice.
  2. Numerical vs categorical vs ordinal? Numerical = amounts; categorical = labels with no order; ordinal = labels with an order. They need different plots and encodings.
  3. What does describe() show? count, mean, std, min, 25%, 50% (median), 75%, max for numerical columns (include="all" adds text columns).
  4. How do you spot skew? Histogram shape, mean far from median, and skew(); right skew → mean > median.
  5. How is a box plot built? Box Q1 to Q3 (the IQR), median line, whiskers to the furthest point within 1.5 × IQR, dots beyond = outliers.
  6. Does a high correlation mean a feature is useful? Does zero correlation mean useless? Pearson only measures straight-line relationships; a curved relationship can have low Pearson correlation, and correlation is not causation (Episode 11). It's a first hint, not a verdict.
  7. What would you do with an outlier? Investigate: fix if it's a typo, keep if it's real, cap or use robust methods if it distorts the model (Episode 14).
  8. Why plot instead of only computing statistics? Different data can share the same summary numbers; one bad row changed our correlation from 0.71 to 0.58.

Glossary

Term Meaning Example
Exploratory data analysis (EDA) Looking at, summarising and plotting data before modelling today's checklist
Dataset A table of examples 300 flat listings
Example (sample, row) One record in the table one flat
Feature An input column area
Label (target) The column we want to predict rent
Regression Predicting a number predicting rent
Data type (dtype) The kind of values in a column float64, int64, str
float64 / int64 Decimal numbers / whole numbers (64 bits each) 1498.0 / 3
Numerical feature A column of amounts area, rent
Categorical (nominal) feature Labels with no order locality
Ordinal feature Labels with an order furnishing: none < semi < full
Encoding Turning text categories into numbers none → 0, semi → 1, full → 2
Summary statistics A few numbers that summarise a column mean, median, std
describe() pandas summary of every numerical column count, mean, std, quartiles
Mean The average 19,543.7
Median The middle value; the 50% quartile 17,650
Standard deviation (std) Typical distance from the mean 9,503.5
Quartiles (Q1, Q3) Values with 25% / 75% of the data below 12,700 / 24,250
Interquartile range (IQR) Q3 − Q1: spread of the middle half 11,550
value_counts() Count of each distinct value Kothrud 90
Distribution How a column's values are spread most rents 10k to 25k
Target distribution The distribution of the label rent histogram
Histogram Bars counting values per bin plt.hist(df["rent"], bins=30)
Bin One interval of a histogram one bar of the rent histogram
Skewness (skew) Lopsidedness; positive = long right tail 0.87
Log transform Replacing x by log(x) to tame a right tail skew 0.87 → −0.4
Box plot Box (Q1 to Q3), median line, whiskers, outlier dots rent by locality
Whiskers Lines reaching the furthest point within 1.5 × IQR the lines above and below each box
Outlier A value far from the rest; not always an error row 17
Scatter plot One dot per row, two numerical features area vs rent
x axis / y axis Horizontal / vertical axis area / rent
Pearson correlation Straight-line relationship, −1 to +1 area–rent 0.71
Correlation matrix Correlation of every column with every other df.corr(numeric_only=True)
Correlation heatmap The matrix drawn as colours red = +1, blue = −1
Pair plot (scatter matrix) Grid of scatter plots for every pair pd.plotting.scatter_matrix
numeric_only=True Use only the numerical columns df.corr(numeric_only=True)
savefig Save a matplotlib figure to a file plt.savefig("rent_hist.png")
Traceback Python's list of calls that led to an error hidden with sys.tracebacklimit = 0

Summary

  • EDA: look at, summarise and plot data before modelling. A model trained on an unexamined table crashed on the first text column.
  • Name each feature's type: numerical, categorical, ordinal.
  • describe() and value_counts() reveal odd values; a mean far above the median means a right-skewed column.
  • Plot the target distribution, then box plots (groups and outliers), scatter and pair plots (pairs), and a correlation heatmap (numeric_only=True).
  • One typo changed a correlation from 0.71 to 0.58: plots catch what summaries hide. Every surprise is a task for the next step.

Next episode: Data cleaning: missing values (MCAR, MAR, MNAR), dropping vs imputing, mean, median, mode and KNN imputation, the missing-indicator feature, duplicates, outliers with IQR and z-scores, capping, and fixing types and categories.