Machine Learning Full Course · Episode 88. Every program below was run on a Linux machine with Python 3.13, scikit-learn 1.9.1, pandas 3.0.5 and NumPy 2.5.3 (one CPU thread), and the output shown is the real output (timings will differ on your computer). The flats data is made up: the generator is at the end of these notes, so you can make exactly the same file. The locality names are only labels, not real market prices.
The big idea: a project is one problem solved end to end, from a raw file to a result someone else can trust. The order matters: frame the problem → look at the data (EDA) → clean it → split once → baselines → features → a stronger model → tuning → open the test set once → error analysis → a model report. Most of the improvement in this project came from cleaning and from matching features to the target, not from the fancy model.
Words you need first
- Project: one problem solved end to end, from raw data to a result someone uses.
- Regression: predicting a number (a price), not a class (Episode 1).
- Lakh: 1,00,000 rupees. 120 lakh = 1.2 crore rupees.
- MAE, mean absolute error: the average size of the mistakes. Predict 100, truth 110 → error 10 lakh (Episode 19).
- Cross-validation (CV): split the training data into 5 slices; train on 4, score on the 5th, five times, and average (Episode 26).
- Pipeline: a chain of steps (fill gaps → scale → model) that is fitted as one object (Episode 17).
Step 1: frame the problem
Real-life example: three brokers, three prices. Asha wants to sell her 2 BHK, 950 sq ft flat in Kothrud, Pune. Three brokers quote 118, 135 and 150 lakh: a 32 lakh gap. We have 3,018 past sales and want a fair price with an honest range.
| Question | Our answer |
|---|---|
| Target (label) | price_lakh |
| Inputs | only what a seller knows before the sale |
| Metric | MAE in lakh (easy to explain) + MAPE, the mean absolute percentage error |
| Success | most predictions within 10% of the real price |
MAPE matters because a 10 lakh miss on a 50 lakh flat (20%) is much worse than on a 300 lakh flat (3.3%).
Step 2: three failed first attempts
Attempt 1: throw everything into a linear regression.
import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv("/data/pune_flats.csv") # 3,018 made-up Pune flats
print(df.shape); print(df.head(3).to_string(index=False))
X, y = df.drop(columns="price_lakh"), df["price_lakh"]
try: model = LinearRegression().fit(X, y) # obvious attempt: throw it all in
except ValueError as e: print("ValueError:", e)
Output:
(3018, 11)
locality bhk area_sqft floor total_floors age_years parking furnishing metro_km price_lakh price_per_sqft
Baner 4 1795 2 21 10.0 yes semi 1.0 231.1 12875.0
Viman Nagar 4 2164 0 6 7.0 no unfurnished 1.9 181.9 8406.0
Hinjewadi 3 1596 3 6 0.0 yes unfurnished NaN 140.5 8803.0
ValueError: could not convert string to float: 'Baner'
(We catch the ValueError and print its message so it is easy to read; without try, Python prints the full traceback ending in the same line.) A linear model only understands numbers, and locality is text.
Attempt 2: numbers only.
import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv("/data/pune_flats.csv")
X = df.select_dtypes("number").drop(columns="price_lakh") # attempt 2: numbers only
try: model = LinearRegression().fit(X, df["price_lakh"])
except ValueError as e: print("ValueError:", str(e).split(". ")[0] + ".")
Output:
ValueError: Input X contains NaN.
LinearRegression does not accept missing values encoded as NaN natively.
NaN means "not a number": an empty cell. Some sellers never gave the building's age.
Attempt 3: drop the gaps, numbers only, a gradient boosting model.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score
df = pd.read_csv("/data/pune_flats.csv").dropna() # attempt 3: drop rows with gaps
X, y = df.select_dtypes("number").drop(columns="price_lakh"), df["price_lakh"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)
m = HistGradientBoostingRegressor(random_state=0).fit(Xtr, ytr)
p = m.predict(Xte)
print(f"rows left {len(df)} test R2 {r2_score(yte, p):.3f} MAE {mean_absolute_error(yte, p):.1f} lakh")
print("columns used:", list(X.columns))
Output:
rows left 2809 test R2 0.992 MAE 1.8 lakh
columns used: ['bhk', 'area_sqft', 'floor', 'total_floors', 'age_years', 'metro_km', 'price_per_sqft']
R² (R-squared) is the share of the variation in prices the model explains; 1 is perfect. 0.992 looks like a miracle, and that is exactly why you should be suspicious. price_per_sqft = price ÷ area is computed from the answer. Asha doesn't know it before the sale. This is target leakage: an input that secretly contains the target. Ask of every column: would I know this at prediction time? dropna also threw away 209 sales, and we had not looked at the data at all.
Step 3: exploratory data analysis (EDA)
Real-life example: a doctor's check-up. Basic numbers first (summary statistics), then missing reports and wrong entries (missing values, duplicates, typos), then what moves together (correlation and skew).
- Correlation (Pearson): a number from −1 to 1 for how strongly two columns rise together.
- Skew: how lopsided a distribution is; positive skew = a long tail of big values.
import pandas as pd
df = pd.read_csv("/data/pune_flats.csv")
print("missing:", df.isna().sum()[lambda s: s > 0].to_dict(), " duplicate rows:", df.duplicated().sum())
print(df[["area_sqft", "age_years", "metro_km", "price_lakh"]].describe().round(1).to_string())
print(df["locality"].value_counts().to_dict())
corr = df.select_dtypes("number").corr()["price_lakh"].drop(["price_lakh", "price_per_sqft"])
print(f"price skew {df.price_lakh.skew():.2f} corr with price:", corr.round(2).to_dict())
Output:
missing: {'age_years': 120, 'metro_km': 90} duplicate rows: 18
area_sqft age_years metro_km price_lakh
count 3018.0 2898.0 2928.0 3018.0
mean 1345.9 9.0 3.1 116.2
std 4450.2 6.3 2.2 55.7
min 320.0 0.0 0.0 23.9
25% 905.2 4.0 1.5 75.8
50% 1145.0 8.0 2.6 108.0
75% 1476.0 12.0 4.1 143.1
max 146300.0 45.0 17.9 411.9
{'Kharadi': 402, 'Hadapsar': 393, 'Hinjewadi': 379, 'Wakad': 376, 'Viman Nagar': 372, 'Baner': 367, 'Kothrud': 359, 'Aundh': 345, 'kothrud ': 25}
price skew 1.31 corr with price: {'bhk': 0.77, 'area_sqft': 0.08, 'floor': 0.02, 'total_floors': -0.01, 'age_years': -0.13, 'metro_km': -0.07}
What EDA found:
| Finding | Evidence |
|---|---|
| Missing values | 120 ages, 90 metro distances |
| Duplicate rows | 18 (the same sale entered twice) |
| Typos | area max 146,300 sq ft (median 1,145): extra zeros |
| Inconsistent category | "kothrud " (lowercase, trailing space) |
| Skewed target | price skew 1.31: a long tail of expensive flats |
| A hidden signal | area vs price correlation only 0.08, because of 4 typos |
Step 4: cleaning, and splitting once
The plan, from the EDA:
- Drop the duplicates. Otherwise one copy can land in training and the other in the test: train-test contamination.
- Strip spaces and fix capitals in
locality. - Drop the 4 area typos (in a real job, ask the data owner first).
- Drop
price_per_sqft(leakage). - Keep the rows with missing age/metro; fill them inside the pipeline, learning the fill values from the training rows only.
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("/data/pune_flats.csv")
df = df.drop_duplicates() # 18 copied rows
df["locality"] = df["locality"].str.strip().str.title() # "kothrud " -> "Kothrud"
print(df[df.area_sqft > 20000][["locality", "bhk", "area_sqft", "price_lakh"]].to_string())
df = df[df.area_sqft < 20000] # 4 typos: extra zeros
df = df.drop(columns="price_per_sqft") # computed FROM the price: leak
print("rows", len(df), " localities", df.locality.nunique())
corr = df.select_dtypes("number").corr()["price_lakh"].drop("price_lakh")
print("corr with price:", corr.round(2).to_dict())
train, test = train_test_split(df, test_size=0.2, random_state=88) # split ONCE, before any modelling
train.to_csv("/data/flats_train.csv", index=False); test.to_csv("/data/flats_test.csv", index=False)
print("train", len(train), " test", len(test), "(locked away until the very end)")
Output:
locality bhk area_sqft price_lakh
1026 Baner 2 102200 101.1
1044 Kharadi 2 116100 119.6
1664 Kharadi 3 146300 143.0
2003 Hadapsar 2 122700 78.8
rows 2996 localities 8
corr with price: {'bhk': 0.77, 'area_sqft': 0.83, 'floor': 0.02, 'total_floors': -0.01, 'age_years': -0.13, 'metro_km': -0.07}
train 2396 test 600 (locked away until the very end)
The area-price correlation jumped from 0.08 to 0.83: cleaning revealed the strongest signal. The test file is now locked away until the very end.
Step 5: baselines
Real-life example: is the new coach any good? In a cricket academy, the no-skill guess is "every batter scores the team average"; a simple coaching plan is cheap and explainable; a star coach must beat both clearly.
- Baseline: a simple score to beat.
- Dummy baseline (
DummyRegressor): always predicts the training mean. - Linear baseline: the simplest real model, inside a pipeline:
SimpleImputer(strategy="median")fills gaps with the median,StandardScalerrescales each number column to mean 0 and spread 1,OneHotEncoderturns each text category into its own 0/1 column (locality_Baner,locality_Kothrud, ...).
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
tr = pd.read_csv("/data/flats_train.csv") # training rows only
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
num = ["bhk", "area_sqft", "floor", "total_floors", "age_years", "metro_km"]
prep = make_column_transformer(
(make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num), # fill gaps, then scale
(OneHotEncoder(handle_unknown="ignore"), ["locality", "parking", "furnishing"])) # text -> 0/1 columns
for name, model in [("dummy (mean)", DummyRegressor()), ("linear", make_pipeline(prep, LinearRegression()))]:
mae = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
print(f"{name:13} CV MAE {mae.mean():5.1f} lakh (folds {mae.round(1)})")
Output:
dummy (mean) CV MAE 41.1 lakh (folds [41. 40.6 41.9 40.8 41.2])
linear CV MAE 12.8 lakh (folds [13.3 12.6 12.6 12.7 12.6])
Because the imputer, scaler and encoder are inside the pipeline, cross_val_score re-fits them on each training fold only, so no information from the held-out fold leaks in.
Step 6: feature engineering
Feature engineering: making new input columns from the ones you have, using what you know about the problem.
| New feature | Why |
|---|---|
floor_ratio = floor / total_floors |
0 = ground, 1 = top floor |
age_missing |
1 if the seller didn't say; a gap can be a signal |
log_area = log(area) |
prices multiply (area × rate × extras); a log turns × into + |
Log target: TransformedTargetRegressor(..., func=np.log, inverse_func=np.exp) trains the model on log(price) and turns predictions back with exp. This suits multiplicative targets and the skewed price.
import numpy as np, pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.compose import make_column_transformer, TransformedTargetRegressor
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LinearRegression
tr = pd.read_csv("/data/flats_train.csv")
tr["log_area"], tr["floor_ratio"] = np.log(tr.area_sqft), tr.floor / tr.total_floors # from inputs only
tr["age_missing"] = tr.age_years.isna().astype(int) # "the seller didn't say"
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
num = ["bhk", "log_area", "floor_ratio", "age_years", "age_missing", "metro_km"]
prep = make_column_transformer((make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
(OneHotEncoder(handle_unknown="ignore"), ["locality", "parking", "furnishing"]))
linear = make_pipeline(prep, LinearRegression())
logged = TransformedTargetRegressor(linear, func=np.log, inverse_func=np.exp) # learn log(price)
for name, m in [("new features", linear), ("+ log target", logged)]:
mae = -cross_val_score(m, X, y, cv=5, scoring="neg_mean_absolute_error").mean()
print(f"{name:13} CV MAE {mae:5.1f} lakh")
Output:
new features CV MAE 14.8 lakh
+ log target CV MAE 9.4 lakh
The new features alone made the error worse (14.8 vs 12.8): log(area) fits log(price), not price. With the log target, both sides match and the error falls to 9.4. Features and target must fit together. Always measure, never assume.
Step 7: gradient boosting
Real-life example: a team of trainee valuers. Trainee 1 makes a rough guess; trainee 2 studies only trainee 1's mistakes and fixes them; 100 trainees each fix what is left by a small step (the learning rate); add up all the corrections. Each trainee is a small decision tree, a chain of yes/no questions such as "is the area over 1,200?". This is gradient boosting (Episodes 35-36). It captures patterns that are not straight lines: the new-building premium, the ground-floor discount, the luxury bump for big flats in some areas.
HistGradientBoostingRegressor bins each number into a histogram (fast), splits directly on category columns, and handles NaN by itself, so no imputer is needed.
import numpy as np, pandas as pd, time
from sklearn.model_selection import cross_val_score
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
tr = pd.read_csv("/data/flats_train.csv")
tr["floor_ratio"] = tr.floor / tr.total_floors
for c in ["locality", "parking", "furnishing"]: tr[c] = tr[c].astype("category") # tree-friendly
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", random_state=0) # NaN ok
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp)
t = time.perf_counter()
mae = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
print(f"gradient boosting CV MAE {mae.mean():.1f} lakh folds {mae.round(1)} {time.perf_counter() - t:.1f} s")
print("trees in the model:", model.fit(X, y).regressor_.n_iter_)
Output:
gradient boosting CV MAE 8.7 lakh folds [9.3 8.7 8.2 8.6 8.8] 0.9 s
trees in the model: 100
Step 8: tuning the hyperparameters
- Hyperparameter: a setting we choose before training (learning rate, number of trees, tree size). The model does not learn it.
- Grid search: try every combination: 4 × 3 × 3 × 3 = 108 settings.
- Random search (
RandomizedSearchCV): try a random sample (15 here). Usually almost as good, far cheaper. - Each setting is scored by cross-validation on the training data. Never tune on the test set.
The regressor__ prefix in the names means "the learning_rate of the step called regressor inside TransformedTargetRegressor".
import numpy as np, pandas as pd, time
from sklearn.model_selection import RandomizedSearchCV, cross_val_score
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
tr = pd.read_csv("/data/flats_train.csv"); tr["floor_ratio"] = tr.floor / tr.total_floors
for c in ["locality", "parking", "furnishing"]: tr[c] = tr[c].astype("category")
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", random_state=0)
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp)
grid = {"regressor__learning_rate": [0.03, 0.05, 0.1, 0.2], "regressor__max_iter": [100, 200, 400],
"regressor__max_leaf_nodes": [7, 15, 31], "regressor__min_samples_leaf": [10, 20, 40]}
search = RandomizedSearchCV(model, grid, n_iter=15, cv=3, scoring="neg_mean_absolute_error", random_state=0)
base = -cross_val_score(model, X, y, cv=3, scoring="neg_mean_absolute_error").mean()
t = time.perf_counter(); search.fit(X, y)
print(f"15 settings x 3 folds in {time.perf_counter() - t:.0f} s")
print(f"defaults CV MAE {base:.1f} lakh")
print(f"best CV MAE {-search.best_score_:.1f} lakh")
print("best settings:", {k.split("__")[1]: v for k, v in search.best_params_.items()})
print(f"worst CV MAE {-search.cv_results_['mean_test_score'].min():.1f} lakh")
Output:
15 settings x 3 folds in 13 s
defaults CV MAE 8.7 lakh
best CV MAE 8.3 lakh
best settings: {'min_samples_leaf': 10, 'max_leaf_nodes': 7, 'max_iter': 400, 'learning_rate': 0.05}
worst CV MAE 11.5 lakh
Tuning gave a modest gain (8.7 → 8.3) and protected us from a bad setting (11.5). The big wins came earlier: cleaning (area correlation 0.08 → 0.83) and the log target (12.8 → 9.4).
Step 9: open the test set once, and error analysis
import numpy as np, pandas as pd, joblib
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error as mae, mean_absolute_percentage_error as mape, r2_score
def load(f):
d = pd.read_csv(f); d["floor_ratio"] = d.floor / d.total_floors
for c in ["locality", "parking", "furnishing"]: d[c] = d[c].astype("category")
return d.drop(columns="price_lakh"), d["price_lakh"]
(Xtr, ytr), (Xte, yte) = load("/data/flats_train.csv"), load("/data/flats_test.csv")
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", learning_rate=0.05, max_iter=400,
max_leaf_nodes=7, min_samples_leaf=10, random_state=0) # tuned
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp).fit(Xtr, ytr)
joblib.dump(model, "/data/flat_price_v1.joblib"); p = model.predict(Xte) # save; test set opened ONCE
print(f"TEST MAE {mae(yte, p):.1f} lakh MAPE {mape(yte, p):.1%} R2 {r2_score(yte, p):.3f}")
print(f"within 10% of the real price: {(abs(p - yte) / yte < 0.10).mean():.0%} of flats")
err = Xte.assign(price=yte, pred=p.round(1), pct=(abs(p - yte) / yte * 100).round(1))
print("mean % error by BHK:", err.groupby("bhk")["pct"].mean().round(1).to_dict())
print("age missing?", err.groupby(err.age_years.isna())["pct"].mean().round(1).to_dict())
print(err.nlargest(3, "pct")[["locality", "bhk", "area_sqft", "age_years", "price", "pred", "pct"]])
Output:
TEST MAE 8.7 lakh MAPE 7.3% R2 0.956
within 10% of the real price: 74% of flats
mean % error by BHK: {1: 7.4, 2: 7.3, 3: 7.2, 4: 7.8}
age missing? {False: 7.3, True: 8.9}
locality bhk area_sqft age_years price pred pct
343 Baner 1 769 2.0 64.4 84.6 31.4
278 Hadapsar 1 454 7.0 26.1 32.8 25.7
541 Baner 4 1850 NaN 299.3 222.4 25.7
- CV said 8.3-8.7; the test says 8.7. The match means the process was honest.
- Error analysis: breaking the errors into groups and reading the worst cases. By BHK the errors are even (~7%); flats with a missing age are worse (8.9% vs 7.3%).
- We built 8% random noise into this made-up data, so about 6-7% error is the floor. Irreducible error is noise no model can remove (bargaining, a seller in a hurry). Our 7.3% is close to it.
- The most useful next step is data (ask sellers for the age), not a bigger model.
- Do not go back now and tweak until the test says 8.5: the test set is used once.
Step 10: using the model, and the model report
Permutation importance: shuffle one column and measure how much worse the error gets; a big increase means the model relies on that column (Episode 83). The range uses the test set's real/predicted ratios: 80% of test flats fell between the 10th and 90th percentile ratio.
import numpy as np, pandas as pd, joblib
from sklearn.inspection import permutation_importance
model = joblib.load("/data/flat_price_v1.joblib") # the saved pipeline from the last program
te = pd.read_csv("/data/flats_test.csv"); te["floor_ratio"] = te.floor / te.total_floors
for c in ["locality", "parking", "furnishing"]: te[c] = te[c].astype("category")
X, y = te.drop(columns="price_lakh"), te["price_lakh"]
imp = permutation_importance(model, X, y, scoring="neg_mean_absolute_error", n_repeats=3, random_state=0)
print("MAE gets worse by (lakh) when a column is shuffled:")
print(pd.Series(imp.importances_mean, X.columns).nlargest(5).round(1).to_dict())
lo, hi = np.percentile(y / model.predict(X), [10, 90]) # real / predicted, on the test flats
asha = {"locality": "Kothrud", "bhk": 2, "area_sqft": 950, "floor": 3, "total_floors": 7, "age_years": 6,
"parking": "yes", "furnishing": "semi", "metro_km": 0.8, "floor_ratio": 3 / 7} # new flat
asha = pd.DataFrame([asha])[X.columns].astype(X.dtypes)
p = model.predict(asha)[0]
print(f"Asha's flat: about {p:.0f} lakh, 80% range {p * lo:.0f} to {p * hi:.0f} lakh")
Output:
MAE gets worse by (lakh) when a column is shuffled:
{'area_sqft': 45.1, 'locality': 17.9, 'age_years': 2.6, 'bhk': 1.3, 'metro_km': 1.3}
Asha's flat: about 131 lakh, 80% range 118 to 149 lakh
All three brokers' quotes (118, 135, 150) were inside the 80% range; the middle one was closest to the model's 131.
The one-page model report
| Section | Content |
|---|---|
| Goal and target | A fair sale price for a Pune flat, in lakh |
| Data | 3,018 sales → 2,996 after removing 18 duplicates and 4 area typos; price_per_sqft dropped (leakage) |
| Method | Pipeline, log target, histogram gradient boosting, random search (15 settings × 3 folds) |
| Results | dummy 41.1 · linear 12.8 · log-linear 9.4 · boosting 8.7 · tuned 8.3 (CV MAE, lakh) → test MAE 8.7, MAPE 7.3%, R² 0.956, 74% within 10% |
| Errors | worse when the age is missing (8.9% vs 7.3%); biggest misses are small flats and a 4 BHK with no age |
| Limits | Pune only, these 8 localities, no villas; always quote the 80% range, not one number |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
Text columns into LinearRegression |
ValueError: could not convert string to float: 'Baner' |
models need numbers | OneHotEncoder in a pipeline |
Empty cells into LinearRegression |
ValueError: Input X contains NaN. |
it can't use gaps | SimpleImputer, fitted on train |
price_per_sqft as an input |
test R² 0.992 (too good) | target leakage | drop columns built from the target |
| Skipping EDA | area vs price correlation 0.08 | 4 typos hid the signal | describe(), then clean |
| Duplicates left in | the same sale in train and test | train-test contamination | drop_duplicates() before the split |
| New features without the log target | MAE 12.8 → 14.8 | features and target don't match | measure; add the log target |
| Tuning on the test set | a test score you can't trust | you chose the answer | CV on train; test once |
Mini project
Make it yours: fix the 4 typos by dividing by 100 instead of dropping them, add a new_building = age < 2 feature, and compare test MAE. Then write the one-page model report for your result.
Practice
1. Why is days_to_sell not allowed as an input when predicting the sale price?
Solution
It is only known after the sale. At prediction time it does not exist, so using it is target leakage.
2. Fix the typos instead of dropping them.
Solution
typo = df.area_sqft > 20000
df.loc[typo, "area_sqft"] = df.loc[typo, "area_sqft"] // 100 # 102200 -> 1022
Check each fixed row by eye (price per square foot should look like its neighbours') before trusting it.
3. Why must drop_duplicates() come before train_test_split?
Solution
If the same sale is in both sets, the model has effectively seen part of the test set, so the test score is too optimistic (train-test contamination).
4. Use RMSE instead of MAE as the CV scoring. What changes?
Solution
Use scoring="neg_root_mean_squared_error". RMSE squares errors before averaging, so big misses (expensive flats) count more; it is always ≥ MAE. Choose MAE when every lakh counts the same; RMSE when big misses are especially costly.
5. Why did the new features make the linear model worse before the log target?
Solution
log_area replaced area_sqft. A straight line from log(area) to price fits badly, because price grows roughly in proportion to area. With the log target, log(price) ≈ log(area) + log(rate) + ..., which is linear, so the model fits well.
Quick quiz
Predicting a flat's sale price, which column is safe: area_sqft, price_per_sqft, days_to_sell or broker_commission (2% of the price)? Only area_sqft. The others are computed from the price or measured after the sale: all three are leakage.
Interview questions
- Walk me through an end-to-end regression project. What do you do before training any model?
- What is target leakage? Give two examples and how you catch them.
- Your model scores R² 0.99 on the first try. What do you check?
- Why split before modelling, and why drop duplicates first?
- Why use a dummy baseline?
- When does a log target help? How do you turn predictions back?
- MAE vs RMSE vs MAPE: when would you choose each?
- Grid search vs random search: trade-offs?
- Your CV MAE is 8.3 and the test MAE is 8.7. Good or bad? What if the test were 15?
- What goes into a model report, and why include limits?
Glossary
| Term | Meaning, with an example |
|---|---|
| Project | One problem solved end to end, from raw data to a result someone uses. |
| End to end | Every step, from the raw CSV to the model report. |
| Regression | Predicting a number: a flat's price. |
| Target (label) | The column we predict: price_lakh. |
| Feature (input) | A column used to predict: area, locality, age. |
| Lakh / crore | 1,00,000 rupees / 1,00,00,000 rupees; 120 lakh = 1.2 crore. |
| BHK | Bedrooms, hall, kitchen: a 2 BHK has two bedrooms. |
| CSV | A plain-text table, one row per line, values separated by commas. |
| Framing | Deciding target, inputs, metric and success before any code. |
| MAE | Mean absolute error: average size of mistakes; 8.7 lakh on the test set. |
| MAPE | Mean absolute percentage error: average mistake as a % of the price; 7.3%. |
| RMSE | Root mean squared error: like MAE but big misses count more. |
| R² (R-squared) | Share of the price variation explained; 1 = perfect, 0.956 here. |
| Cross-validation (CV) | Score on several held-out slices of training data and average: 5 folds. |
| Fold | One of those slices. |
| Target leakage | An input that secretly contains the target: price_per_sqft. |
| Train-test contamination | The same (or nearly the same) row in train and test: duplicates. |
| EDA | Exploratory data analysis: summary statistics, missing values, typos, correlations, before modelling. |
| Summary statistics | count, mean, std, min, quartiles, max (describe()). |
| Missing value (NaN) | An empty cell; "not a number". 120 ages missing. |
| Duplicate row | The same record twice: 18 here. |
| Outlier / typo | A value far from the rest; area 146,300 sq ft was extra zeros. |
| Correlation | −1 to 1, how strongly two columns rise together: area vs price 0.83 after cleaning. |
| Skew | How lopsided a distribution is: price skew 1.31, a long tail of costly flats. |
| Data cleaning | Fixing or removing duplicates, typos, inconsistent categories and leaks. |
| Train / test split | Training rows to learn from, test rows kept locked for one final score (2,396 / 600). |
| Baseline | A simple score to beat. |
| Dummy baseline | Always predict the mean: CV MAE 41.1 lakh. |
| Linear baseline | The simplest real model: CV MAE 12.8 lakh. |
| Pipeline | Steps fitted together as one object, refitted inside each CV fold. |
| SimpleImputer | Fills missing values, e.g. with the training median. |
| StandardScaler | Rescales a column to mean 0, spread 1. |
| One-hot encoding | One 0/1 column per category: locality_Kothrud. |
| ColumnTransformer | Applies different steps to different columns (make_column_transformer). |
| Feature engineering | New inputs made from existing ones: floor_ratio, log_area. |
| Log target | Train on log(price), predict with exp: CV MAE 12.8 → 9.4. |
| Decision tree | A chain of yes/no questions ending in a prediction. |
| Gradient boosting | Many small trees, each fixing the previous ones' errors: CV MAE 8.7. |
| Learning rate | How big a step each new tree takes (0.05 after tuning). |
| HistGradientBoostingRegressor | scikit-learn's fast boosting; handles categories and NaN itself. |
| Hyperparameter | A setting chosen before training: learning rate, max_leaf_nodes. |
| Tuning | Searching for good hyperparameters with CV on training data. |
| Grid search | Try every combination (108 here). |
| Random search | Try a random sample of combinations (15 here). |
| Test set used once | Opened only at the end, for one honest score. |
| Error analysis | Splitting errors by group and reading the worst cases. |
| Irreducible error | Noise no model can remove; here the 8% noise we built in. |
| Permutation importance | Shuffle a column, see how much worse the error gets: area +45.1 lakh. |
| Prediction range | An interval from past errors: 118 to 149 lakh for Asha's flat. |
| joblib | Saves and loads Python objects such as a fitted pipeline. |
| Model report | One page: goal, data, method, results, errors, limits. |
The data generator
Run this once to make /data/pune_flats.csv (seed 88; every number is invented):
"""Make the made-up Pune flats dataset for ml_ep88 (house price project). Writes /data/pune_flats.csv.
Every number is invented (random, seed 88); locality names are only labels, not real market prices."""
import numpy as np, pandas as pd, os
rng = np.random.default_rng(88); n = 3000
loc = {"Kothrud": 11500, "Baner": 10500, "Aundh": 11000, "Viman Nagar": 10800, "Kharadi": 9200,
"Wakad": 7800, "Hinjewadi": 7000, "Hadapsar": 7200}
L = rng.choice(list(loc), n)
bhk = rng.choice([1, 2, 3, 4], n, p=[.2, .45, .28, .07])
area = np.round(bhk * rng.normal(480, 70, n) + rng.normal(120, 60, n)).clip(320, None)
total = rng.integers(4, 24, n); floor = np.minimum((rng.random(n) * (total + 1)).astype(int), total)
age = rng.gamma(2.0, 4.5, n).round(); metro = rng.gamma(2.2, 1.4, n).round(1)
parking = rng.random(n) < (0.35 + 0.15 * bhk); furn = rng.choice(["unfurnished", "semi", "full"], n, p=[.45, .4, .15])
rate = np.array([loc[x] for x in L])
price = area * rate * np.where(age < 2, 1.12, 1) * np.where(age > 20, 0.80, 1 - 0.006 * age) * np.where(parking, 1.06, 1)
price *= np.where(floor == 0, 0.92, 1) * np.where((floor == total) & (total > 10), 1.10, 1) # ground cheaper, top-floor views
price *= np.select([furn == "semi", furn == "full"], [1.04, 1.10], 1) * np.where(metro < 1, 1.12, 1)
price *= np.where((bhk == 4) & np.isin(L, ["Kothrud", "Aundh", "Baner"]), 1.25, 1) # luxury bump: an interaction
price = price * np.exp(rng.normal(0, 0.08, n)) / 1e5 # rupees -> lakh, 8% noise
df = pd.DataFrame({"locality": L, "bhk": bhk, "area_sqft": area.astype(int), "floor": floor, "total_floors": total,
"age_years": age, "parking": np.where(parking, "yes", "no"), "furnishing": furn, "metro_km": metro,
"price_lakh": price.round(1)})
df["price_per_sqft"] = (df.price_lakh * 1e5 / df.area_sqft).round() # leaks the answer
df.loc[rng.choice(n, 120, replace=False), "age_years"] = np.nan
df.loc[rng.choice(n, 90, replace=False), "metro_km"] = np.nan
typo = rng.choice(n, 4, replace=False); df.loc[typo, "area_sqft"] *= 100 # extra zeros typed
messy = rng.choice(np.where(df.locality == "Kothrud")[0], 25, replace=False); df.loc[messy, "locality"] = "kothrud "
df = pd.concat([df, df.sample(18, random_state=1)]).sample(frac=1, random_state=2).reset_index(drop=True)
os.makedirs("/data", exist_ok=True); df.to_csv("/data/pune_flats.csv", index=False); print(df.shape)
Summary
Frame the problem first (target, inputs known before the sale, metric, success). Be suspicious of a perfect score: price_per_sqft was leakage. EDA found duplicates, typos, a messy category and a skewed target, and cleaning lifted area's correlation from 0.08 to 0.83. Split once, beat a dummy (41.1) and a linear baseline (12.8), match features to a log target (9.4), boost (8.7), tune (8.3), open the test set once (8.7, MAPE 7.3%), analyse the errors, and write a report with honest limits.