SyntaxPilot
Courses/Machine Learning/Lesson 88
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 88 of 90

Project 1: house price prediction

end-to-end regression projectEDAcleaningfeature engineeringlinear baselinegradient boostingtuningerror analysismodel report

Machine Learning Full Course · Episode 88. Every program below was run on a Linux machine with Python 3.13, scikit-learn 1.9.1, pandas 3.0.5 and NumPy 2.5.3 (one CPU thread), and the output shown is the real output (timings will differ on your computer). The flats data is made up: the generator is at the end of these notes, so you can make exactly the same file. The locality names are only labels, not real market prices.

The big idea: a project is one problem solved end to end, from a raw file to a result someone else can trust. The order matters: frame the problem → look at the data (EDA) → clean it → split once → baselines → features → a stronger model → tuning → open the test set once → error analysis → a model report. Most of the improvement in this project came from cleaning and from matching features to the target, not from the fancy model.

Words you need first

  • Project: one problem solved end to end, from raw data to a result someone uses.
  • Regression: predicting a number (a price), not a class (Episode 1).
  • Lakh: 1,00,000 rupees. 120 lakh = 1.2 crore rupees.
  • MAE, mean absolute error: the average size of the mistakes. Predict 100, truth 110 → error 10 lakh (Episode 19).
  • Cross-validation (CV): split the training data into 5 slices; train on 4, score on the 5th, five times, and average (Episode 26).
  • Pipeline: a chain of steps (fill gaps → scale → model) that is fitted as one object (Episode 17).

Step 1: frame the problem

Real-life example: three brokers, three prices. Asha wants to sell her 2 BHK, 950 sq ft flat in Kothrud, Pune. Three brokers quote 118, 135 and 150 lakh: a 32 lakh gap. We have 3,018 past sales and want a fair price with an honest range.

Question Our answer
Target (label) price_lakh
Inputs only what a seller knows before the sale
Metric MAE in lakh (easy to explain) + MAPE, the mean absolute percentage error
Success most predictions within 10% of the real price

MAPE matters because a 10 lakh miss on a 50 lakh flat (20%) is much worse than on a 300 lakh flat (3.3%).

Step 2: three failed first attempts

Attempt 1: throw everything into a linear regression.

import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv("/data/pune_flats.csv")             # 3,018 made-up Pune flats
print(df.shape); print(df.head(3).to_string(index=False))
X, y = df.drop(columns="price_lakh"), df["price_lakh"]
try: model = LinearRegression().fit(X, y)            # obvious attempt: throw it all in
except ValueError as e: print("ValueError:", e)

Output:

(3018, 11)
   locality  bhk  area_sqft  floor  total_floors  age_years parking  furnishing  metro_km  price_lakh  price_per_sqft
      Baner    4       1795      2            21       10.0     yes        semi       1.0       231.1         12875.0
Viman Nagar    4       2164      0             6        7.0      no unfurnished       1.9       181.9          8406.0
  Hinjewadi    3       1596      3             6        0.0     yes unfurnished       NaN       140.5          8803.0
ValueError: could not convert string to float: 'Baner'

(We catch the ValueError and print its message so it is easy to read; without try, Python prints the full traceback ending in the same line.) A linear model only understands numbers, and locality is text.

Attempt 2: numbers only.

import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.read_csv("/data/pune_flats.csv")
X = df.select_dtypes("number").drop(columns="price_lakh")   # attempt 2: numbers only
try: model = LinearRegression().fit(X, df["price_lakh"])
except ValueError as e: print("ValueError:", str(e).split(". ")[0] + ".")

Output:

ValueError: Input X contains NaN.
LinearRegression does not accept missing values encoded as NaN natively.

NaN means "not a number": an empty cell. Some sellers never gave the building's age.

Attempt 3: drop the gaps, numbers only, a gradient boosting model.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score
df = pd.read_csv("/data/pune_flats.csv").dropna()            # attempt 3: drop rows with gaps
X, y = df.select_dtypes("number").drop(columns="price_lakh"), df["price_lakh"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)
m = HistGradientBoostingRegressor(random_state=0).fit(Xtr, ytr)
p = m.predict(Xte)
print(f"rows left {len(df)}   test R2 {r2_score(yte, p):.3f}   MAE {mean_absolute_error(yte, p):.1f} lakh")
print("columns used:", list(X.columns))

Output:

rows left 2809   test R2 0.992   MAE 1.8 lakh
columns used: ['bhk', 'area_sqft', 'floor', 'total_floors', 'age_years', 'metro_km', 'price_per_sqft']

R² (R-squared) is the share of the variation in prices the model explains; 1 is perfect. 0.992 looks like a miracle, and that is exactly why you should be suspicious. price_per_sqft = price ÷ area is computed from the answer. Asha doesn't know it before the sale. This is target leakage: an input that secretly contains the target. Ask of every column: would I know this at prediction time? dropna also threw away 209 sales, and we had not looked at the data at all.

Step 3: exploratory data analysis (EDA)

Real-life example: a doctor's check-up. Basic numbers first (summary statistics), then missing reports and wrong entries (missing values, duplicates, typos), then what moves together (correlation and skew).

  • Correlation (Pearson): a number from −1 to 1 for how strongly two columns rise together.
  • Skew: how lopsided a distribution is; positive skew = a long tail of big values.
import pandas as pd
df = pd.read_csv("/data/pune_flats.csv")
print("missing:", df.isna().sum()[lambda s: s > 0].to_dict(), "  duplicate rows:", df.duplicated().sum())
print(df[["area_sqft", "age_years", "metro_km", "price_lakh"]].describe().round(1).to_string())
print(df["locality"].value_counts().to_dict())
corr = df.select_dtypes("number").corr()["price_lakh"].drop(["price_lakh", "price_per_sqft"])
print(f"price skew {df.price_lakh.skew():.2f}   corr with price:", corr.round(2).to_dict())

Output:

missing: {'age_years': 120, 'metro_km': 90}   duplicate rows: 18
       area_sqft  age_years  metro_km  price_lakh
count     3018.0     2898.0    2928.0      3018.0
mean      1345.9        9.0       3.1       116.2
std       4450.2        6.3       2.2        55.7
min        320.0        0.0       0.0        23.9
25%        905.2        4.0       1.5        75.8
50%       1145.0        8.0       2.6       108.0
75%       1476.0       12.0       4.1       143.1
max     146300.0       45.0      17.9       411.9
{'Kharadi': 402, 'Hadapsar': 393, 'Hinjewadi': 379, 'Wakad': 376, 'Viman Nagar': 372, 'Baner': 367, 'Kothrud': 359, 'Aundh': 345, 'kothrud ': 25}
price skew 1.31   corr with price: {'bhk': 0.77, 'area_sqft': 0.08, 'floor': 0.02, 'total_floors': -0.01, 'age_years': -0.13, 'metro_km': -0.07}

What EDA found:

Finding Evidence
Missing values 120 ages, 90 metro distances
Duplicate rows 18 (the same sale entered twice)
Typos area max 146,300 sq ft (median 1,145): extra zeros
Inconsistent category "kothrud " (lowercase, trailing space)
Skewed target price skew 1.31: a long tail of expensive flats
A hidden signal area vs price correlation only 0.08, because of 4 typos

Step 4: cleaning, and splitting once

The plan, from the EDA:

  1. Drop the duplicates. Otherwise one copy can land in training and the other in the test: train-test contamination.
  2. Strip spaces and fix capitals in locality.
  3. Drop the 4 area typos (in a real job, ask the data owner first).
  4. Drop price_per_sqft (leakage).
  5. Keep the rows with missing age/metro; fill them inside the pipeline, learning the fill values from the training rows only.
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("/data/pune_flats.csv")
df = df.drop_duplicates()                                    # 18 copied rows
df["locality"] = df["locality"].str.strip().str.title()      # "kothrud " -> "Kothrud"
print(df[df.area_sqft > 20000][["locality", "bhk", "area_sqft", "price_lakh"]].to_string())
df = df[df.area_sqft < 20000]                                # 4 typos: extra zeros
df = df.drop(columns="price_per_sqft")                       # computed FROM the price: leak
print("rows", len(df), "  localities", df.locality.nunique())
corr = df.select_dtypes("number").corr()["price_lakh"].drop("price_lakh")
print("corr with price:", corr.round(2).to_dict())
train, test = train_test_split(df, test_size=0.2, random_state=88)   # split ONCE, before any modelling
train.to_csv("/data/flats_train.csv", index=False); test.to_csv("/data/flats_test.csv", index=False)
print("train", len(train), "  test", len(test), "(locked away until the very end)")

Output:

      locality  bhk  area_sqft  price_lakh
1026     Baner    2     102200       101.1
1044   Kharadi    2     116100       119.6
1664   Kharadi    3     146300       143.0
2003  Hadapsar    2     122700        78.8
rows 2996   localities 8
corr with price: {'bhk': 0.77, 'area_sqft': 0.83, 'floor': 0.02, 'total_floors': -0.01, 'age_years': -0.13, 'metro_km': -0.07}
train 2396   test 600 (locked away until the very end)

The area-price correlation jumped from 0.08 to 0.83: cleaning revealed the strongest signal. The test file is now locked away until the very end.

Step 5: baselines

Real-life example: is the new coach any good? In a cricket academy, the no-skill guess is "every batter scores the team average"; a simple coaching plan is cheap and explainable; a star coach must beat both clearly.

  • Baseline: a simple score to beat.
  • Dummy baseline (DummyRegressor): always predicts the training mean.
  • Linear baseline: the simplest real model, inside a pipeline: SimpleImputer(strategy="median") fills gaps with the median, StandardScaler rescales each number column to mean 0 and spread 1, OneHotEncoder turns each text category into its own 0/1 column (locality_Baner, locality_Kothrud, ...).
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
tr = pd.read_csv("/data/flats_train.csv")                    # training rows only
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
num = ["bhk", "area_sqft", "floor", "total_floors", "age_years", "metro_km"]
prep = make_column_transformer(
    (make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),     # fill gaps, then scale
    (OneHotEncoder(handle_unknown="ignore"), ["locality", "parking", "furnishing"]))   # text -> 0/1 columns
for name, model in [("dummy (mean)", DummyRegressor()), ("linear", make_pipeline(prep, LinearRegression()))]:
    mae = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
    print(f"{name:13} CV MAE {mae.mean():5.1f} lakh  (folds {mae.round(1)})")

Output:

dummy (mean)  CV MAE  41.1 lakh  (folds [41.  40.6 41.9 40.8 41.2])
linear        CV MAE  12.8 lakh  (folds [13.3 12.6 12.6 12.7 12.6])

Because the imputer, scaler and encoder are inside the pipeline, cross_val_score re-fits them on each training fold only, so no information from the held-out fold leaks in.

Step 6: feature engineering

Feature engineering: making new input columns from the ones you have, using what you know about the problem.

New feature Why
floor_ratio = floor / total_floors 0 = ground, 1 = top floor
age_missing 1 if the seller didn't say; a gap can be a signal
log_area = log(area) prices multiply (area × rate × extras); a log turns × into +

Log target: TransformedTargetRegressor(..., func=np.log, inverse_func=np.exp) trains the model on log(price) and turns predictions back with exp. This suits multiplicative targets and the skewed price.

import numpy as np, pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.compose import make_column_transformer, TransformedTargetRegressor
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LinearRegression
tr = pd.read_csv("/data/flats_train.csv")
tr["log_area"], tr["floor_ratio"] = np.log(tr.area_sqft), tr.floor / tr.total_floors   # from inputs only
tr["age_missing"] = tr.age_years.isna().astype(int)          # "the seller didn't say"
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
num = ["bhk", "log_area", "floor_ratio", "age_years", "age_missing", "metro_km"]
prep = make_column_transformer((make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), num),
                               (OneHotEncoder(handle_unknown="ignore"), ["locality", "parking", "furnishing"]))
linear = make_pipeline(prep, LinearRegression())
logged = TransformedTargetRegressor(linear, func=np.log, inverse_func=np.exp)   # learn log(price)
for name, m in [("new features", linear), ("+ log target", logged)]:
    mae = -cross_val_score(m, X, y, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name:13} CV MAE {mae:5.1f} lakh")

Output:

new features  CV MAE  14.8 lakh
+ log target  CV MAE   9.4 lakh

The new features alone made the error worse (14.8 vs 12.8): log(area) fits log(price), not price. With the log target, both sides match and the error falls to 9.4. Features and target must fit together. Always measure, never assume.

Step 7: gradient boosting

Real-life example: a team of trainee valuers. Trainee 1 makes a rough guess; trainee 2 studies only trainee 1's mistakes and fixes them; 100 trainees each fix what is left by a small step (the learning rate); add up all the corrections. Each trainee is a small decision tree, a chain of yes/no questions such as "is the area over 1,200?". This is gradient boosting (Episodes 35-36). It captures patterns that are not straight lines: the new-building premium, the ground-floor discount, the luxury bump for big flats in some areas.

HistGradientBoostingRegressor bins each number into a histogram (fast), splits directly on category columns, and handles NaN by itself, so no imputer is needed.

import numpy as np, pandas as pd, time
from sklearn.model_selection import cross_val_score
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
tr = pd.read_csv("/data/flats_train.csv")
tr["floor_ratio"] = tr.floor / tr.total_floors
for c in ["locality", "parking", "furnishing"]: tr[c] = tr[c].astype("category")   # tree-friendly
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", random_state=0)   # NaN ok
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp)
t = time.perf_counter()
mae = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_absolute_error")
print(f"gradient boosting  CV MAE {mae.mean():.1f} lakh  folds {mae.round(1)}  {time.perf_counter() - t:.1f} s")
print("trees in the model:", model.fit(X, y).regressor_.n_iter_)

Output:

gradient boosting  CV MAE 8.7 lakh  folds [9.3 8.7 8.2 8.6 8.8]  0.9 s
trees in the model: 100

Step 8: tuning the hyperparameters

  • Hyperparameter: a setting we choose before training (learning rate, number of trees, tree size). The model does not learn it.
  • Grid search: try every combination: 4 × 3 × 3 × 3 = 108 settings.
  • Random search (RandomizedSearchCV): try a random sample (15 here). Usually almost as good, far cheaper.
  • Each setting is scored by cross-validation on the training data. Never tune on the test set.

The regressor__ prefix in the names means "the learning_rate of the step called regressor inside TransformedTargetRegressor".

import numpy as np, pandas as pd, time
from sklearn.model_selection import RandomizedSearchCV, cross_val_score
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
tr = pd.read_csv("/data/flats_train.csv"); tr["floor_ratio"] = tr.floor / tr.total_floors
for c in ["locality", "parking", "furnishing"]: tr[c] = tr[c].astype("category")
X, y = tr.drop(columns="price_lakh"), tr["price_lakh"]
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", random_state=0)
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp)
grid = {"regressor__learning_rate": [0.03, 0.05, 0.1, 0.2], "regressor__max_iter": [100, 200, 400],
        "regressor__max_leaf_nodes": [7, 15, 31], "regressor__min_samples_leaf": [10, 20, 40]}
search = RandomizedSearchCV(model, grid, n_iter=15, cv=3, scoring="neg_mean_absolute_error", random_state=0)
base = -cross_val_score(model, X, y, cv=3, scoring="neg_mean_absolute_error").mean()
t = time.perf_counter(); search.fit(X, y)
print(f"15 settings x 3 folds in {time.perf_counter() - t:.0f} s")
print(f"defaults    CV MAE {base:.1f} lakh")
print(f"best CV MAE {-search.best_score_:.1f} lakh")
print("best settings:", {k.split("__")[1]: v for k, v in search.best_params_.items()})
print(f"worst       CV MAE {-search.cv_results_['mean_test_score'].min():.1f} lakh")

Output:

15 settings x 3 folds in 13 s
defaults    CV MAE 8.7 lakh
best CV MAE 8.3 lakh
best settings: {'min_samples_leaf': 10, 'max_leaf_nodes': 7, 'max_iter': 400, 'learning_rate': 0.05}
worst       CV MAE 11.5 lakh

Tuning gave a modest gain (8.7 → 8.3) and protected us from a bad setting (11.5). The big wins came earlier: cleaning (area correlation 0.08 → 0.83) and the log target (12.8 → 9.4).

Step 9: open the test set once, and error analysis

import numpy as np, pandas as pd, joblib
from sklearn.compose import TransformedTargetRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error as mae, mean_absolute_percentage_error as mape, r2_score
def load(f):
    d = pd.read_csv(f); d["floor_ratio"] = d.floor / d.total_floors
    for c in ["locality", "parking", "furnishing"]: d[c] = d[c].astype("category")
    return d.drop(columns="price_lakh"), d["price_lakh"]
(Xtr, ytr), (Xte, yte) = load("/data/flats_train.csv"), load("/data/flats_test.csv")
gb = HistGradientBoostingRegressor(categorical_features="from_dtype", learning_rate=0.05, max_iter=400,
                                   max_leaf_nodes=7, min_samples_leaf=10, random_state=0)   # tuned
model = TransformedTargetRegressor(gb, func=np.log, inverse_func=np.exp).fit(Xtr, ytr)
joblib.dump(model, "/data/flat_price_v1.joblib"); p = model.predict(Xte)   # save; test set opened ONCE
print(f"TEST  MAE {mae(yte, p):.1f} lakh   MAPE {mape(yte, p):.1%}   R2 {r2_score(yte, p):.3f}")
print(f"within 10% of the real price: {(abs(p - yte) / yte < 0.10).mean():.0%} of flats")
err = Xte.assign(price=yte, pred=p.round(1), pct=(abs(p - yte) / yte * 100).round(1))
print("mean % error by BHK:", err.groupby("bhk")["pct"].mean().round(1).to_dict())
print("age missing?", err.groupby(err.age_years.isna())["pct"].mean().round(1).to_dict())
print(err.nlargest(3, "pct")[["locality", "bhk", "area_sqft", "age_years", "price", "pred", "pct"]])

Output:

TEST  MAE 8.7 lakh   MAPE 7.3%   R2 0.956
within 10% of the real price: 74% of flats
mean % error by BHK: {1: 7.4, 2: 7.3, 3: 7.2, 4: 7.8}
age missing? {False: 7.3, True: 8.9}
     locality  bhk  area_sqft  age_years  price   pred   pct
343     Baner    1        769        2.0   64.4   84.6  31.4
278  Hadapsar    1        454        7.0   26.1   32.8  25.7
541     Baner    4       1850        NaN  299.3  222.4  25.7
  • CV said 8.3-8.7; the test says 8.7. The match means the process was honest.
  • Error analysis: breaking the errors into groups and reading the worst cases. By BHK the errors are even (~7%); flats with a missing age are worse (8.9% vs 7.3%).
  • We built 8% random noise into this made-up data, so about 6-7% error is the floor. Irreducible error is noise no model can remove (bargaining, a seller in a hurry). Our 7.3% is close to it.
  • The most useful next step is data (ask sellers for the age), not a bigger model.
  • Do not go back now and tweak until the test says 8.5: the test set is used once.

Step 10: using the model, and the model report

Permutation importance: shuffle one column and measure how much worse the error gets; a big increase means the model relies on that column (Episode 83). The range uses the test set's real/predicted ratios: 80% of test flats fell between the 10th and 90th percentile ratio.

import numpy as np, pandas as pd, joblib
from sklearn.inspection import permutation_importance
model = joblib.load("/data/flat_price_v1.joblib")            # the saved pipeline from the last program
te = pd.read_csv("/data/flats_test.csv"); te["floor_ratio"] = te.floor / te.total_floors
for c in ["locality", "parking", "furnishing"]: te[c] = te[c].astype("category")
X, y = te.drop(columns="price_lakh"), te["price_lakh"]
imp = permutation_importance(model, X, y, scoring="neg_mean_absolute_error", n_repeats=3, random_state=0)
print("MAE gets worse by (lakh) when a column is shuffled:")
print(pd.Series(imp.importances_mean, X.columns).nlargest(5).round(1).to_dict())
lo, hi = np.percentile(y / model.predict(X), [10, 90])      # real / predicted, on the test flats
asha = {"locality": "Kothrud", "bhk": 2, "area_sqft": 950, "floor": 3, "total_floors": 7, "age_years": 6,
        "parking": "yes", "furnishing": "semi", "metro_km": 0.8, "floor_ratio": 3 / 7}   # new flat
asha = pd.DataFrame([asha])[X.columns].astype(X.dtypes)
p = model.predict(asha)[0]
print(f"Asha's flat: about {p:.0f} lakh, 80% range {p * lo:.0f} to {p * hi:.0f} lakh")

Output:

MAE gets worse by (lakh) when a column is shuffled:
{'area_sqft': 45.1, 'locality': 17.9, 'age_years': 2.6, 'bhk': 1.3, 'metro_km': 1.3}
Asha's flat: about 131 lakh, 80% range 118 to 149 lakh

All three brokers' quotes (118, 135, 150) were inside the 80% range; the middle one was closest to the model's 131.

The one-page model report

Section Content
Goal and target A fair sale price for a Pune flat, in lakh
Data 3,018 sales → 2,996 after removing 18 duplicates and 4 area typos; price_per_sqft dropped (leakage)
Method Pipeline, log target, histogram gradient boosting, random search (15 settings × 3 folds)
Results dummy 41.1 · linear 12.8 · log-linear 9.4 · boosting 8.7 · tuned 8.3 (CV MAE, lakh) → test MAE 8.7, MAPE 7.3%, R² 0.956, 74% within 10%
Errors worse when the age is missing (8.9% vs 7.3%); biggest misses are small flats and a 4 BHK with no age
Limits Pune only, these 8 localities, no villas; always quote the 80% range, not one number

Common mistakes

Mistake What you see Why Fix
Text columns into LinearRegression ValueError: could not convert string to float: 'Baner' models need numbers OneHotEncoder in a pipeline
Empty cells into LinearRegression ValueError: Input X contains NaN. it can't use gaps SimpleImputer, fitted on train
price_per_sqft as an input test R² 0.992 (too good) target leakage drop columns built from the target
Skipping EDA area vs price correlation 0.08 4 typos hid the signal describe(), then clean
Duplicates left in the same sale in train and test train-test contamination drop_duplicates() before the split
New features without the log target MAE 12.8 → 14.8 features and target don't match measure; add the log target
Tuning on the test set a test score you can't trust you chose the answer CV on train; test once

Mini project

Make it yours: fix the 4 typos by dividing by 100 instead of dropping them, add a new_building = age < 2 feature, and compare test MAE. Then write the one-page model report for your result.

Practice

1. Why is days_to_sell not allowed as an input when predicting the sale price?

Solution

It is only known after the sale. At prediction time it does not exist, so using it is target leakage.

2. Fix the typos instead of dropping them.

Solution
typo = df.area_sqft > 20000
df.loc[typo, "area_sqft"] = df.loc[typo, "area_sqft"] // 100   # 102200 -> 1022

Check each fixed row by eye (price per square foot should look like its neighbours') before trusting it.

3. Why must drop_duplicates() come before train_test_split?

Solution

If the same sale is in both sets, the model has effectively seen part of the test set, so the test score is too optimistic (train-test contamination).

4. Use RMSE instead of MAE as the CV scoring. What changes?

Solution

Use scoring="neg_root_mean_squared_error". RMSE squares errors before averaging, so big misses (expensive flats) count more; it is always ≥ MAE. Choose MAE when every lakh counts the same; RMSE when big misses are especially costly.

5. Why did the new features make the linear model worse before the log target?

Solution

log_area replaced area_sqft. A straight line from log(area) to price fits badly, because price grows roughly in proportion to area. With the log target, log(price) ≈ log(area) + log(rate) + ..., which is linear, so the model fits well.

Quick quiz

Predicting a flat's sale price, which column is safe: area_sqft, price_per_sqft, days_to_sell or broker_commission (2% of the price)? Only area_sqft. The others are computed from the price or measured after the sale: all three are leakage.

Interview questions

  1. Walk me through an end-to-end regression project. What do you do before training any model?
  2. What is target leakage? Give two examples and how you catch them.
  3. Your model scores R² 0.99 on the first try. What do you check?
  4. Why split before modelling, and why drop duplicates first?
  5. Why use a dummy baseline?
  6. When does a log target help? How do you turn predictions back?
  7. MAE vs RMSE vs MAPE: when would you choose each?
  8. Grid search vs random search: trade-offs?
  9. Your CV MAE is 8.3 and the test MAE is 8.7. Good or bad? What if the test were 15?
  10. What goes into a model report, and why include limits?

Glossary

Term Meaning, with an example
Project One problem solved end to end, from raw data to a result someone uses.
End to end Every step, from the raw CSV to the model report.
Regression Predicting a number: a flat's price.
Target (label) The column we predict: price_lakh.
Feature (input) A column used to predict: area, locality, age.
Lakh / crore 1,00,000 rupees / 1,00,00,000 rupees; 120 lakh = 1.2 crore.
BHK Bedrooms, hall, kitchen: a 2 BHK has two bedrooms.
CSV A plain-text table, one row per line, values separated by commas.
Framing Deciding target, inputs, metric and success before any code.
MAE Mean absolute error: average size of mistakes; 8.7 lakh on the test set.
MAPE Mean absolute percentage error: average mistake as a % of the price; 7.3%.
RMSE Root mean squared error: like MAE but big misses count more.
R² (R-squared) Share of the price variation explained; 1 = perfect, 0.956 here.
Cross-validation (CV) Score on several held-out slices of training data and average: 5 folds.
Fold One of those slices.
Target leakage An input that secretly contains the target: price_per_sqft.
Train-test contamination The same (or nearly the same) row in train and test: duplicates.
EDA Exploratory data analysis: summary statistics, missing values, typos, correlations, before modelling.
Summary statistics count, mean, std, min, quartiles, max (describe()).
Missing value (NaN) An empty cell; "not a number". 120 ages missing.
Duplicate row The same record twice: 18 here.
Outlier / typo A value far from the rest; area 146,300 sq ft was extra zeros.
Correlation −1 to 1, how strongly two columns rise together: area vs price 0.83 after cleaning.
Skew How lopsided a distribution is: price skew 1.31, a long tail of costly flats.
Data cleaning Fixing or removing duplicates, typos, inconsistent categories and leaks.
Train / test split Training rows to learn from, test rows kept locked for one final score (2,396 / 600).
Baseline A simple score to beat.
Dummy baseline Always predict the mean: CV MAE 41.1 lakh.
Linear baseline The simplest real model: CV MAE 12.8 lakh.
Pipeline Steps fitted together as one object, refitted inside each CV fold.
SimpleImputer Fills missing values, e.g. with the training median.
StandardScaler Rescales a column to mean 0, spread 1.
One-hot encoding One 0/1 column per category: locality_Kothrud.
ColumnTransformer Applies different steps to different columns (make_column_transformer).
Feature engineering New inputs made from existing ones: floor_ratio, log_area.
Log target Train on log(price), predict with exp: CV MAE 12.8 → 9.4.
Decision tree A chain of yes/no questions ending in a prediction.
Gradient boosting Many small trees, each fixing the previous ones' errors: CV MAE 8.7.
Learning rate How big a step each new tree takes (0.05 after tuning).
HistGradientBoostingRegressor scikit-learn's fast boosting; handles categories and NaN itself.
Hyperparameter A setting chosen before training: learning rate, max_leaf_nodes.
Tuning Searching for good hyperparameters with CV on training data.
Grid search Try every combination (108 here).
Random search Try a random sample of combinations (15 here).
Test set used once Opened only at the end, for one honest score.
Error analysis Splitting errors by group and reading the worst cases.
Irreducible error Noise no model can remove; here the 8% noise we built in.
Permutation importance Shuffle a column, see how much worse the error gets: area +45.1 lakh.
Prediction range An interval from past errors: 118 to 149 lakh for Asha's flat.
joblib Saves and loads Python objects such as a fitted pipeline.
Model report One page: goal, data, method, results, errors, limits.

The data generator

Run this once to make /data/pune_flats.csv (seed 88; every number is invented):

"""Make the made-up Pune flats dataset for ml_ep88 (house price project). Writes /data/pune_flats.csv.
Every number is invented (random, seed 88); locality names are only labels, not real market prices."""
import numpy as np, pandas as pd, os
rng = np.random.default_rng(88); n = 3000
loc = {"Kothrud": 11500, "Baner": 10500, "Aundh": 11000, "Viman Nagar": 10800, "Kharadi": 9200,
       "Wakad": 7800, "Hinjewadi": 7000, "Hadapsar": 7200}
L = rng.choice(list(loc), n)
bhk = rng.choice([1, 2, 3, 4], n, p=[.2, .45, .28, .07])
area = np.round(bhk * rng.normal(480, 70, n) + rng.normal(120, 60, n)).clip(320, None)
total = rng.integers(4, 24, n); floor = np.minimum((rng.random(n) * (total + 1)).astype(int), total)
age = rng.gamma(2.0, 4.5, n).round(); metro = rng.gamma(2.2, 1.4, n).round(1)
parking = rng.random(n) < (0.35 + 0.15 * bhk); furn = rng.choice(["unfurnished", "semi", "full"], n, p=[.45, .4, .15])
rate = np.array([loc[x] for x in L])
price = area * rate * np.where(age < 2, 1.12, 1) * np.where(age > 20, 0.80, 1 - 0.006 * age) * np.where(parking, 1.06, 1)
price *= np.where(floor == 0, 0.92, 1) * np.where((floor == total) & (total > 10), 1.10, 1)          # ground cheaper, top-floor views
price *= np.select([furn == "semi", furn == "full"], [1.04, 1.10], 1) * np.where(metro < 1, 1.12, 1)
price *= np.where((bhk == 4) & np.isin(L, ["Kothrud", "Aundh", "Baner"]), 1.25, 1)   # luxury bump: an interaction
price = price * np.exp(rng.normal(0, 0.08, n)) / 1e5                                 # rupees -> lakh, 8% noise
df = pd.DataFrame({"locality": L, "bhk": bhk, "area_sqft": area.astype(int), "floor": floor, "total_floors": total,
                   "age_years": age, "parking": np.where(parking, "yes", "no"), "furnishing": furn, "metro_km": metro,
                   "price_lakh": price.round(1)})
df["price_per_sqft"] = (df.price_lakh * 1e5 / df.area_sqft).round()                   # leaks the answer
df.loc[rng.choice(n, 120, replace=False), "age_years"] = np.nan
df.loc[rng.choice(n, 90, replace=False), "metro_km"] = np.nan
typo = rng.choice(n, 4, replace=False); df.loc[typo, "area_sqft"] *= 100                # extra zeros typed
messy = rng.choice(np.where(df.locality == "Kothrud")[0], 25, replace=False); df.loc[messy, "locality"] = "kothrud "
df = pd.concat([df, df.sample(18, random_state=1)]).sample(frac=1, random_state=2).reset_index(drop=True)
os.makedirs("/data", exist_ok=True); df.to_csv("/data/pune_flats.csv", index=False); print(df.shape)

Summary

Frame the problem first (target, inputs known before the sale, metric, success). Be suspicious of a perfect score: price_per_sqft was leakage. EDA found duplicates, typos, a messy category and a skewed target, and cleaning lifted area's correlation from 0.08 to 0.83. Split once, beat a dummy (41.1) and a linear baseline (12.8), match features to a log target (9.4), boost (8.7), tune (8.3), open the test set once (8.7, MAPE 7.3%), analyse the errors, and write a report with honest limits.