SyntaxPilot
Courses/Machine Learning/Lesson 38
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 38 of 90

Feature engineering 1: numerical features

feature engineeringbinning (discretisation)log transformpower transforms (Box-Cox, Yeo-Johnson)interaction featuresratio and difference featuresaggregationsdomain-driven features

Machine Learning Full Course · Episode 38. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Imran's 3,000 Pune flats are made up for teaching (the hidden rule that creates the rents is in the first lines of every program, so you can see exactly what the model is trying to discover).

The big idea: feature engineering means creating new input columns, or reshaping existing ones, so the pattern is easy for a model to see (from a date of birth you make an age column). We keep one model, plain linear regression, and only change the columns. A model on the raw columns predicts a negative rent. The log transform fixes skewed, percent-style (multiplicative) data; power transforms (Box-Cox, Yeo-Johnson) choose how strongly to squeeze; binning lets a linear model follow a relationship that goes up and then down; ratio, difference and interaction features combine columns; aggregations summarise a group; and domain-driven features bring in what an expert already knows. The median error falls from 13.3% to 6.7%.

Sources for the library and paper facts: the scikit-learn documentation for PowerTransformer (methods, defaults, maximum-likelihood λ, and the two papers: G.E.P. Box and D.R. Cox, "An Analysis of Transformations", Journal of the Royal Statistical Society B, 26, 211-252, 1964; I.K. Yeo and R.A. Johnson, "A new family of power transformations to improve normality or symmetry", Biometrika, 87(4), 954-959, 2000) and for KBinsDiscretizer (version 1.9.1: n_bins, encode, strategy).

Words you need first

  • Feature, label (target), regression: Episode 1. Here the features are the flat's columns and the label is the monthly rent.
  • Linear regression, weights, residual: Episode 18. It predicts w1·x1 + w2·x2 + ... + b: it can only add columns times weights.
  • One-hot encoding: one 0/1 column per category (Episode 16).
  • Median, mean, histogram: Episodes 11 and 13.
  • Train / test split, target leakage: Episode 15.
  • Maximum likelihood: choosing the parameter that makes the observed data most probable (Episode 12).
  • Decision tree split: a question like "is area ≤ 700?" (Episodes 29-30).

A negative rent

Imran is a rental broker in Pune with records of 3,000 flats in six localities (Kothrud, Wakad, Hadapsar, Baner, Aundh, Pashan). For each flat he knows the size in square feet, the floor, the building's total floors, whether there is a lift, its age and the distance to the railway station. Owners ask him what rent to ask, so he wants a model that predicts the monthly rent. We train on 2,400 flats and test on the other 600. The obvious attempt: linear regression on the columns as they are.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
print(df[["area_sqft", "floor", "lift", "locality", "rent"]].head(3).to_string())
X = pd.get_dummies(df.drop(columns="rent"), dtype=float)              # locality -> 6 one-hot columns (ep16)
p = LinearRegression().fit(X[:2400], tr.rent).predict(X[2400:])         # the obvious attempt
e = abs(p - te.rent) / te.rent * 100                                   # % error per test flat
print(f"median error {e.median():.1f}%    flats under 500 sqft: {e[te.area_sqft <= 500].mean():.1f}%")
r = te.iloc[p.argmin()]; print("lowest prediction: Rs", round(p.min()), "| real:", r.rent, "| sqft:", r.area_sqft)

Output:

   area_sqft  floor  lift  locality     rent
0     1409.0     21     1     Wakad  21500.0
1      782.0      6     1  Hadapsar  11400.0
2      830.0      2     1     Baner  12700.0
median error 13.3%    flats under 500 sqft: 33.4%
lowest prediction: Rs -2315 | real: 2800.0 | sqft: 312.0
  • Median error 13.3%, but 33.4% on flats under 500 sqft.
  • The lowest prediction is Rs -2,315 for a 312 sqft flat whose real rent is Rs 2,800. A negative rent is impossible.

Why it failed. Linear regression adds fixed amounts: so many rupees per square foot, so many rupees for Aundh, the same for every flat. Real rents work in percent: a good locality makes a small flat and a big flat both roughly 40% more expensive. Effects that multiply instead of add are called multiplicative. The line is pulled hardest by the big expensive flats, so for tiny flats it overshoots downwards, even below zero. On top of that, rent is skewed.

Skew and the log transform

Skew measures how lopsided a set of values is. Example: salaries in a company. Most people earn a modest amount and a few earn enormous amounts, so the values bunch on one side with a long tail on the other. A skew of 0 means symmetric; a positive skew means a long tail to the right; a negative skew, a long tail to the left. In a right-skewed column the mean is pulled above the median by the few huge values.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
print("skew of rent:", round(df.rent.skew(), 2), "  skew of area:", round(df.area_sqft.skew(), 2))
print("mean rent:", round(df.rent.mean()), "  median rent:", df.rent.median(), "  max:", df.rent.max())
lr, la = np.log(df.rent), np.log(df.area_sqft)                         # natural log
print("skew of log(rent):", round(lr.skew(), 2), "  skew of log(area):", round(la.skew(), 2))
for a in [400, 800, 1600]: print(f"{a} sqft -> log {np.log(a):.2f}")      # doubling = +0.69 every time

Output:

skew of rent: 2.07   skew of area: 1.65
mean rent: 12703   median rent: 11200.0   max: 88400.0
skew of log(rent): -0.05   skew of log(area): -0.01
400 sqft -> log 5.99
800 sqft -> log 6.68
1600 sqft -> log 7.38

Rent and log(rent) histograms

The log transform replaces a column x with log(x) (here the natural log, np.log). Two reasons it helps:

  1. Times becomes plus. log(a × b) = log(a) + log(b). A "40% dearer" effect (× 1.4) becomes "+ log(1.4)", a fixed amount, which is exactly what linear regression can learn.
  2. Doubling becomes an even step. 400 → 800 → 1,600 sqft becomes 5.99 → 6.68 → 7.38: each doubling adds the same 0.69 (= log 2). The long tail is squeezed, and the column becomes symmetric (skew 2.07 → -0.05).

Real-life example: a Rs 2,000 raise is huge on a Rs 10,000 salary and tiny on Rs 1,00,000. What matters is the percent, and the log turns percents into plain differences.

If you train on log(rent), the model predicts log-rupees. Turn them back with the inverse, the exponential np.exp. Because exp of any number is positive, a negative rent becomes impossible.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent"); ev(F, "raw rent", log=False); ev(F, "log(rent)")
ev(F.assign(area_sqft=np.log(df.area_sqft)), "log(rent) + log(area)")

Output:

raw rent                 median error 13.3%   under 500 sqft 33.4%
log(rent)                median error 11.5%   under 500 sqft 23.3%
log(rent) + log(area)    median error  9.7%   under 500 sqft 11.4%
Columns Median error Flats under 500 sqft
raw rent 13.3% 33.4%
log(rent) 11.5% 23.3%
log(rent) + log(area) 9.7% 11.4%

The log of zero: log1p

A log only works for numbers above zero. 69 flats are brand new (age 0):

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
print("new flats (age 0):", (df.age_years == 0).sum())
age = df.age_years.to_numpy(); print("log(age) smallest:", np.log(age).min())
print("log1p(age) smallest:", np.log1p(age).min())          # log1p(x) = log(1 + x)

Output:

main.py:12: RuntimeWarning: divide by zero encountered in log
  age = df.age_years.to_numpy(); print("log(age) smallest:", np.log(age).min())
new flats (age 0): 69
log(age) smallest: -inf
log1p(age) smallest: 0.0

log(0) is minus infinity, and NumPy warns: RuntimeWarning: divide by zero encountered in log. The usual fix is np.log1p(x) = log(1 + x): 0 stays 0 and large values are still squeezed. Its inverse is np.expm1.

Power transforms: Box-Cox and Yeo-Johnson

Is the log always the right amount of squeezing? A power transform is a whole family of squeezes controlled by one number, λ (lambda):

  • Box-Cox (Box and Cox, 1964): (x^λ − 1) / λ, and for λ = 0 it is defined as log(x). λ = 1 leaves the shape unchanged (it only shifts by 1); λ = 0.5 behaves like a square root (a gentle squeeze); λ = 0 is the log (a strong squeeze). It needs strictly positive data (every value > 0).
  • Yeo-Johnson (Yeo and Johnson, 2000): the same idea extended so it also works for zero and negative values.

scikit-learn's PowerTransformer supports both (method="yeo-johnson" is the default), estimates λ for each column separately by maximum likelihood, and by default standardises the result (standardize=True: mean 0, standard deviation 1).

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
from sklearn.preprocessing import PowerTransformer
for col in ["area_sqft", "rent"]:
    pt = PowerTransformer(method="box-cox").fit(tr[[col]])               # finds the best lambda
    after = pd.Series(pt.transform(tr[[col]])[:, 0]).skew()
    print(f"{col:<9}  skew {tr[col].skew():.2f}   best lambda {pt.lambdas_[0]:.2f}   skew after {after:.2f}")
try: PowerTransformer(method="box-cox").fit(tr[["age_years"]])          # age can be 0
except ValueError as e: print("ValueError:", e)
yj = PowerTransformer(method="yeo-johnson").fit(tr[["age_years"]])  # zero and negatives are fine
print("Yeo-Johnson on age: lambda", yj.lambdas_[0].round(2))

Output:

area_sqft  skew 1.56   best lambda 0.01   skew after 0.00
rent       skew 1.87   best lambda 0.05   skew after 0.00
ValueError: The Box-Cox transformation can only be applied to strictly positive data
Yeo-Johnson on age: lambda 0.73
  • The best λ for area is 0.01 and for rent 0.05: almost exactly 0, so the data itself says the log was the right choice.
  • Box-Cox on age fails with the real ValueError because new flats have age 0.
  • Yeo-Johnson accepts the zeros and picks λ = 0.73, a gentle squeeze, because age was not badly skewed.

Binning (discretisation)

Imran knows tenants are fussy about the station: right next to it there are horns and crowds (rents lower), a 10-20 minute walk (0.5-2 km) is ideal (rents highest), and far away tenants need an auto ride every day (rents fall). A relationship that only ever goes one way (bigger flat → higher rent) is called monotonic. Distance to the station is not monotonic: it goes up, then down.

Median rent per sqft vs distance

A linear model gives distance one slope, so rent can only rise or only fall with distance. It cannot draw this hill.

Binning, also called discretisation, replaces a number with the bin (a range of values) it falls in, like age groups child / adult / senior. "Discrete" means separate steps instead of a smooth scale. After one-hot encoding the bins, each bin gets its own weight, so the weights can go up and then down.

scikit-learn's KBinsDiscretizer(n_bins=5, encode="onehot", strategy="quantile") (defaults shown):

Parameter Choices
n_bins how many bins (at least 2)
strategy "uniform": equal widths · "quantile" (default): about the same number of rows per bin · "kmeans": bins around 1-D k-means centres
encode "onehot" (default, sparse) · "onehot-dense" · "ordinal" (the bin number)

Or choose the edges yourself with pd.cut(x, edges), when you know where they belong.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
from sklearn.preprocessing import KBinsDiscretizer; F = df.drop(columns="rent")
F = F.assign(area_sqft=np.log(df.area_sqft)); ev(F, "log features (so far)")      # the best from the log step
kb = KBinsDiscretizer(n_bins=8, encode="ordinal", strategy="uniform"); kb.fit(tr[["km_to_station"]])
ev(F.assign(km_bin=kb.transform(df[["km_to_station"]])[:, 0].astype(str)), "+ 8 equal-width bins")
ev(F.assign(km_band=pd.cut(df.km_to_station, [0, .5, 2, 4, 8]).astype(str)), "+ Imran's 4 bands")

Output:

log features (so far)    median error  9.7%   under 500 sqft 11.4%
+ 8 equal-width bins     median error  8.7%   under 500 sqft  9.9%
+ Imran's 4 bands        median error  7.3%   under 500 sqft  8.3%

Eight equal-width bins (each about 1 km wide) help: 9.7% → 8.7%. Imran's four bands, chosen from experience, do better with fewer bins: 7.3%. The equal-width edges cut right through the best stretch (0.5-2 km); Imran's edges sit exactly where tenants change their minds. Bin edges that need fitting (KBinsDiscretizer) are fitted on the training rows only.

Ratio, difference and interaction features

Real-life example: 45 marks means little until you know "out of 50" or "out of 100". The 5th floor is the top of a 5-storey building but low in a 20-storey tower.

  • Ratio feature: one column divided by another. height = floor / total_floors is 0 at the ground and 1 at the top.
  • Difference feature: one column minus another. floors_above = total_floors − floor.
  • Interaction feature: when one feature's effect depends on another, their product. Carrying a gas cylinder up to the 4th floor only hurts when there is no lift: climb = floor × (1 − lift), "floors to climb".
import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft), km_band=pd.cut(d, [0, .5, 2, 4, 8]).astype(str))
ev(F, "log + bands (so far)")
ev(F.assign(floors_above=df.total_floors - df.floor), "+ difference")         # floors above you
ev(F.assign(height=df.floor / df.total_floors), "+ ratio")                    # 0 = ground, 1 = top
ev(F.assign(height=df.floor / df.total_floors, climb=df.floor * (1 - df.lift)), "+ ratio + climb")

Output:

log + bands (so far)     median error  7.3%   under 500 sqft  8.3%
+ difference             median error  7.3%   under 500 sqft  8.3%
+ ratio                  median error  7.1%   under 500 sqft  8.7%
+ ratio + climb          median error  6.7%   under 500 sqft  8.8%
  • The difference changes nothing (7.3% → 7.3%). A linear model already has a weight for total_floors and one for floor, so it can compute total_floors − floor itself. A new column that is a sum, difference or multiple of existing columns adds nothing to a linear model. (It can still help a tree, which cannot add columns.)
  • The ratio helps (7.1%): a linear model cannot divide.
  • Floors to climb helps most (6.7%). Small flats move slightly the other way (8.3% → 8.8%), so always look at more than one number.

Mistake: dividing by something that can be zero

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft), km_band=pd.cut(d, [0, .5, 2, 4, 8]).astype(str))
F["per_floor"] = df.total_floors / df.floor          # "how many times higher is the building?"
print("smallest:", F.per_floor.min(), "  largest:", F.per_floor.max())
try: ev(F, "+ total / floor")
except ValueError as e: print("ValueError:", e)

Output:

smallest: 1.0   largest: inf
ValueError: Input X contains infinity or a value too large for dtype('float64').

Ground-floor flats have floor = 0, so total_floors / floor is inf (infinity), and scikit-learn refuses it. Before dividing, ask: can the bottom ever be zero? Divide by something that can't be (here total_floors ≥ 2), or add 1 to the bottom.

Aggregations

Brokers compare localities in rupees per square foot, like comparing your marks with the class average. An aggregation is one number that summarises many rows (mean, median, count, max...) of a group; as a feature, it is mapped back onto every row of that group.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft), km_band=pd.cut(d, [0, .5, 2, 4, 8]).astype(str))
F = F.assign(height=df.floor / df.total_floors, climb=df.floor * (1 - df.lift))
rate = (tr.rent / tr.area_sqft).groupby(tr.locality).median()        # aggregation: training flats only
print(rate.round(1).to_string())
ev(F.drop(columns="locality").assign(loc_rate=np.log(df.locality.map(rate))), "1 aggregate, no one-hot")

Output:

locality
Aundh       18.2
Baner       16.8
Hadapsar    11.5
Kothrud     17.3
Pashan      12.2
Wakad       13.6
1 aggregate, no one-hot  median error  6.9%   under 500 sqft  8.7%

One aggregate column gives 6.9%, almost as good as six one-hot columns (6.7%). With six localities that is a draw; with 600 localities, one column instead of 600 is a big win.

Warning: this aggregate uses rent, the thing we predict. Compute it from training rows only; otherwise the test answers leak into the features (target leakage, Episode 15), and there is no error message to warn you. Episode 39 treats target-based aggregates properly as target encoding, with smoothing.

Domain-driven features, and what trees think of all this

A domain is the field a problem lives in: renting, banking, farming, cricket. A domain-driven feature is built from what experts in that field already know. Imran's walking-distance bands, floors to climb and rupees per square foot are all domain-driven. The cheapest way to find them: ask an expert what they look at.

A decision tree split asks "is area ≤ 700?". The log keeps the order of the values (it is monotonic), so exactly the same flats fall on each side. Trees should not care:

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
from sklearn.tree import DecisionTreeRegressor
X = pd.get_dummies(df.drop(columns="rent"), dtype=float)
for name, Z in [("raw area", X), ("log(area)", X.assign(area_sqft=np.log(X.area_sqft)))]:
    t = DecisionTreeRegressor(max_depth=8, random_state=0).fit(Z[:2400], tr.rent)
    e = abs(t.predict(Z[2400:]) - te.rent) / te.rent * 100
    print(f"tree, {name:<10} median error {e.median():.1f}%   splits on area: {(t.tree_.feature == 0).sum()}")

Output:

tree, raw area   median error 14.4%   splits on area: 47
tree, log(area)  median error 14.4%   splits on area: 47

Identical: 14.4% and 47 splits on area both times. Log and power transforms matter for models that add things up or measure distances (linear and logistic regression, k-nearest neighbours, SVMs), not for trees. Ratios, interactions and aggregates are genuinely new information, so trees can benefit from those too.

Error after each step

Step Median error
raw columns 13.3%
log(rent) 11.5%
+ log(area) 9.7%
+ 8 equal-width distance bins 8.7%
Imran's 4 distance bands instead 7.3%
+ ratio (floor / total floors) 7.1%
+ interaction (floors to climb) 6.7%

Common mistakes

Mistake What you see Why Fix
Skewed target in a linear model lowest prediction: Rs -2315 Effects are in percent; big values dominate Train on log(rent), exp back
np.log on a column with zeros RuntimeWarning: divide by zero encountered in log and -inf log(0) is undefined np.log1p
Box-Cox on data with 0 or negatives ValueError: The Box-Cox transformation can only be applied to strictly positive data Box-Cox needs x > 0 method="yeo-johnson"
Dividing by a column that can be 0 ValueError: Input X contains infinity or a value too large for dtype('float64'). Ground floor = 0 Divide by total_floors, or + 1
Forgetting to exp the predictions errors near 100% comparing log-rupees with rupees np.exp(p) or TransformedTargetRegressor
Aggregating the target over all rows no error, an optimistic test score test answers leak in Fit aggregates on training rows only
Adding a difference of existing columns to a linear model no change (7.3% → 7.3%) the model can already build it Add ratios / products instead

Mini project: the whole recipe as a leakage-safe pipeline

TransformedTargetRegressor trains on log(rent) and applies exp to the predictions for you; FunctionTransformer wraps np.log as a pipeline step; the bands use fixed edges, so there is nothing to fit (and nothing to leak). Cross-validation (Episode 26) then checks the result on five different splits.

import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
from sklearn.compose import ColumnTransformer, TransformedTargetRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.model_selection import cross_val_score
df["height"] = df.floor / df.total_floors; df["climb"] = df.floor * (1 - df.lift)    # ratio + interaction
df["km_band"] = pd.cut(df.km_to_station, [0, .5, 2, 4, 8]).astype(str)             # fixed edges: nothing to fit
X = df.drop(columns="rent")
prep = ColumnTransformer([
    ("log", FunctionTransformer(np.log), ["area_sqft"]),                               # log transform
    ("onehot", OneHotEncoder(), ["locality", "km_band"]),
], remainder="passthrough")                                                            # other numbers as they are
model = TransformedTargetRegressor(regressor=make_pipeline(prep, LinearRegression()), func=np.log, inverse_func=np.exp)
model.fit(X[:2400], df.rent[:2400]); p = model.predict(X[2400:])
print(f"test median % error: {(abs(p - te.rent) / te.rent * 100).median():.1f}")
cv = cross_val_score(model, X, df.rent, cv=5, scoring="neg_mean_absolute_percentage_error")
print(f"5-fold CV mean absolute % error: {-cv.mean() * 100:.1f}  (spread {cv.std() * 100:.1f})")

Output:

test median % error: 6.7
5-fold CV mean absolute % error: 8.2  (spread 0.3)

The pipeline reproduces the video's 6.7% on the test flats. The cross-validation number is a mean (not a median) absolute percentage error, so it is higher: a few badly predicted flats pull a mean up.

Practice

1. Try n_bins 4, 8 and 16 with strategy="uniform" and "quantile". Do more bins always help? Do they beat Imran's 4 bands?

Solution
import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
from sklearn.preprocessing import KBinsDiscretizer
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft))
for k in [4, 8, 16]:
    for strat in ["uniform", "quantile"]:
        kb = KBinsDiscretizer(n_bins=k, encode="ordinal", strategy=strat).fit(tr[["km_to_station"]])
        ev(F.assign(km_bin=kb.transform(df[["km_to_station"]])[:, 0].astype(str)), f"{k} bins, {strat}")

Output:

4 bins, uniform          median error  8.8%   under 500 sqft 10.2%
4 bins, quantile         median error  8.5%   under 500 sqft 10.1%
8 bins, uniform          median error  8.7%   under 500 sqft  9.9%
8 bins, quantile         median error  8.4%   under 500 sqft  9.9%
16 bins, uniform         median error  7.7%   under 500 sqft  8.8%
16 bins, quantile        median error  7.6%   under 500 sqft  8.9%

More bins help here (16 bins: 7.6-7.7%), and quantile bins are slightly better than equal widths, but 4 domain bands (7.3%) still beat 16 automatic bins. Many bins also mean many weights to learn from few rows each.

2. Add age bands: new (0), 1-10, 11-25, 26+ years. Does it help?

Solution
import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft), km_band=pd.cut(d, [0, .5, 2, 4, 8]).astype(str))
F = F.assign(height=df.floor / df.total_floors, climb=df.floor * (1 - df.lift)); ev(F, "best from the video")
ev(F.assign(age_band=pd.cut(df.age_years, [-1, 0, 10, 25, 40]).astype(str)), "+ age bands")

Output:

best from the video      median error  6.7%   under 500 sqft  8.8%
+ age bands              median error  6.8%   under 500 sqft  8.8%

No (6.7% → 6.8%). In this made-up data age lowers rent steadily (a monotonic effect), so the plain age_years column already captures it. Bins are for effects that change direction or jump.

3. Fit Box-Cox on km_to_station. What λ does it pick, and why?

Solution
import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
from sklearn.preprocessing import PowerTransformer
pt = PowerTransformer(method="box-cox").fit(tr[["km_to_station"]])
print("skew before", round(tr.km_to_station.skew(), 2), " lambda", pt.lambdas_[0].round(2))

Output:

skew before 0.04  lambda 0.7

Skew is already about 0 (distances are spread evenly), so it only needs a gentle squeeze, λ ≈ 0.7. A power transform fixes skew; it cannot fix the up-then-down shape. That needs bins.

4. Aggregate: the median age of flats in each locality (training rows only). Does it help?

Solution
import numpy as np, pandas as pd; from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(38); n = 3000; U, I = rng.uniform, rng.integers
locs = ["Kothrud", "Wakad", "Hadapsar", "Baner", "Aundh", "Pashan"]
df = pd.DataFrame({"area_sqft": rng.lognormal(6.6, .45, n).round(), "total_floors": I(2, 25, n),
    "km_to_station": U(.1, 8, n).round(2), "age_years": I(0, 40, n), "locality": rng.choice(locs, n)})
df["floor"] = (U(0, 1, n) * (df.total_floors + 1)).astype(int); df["lift"] = (df.total_floors >= 5).astype(int)
d, f = df.km_to_station, df.floor; boost = df.locality.map(dict(zip(locs, [.35, .1, -.1, .3, .4, 0])))
logr = (2.6 + np.log(df.area_sqft) + boost - .15*(d < .5) + .25*d.between(.5, 2) - .03*d - .1*f*(1 - df.lift)
        + .3*f/df.total_floors - .006*df.age_years + rng.normal(0, .1, n))             # hidden made-up rule
df["rent"] = (np.exp(logr) / 100).round() * 100; tr, te = df[:2400], df[2400:]   # Imran's flats, made up
def ev(F, name, log=True):                   # linear regression on F, % error on the 600 test flats
    X = pd.get_dummies(F, dtype=float); m = LinearRegression().fit(X[:2400], np.log(tr.rent) if log else tr.rent)
    p = m.predict(X[2400:]); p = np.exp(p) if log else p; e = abs(p - te.rent) / te.rent * 100   # exp: back to Rs
    print(f"{name:<24} median error {e.median():4.1f}%   under 500 sqft {e[te.area_sqft <= 500].mean():4.1f}%")
F = df.drop(columns="rent").assign(area_sqft=np.log(df.area_sqft), km_band=pd.cut(d, [0, .5, 2, 4, 8]).astype(str))
F = F.assign(height=df.floor / df.total_floors, climb=df.floor * (1 - df.lift))
med_age = tr.groupby("locality").age_years.median(); print(med_age.to_string())
ev(F.assign(loc_med_age=df.locality.map(med_age)), "+ median age per locality")

Output:

locality
Aundh       21.0
Baner       19.0
Hadapsar    19.0
Kothrud     22.0
Pashan      19.0
Wakad       19.0
+ median age per locality median error  6.7%   under 500 sqft  8.8%

No change (6.7%). Ages are spread the same way in every made-up locality, so this aggregate carries no new information. Aggregates help when the group summary really matters to the target (like rupees per square foot).

5. By hand: a model trained on log(rent) predicts 9.21 for a flat. What rent is that? And what is log(2)?

Solution

exp(9.21) ≈ 10,000 rupees. log(2) ≈ 0.69, the step for every doubling.

Quick quiz

  1. A linear model already uses floor and total_floors. Which new column can improve it: total_floors − floor, floor + total_floors, floor / total_floors or 2 × floor? The ratio, floor / total_floors; the others it can build itself.
  2. Box-Cox picks λ ≈ 0. Which transform is that? The log.
  3. Which transform works with zeros and negative values? Yeo-Johnson.
  4. Is distance to the station a monotonic feature for rent? No: rent rises, then falls.
  5. Why don't decision trees care about a log transform? It keeps the order of values, so every split separates the same rows.
  6. What must an aggregate of the target be computed from? Training rows only.

Interview questions

  1. What is feature engineering, and why can it beat a better model? Creating or reshaping input columns so the pattern is easy to learn; a simple model with the right features (here 6.7%) can beat a complex model on raw ones.
  2. When do you log-transform a feature or target? Right-skewed, positive data with multiplicative (percent) effects, e.g. prices, incomes, counts; use log1p when zeros appear; predict with exp.
  3. Box-Cox vs Yeo-Johnson? Both are power transforms with λ chosen by maximum likelihood; Box-Cox needs strictly positive data, Yeo-Johnson also handles zero and negatives.
  4. Why bin a numerical feature? To let a linear model capture a non-monotonic or step-shaped effect; at the cost of losing detail inside each bin.
  5. Uniform vs quantile binning? Equal widths vs equal counts per bin; quantile avoids nearly empty bins on skewed data.
  6. Why does a difference feature not help linear regression? It is a linear combination of existing columns, which the model can already represent; ratios and products are not.
  7. What is an interaction feature? Give an example. A product of features when one's effect depends on another: floor × (no lift).
  8. What is the leakage risk of aggregations? Aggregates that use the target must be computed on training data only (inside each CV fold); otherwise the score is optimistic.
  9. Do tree models need log transforms or scaling? No, splits depend only on value order; but they can benefit from ratios, interactions and aggregates.
  10. How do you find good domain-driven features? Ask domain experts what they look at, look at plots of the target against each feature (like the distance hill), and validate each new feature with cross-validation.

Glossary

Term Meaning (plain words) Example from this episode
Feature engineering Creating or reshaping input columns so the pattern is easy to see date of birth → age; floor × no lift
Multiplicative effect An effect that changes the target by a percent, not a fixed amount a good locality: about 40% dearer
Skew (skewness) How lopsided a set of values is; 0 = symmetric, positive = long right tail rent skew 2.07
Long tail A few values far bigger than the rest rents up to Rs 88,400
Natural log (np.log) The inverse of exp; turns × into + log(800) = 6.68
Log transform Replacing x with log(x) log(rent), log(area)
Exponential (np.exp) The inverse of log; turns predictions back into rupees exp(9.21) ≈ 10,000
log1p log(1 + x): a log that works for 0 log1p(0) = 0
Power transform A family of squeezes x → f(x, λ) Box-Cox, Yeo-Johnson
λ (lambda) The power that sets how strong the squeeze is 0.01 for area
Box-Cox transform (x^λ − 1)/λ, log when λ = 0; needs x > 0 (1964) fails on age 0
Yeo-Johnson transform A power transform that also handles 0 and negatives (2000) λ 0.73 on age
PowerTransformer scikit-learn's power transforms, λ by maximum likelihood method="box-cox"
Standardise Shift and scale to mean 0, standard deviation 1 PowerTransformer's default
Monotonic Always moving one way as the input grows bigger flat → higher rent
Non-monotonic Going up and then down (or the reverse) rent vs distance to the station
Bin A range of values grouped together 0.5-2 km
Binning (discretisation) Replacing a number with the bin it falls in km → 4 bands
KBinsDiscretizer scikit-learn's binning (n_bins, strategy, encode) 8 uniform bins
Uniform / quantile bins Equal widths / equal numbers of rows edges every ~1 km
pd.cut pandas binning with edges you choose [0, .5, 2, 4, 8]
Ratio feature One column divided by another floor / total floors
Difference feature One column minus another total floors − floor
Interaction feature A product of features whose effects depend on each other floor × (1 − lift)
Aggregation One number summarising a group (mean, median, count, max) median Rs/sqft per locality
groupby pandas: split rows into groups, then summarise each tr.groupby("locality")
Target leakage Features that secretly contain the answer aggregating rent over test rows
Domain The field a problem lives in renting
Domain-driven feature A feature from experts' knowledge of the field walking-distance bands
TransformedTargetRegressor Trains on a transformed target and inverts predictions log/exp of rent
FunctionTransformer Wraps any function as a pipeline step np.log on area

Summary

  • Feature engineering changes the columns, not the model: the same linear regression went from 13.3% to 6.7% median error, and stopped predicting negative rents.
  • Skewed, percent-style columns need a log (log1p with zeros; exp to go back). Box-Cox (x > 0) and Yeo-Johnson (any x) pick the squeeze λ by maximum likelihood.
  • Binning lets a linear model follow non-monotonic effects; domain edges beat automatic ones here.
  • Ratios and interactions add what a linear model can't compute; differences of existing columns add nothing to it.
  • Aggregations summarise a group; if they use the target, compute them on training rows only.
  • Trees ignore monotonic transforms like the log.
  • Next episode: feature engineering for categories, dates and text: high-cardinality categories, target encoding, the hashing trick, date-time and cyclical features, and TF-IDF.