Machine Learning Full Course · Episode 39. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Kiran's 30,000 delivery orders and the 6,000 reviews are made up for teaching.
The big idea: models only eat numbers, but real tables are full of names (restaurants, customers), timestamps and free text. A restaurant column with thousands of values is a high-cardinality category: one-hot encoding makes thousands of mostly-zero columns and breaks on tomorrow's new restaurants. Frequency encoding replaces a name with its count; target encoding replaces it with the average target of that name (its late rate). Done naively, target encoding leaks each row's own answer into its feature (train 0.916, test 0.628 in this episode); cross fitting (scikit-learn's TargetEncoder.fit_transform) and smoothing fix it. When names never stop coming, the hashing trick maps any name to one of a fixed number of columns. Timestamps become date-time features (hour, weekday, weekend), and cyclical ones go on a circle with sine and cosine, so 23:00 and 00:00 are neighbours. Text becomes numbers with TF-IDF, which weights a word by how often it appears in this review and how rare it is overall; n-grams catch phrases like "not fresh".
Sources for the product facts in this episode: scikit-learn's TargetEncoder documentation, FeatureHasher documentation and TfidfTransformer documentation (the formulas used by TfidfVectorizer), and the feature hashing paper by Weinberger, Dasgupta, Attenberg, Langford and Smola (arXiv 0902.2206, 2009).
Words you need first
- Feature, target (label): an input column, and the thing we predict (Episode 1). Here the target is
late(1 = delivered late). - Categorical feature: a column whose values are names from a list (Episode 13).
- One-hot encoding,
handle_unknown: one 0/1 column per category (Episode 16). - Train / test split, target leakage: Episode 15. Cross-validation, folds: Episode 26.
- Gradient boosting (
HistGradientBoostingClassifier): Episodes 35-36. Logistic regression: Episodes 22-23. - k-nearest neighbours (kNN): classify by a vote of the most similar training rows (Episode 27).
- Bag of words,
CountVectorizer: one count column per word (Episode 28). - Pipeline: preprocessing and model chained into one object (Episode 17). L2 norm: a vector's length (Episode 3).
The problem: thousands of categories
Kiran runs a food-delivery app. Each of 30,000 made-up orders has a restaurant (2,823 different ones), a customer (12,980), a distance and an order time. Question: will this order be late? The first 22,000 orders are for training, the last 8,000 for testing (they stand for "tomorrow").
The number of different values in a column is its cardinality. A plan column with veg, nonveg and jain has cardinality 3; the restaurant column has cardinality 2,823. A column with hundreds or thousands of values is a high-cardinality category.
The obvious attempt is one-hot encoding:
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.preprocessing import OneHotEncoder
print(df.head(3)); c = df.restaurant[tr].value_counts()
print("restaurants:", df.restaurant.nunique(), " customers:", df.customer.nunique())
print("median orders per restaurant:", int(c.median()))
oh = OneHotEncoder().fit(df[["restaurant"]][tr]) # one 0/1 column per restaurant
print("one-hot columns:", oh.transform(df[["restaurant"]][tr]).shape[1])
try: oh.transform(df[["restaurant"]][te]) # tomorrow's orders
except ValueError as e: print("ValueError:", str(e)[:35], "...", str(e)[-30:])
Output (the list of unknown names differs from run to run, because Python lists a set in a different order each time):
restaurant customer distance_km ordered_at
0 R2508 C04879 5.552547 2026-03-22 02:06:00
1 R2228 C06195 2.742027 2026-01-13 19:06:00
2 R1683 C06265 7.705933 2026-03-01 10:14:00
restaurants: 2823 customers: 12980
median orders per restaurant: 3
one-hot columns: 2667
ValueError: Found unknown categories ['R1581', ... ] in column 0 during transform
Why one-hot struggles here:
- 2,667 columns, one per restaurant seen in training, almost all zeros.
- A typical restaurant has only 3 orders: too little to learn a weight per column.
- New restaurants arrive every day and have no column:
ValueError: Found unknown categories [...] in column 0 during transform. Withhandle_unknown="ignore"there is no error, but the model knows nothing about them.
The goal: replace each restaurant with one number that means something about lateness.
Label codes and frequency encoding
- Label codes: use the number in the name (
R0370→ 370). The order is fake: restaurant 370 is not "more" than 369. - Frequency encoding: replace each category with how many training rows have it. New restaurant → 0. It works when "how common" matters (busy kitchens are late more often in Kiran's data).
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
codes = df.restaurant.str[1:].astype(int) # 'R0370' -> 370
counts = df.restaurant[tr].value_counts() # orders per restaurant, from TRAIN rows only
freq = df.restaurant.map(counts).fillna(0) # frequency encoding (new restaurant -> 0)
for name, col in [("label codes", codes), ("frequency", freq)]:
X = pd.DataFrame({"restaurant": col, "distance_km": df.distance_km})
m = HistGradientBoostingClassifier(random_state=0).fit(X[tr], y[tr])
print(f"{name:<12} train {m.score(X[tr], y[tr]):.3f} test {m.score(X[te], y[te]):.3f}")
Output:
label codes train 0.724 test 0.697
frequency train 0.734 test 0.715
Frequency beats the fake codes, but two restaurants with the same count look identical.
Target encoding
Target encoding replaces each category with the average target of that category in the training data. For a 0/1 target that is the category's rate (here: its late rate); for a number target (delivery minutes) it is the category's average. New categories get the global average.
Example: restaurant R0082 had 29 training orders, 22 of them late → 22 / 29 = 0.759.
Real-life picture: before ordering, you ask a friend "how often has this place been late before?", its track record.
Naive version with pandas groupby:
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
def naive_target(col): # late rate per category, ALL train rows
rate = y[tr].groupby(df[col][tr]).mean(); return df[col].map(rate).fillna(y[tr].mean())
for cols in [["restaurant"], ["restaurant", "customer"]]:
X = pd.DataFrame({c: naive_target(c) for c in cols}).assign(distance_km=df.distance_km)
m = HistGradientBoostingClassifier(random_state=0).fit(X[tr], y[tr])
print(f"naive target {'+'.join(cols):<20} train {m.score(X[tr], y[tr]):.3f} test {m.score(X[te], y[te]):.3f}")
Output:
naive target restaurant train 0.810 test 0.752
naive target restaurant+customer train 0.916 test 0.628
Restaurant alone: test 0.752, the best so far. But adding the customer makes train 0.916 and test 0.628, worse than the fake label codes.
Target-encoding leakage and cross fitting
Why it failed. Most customers have one or two orders; 5,040 have exactly one training order. Customer C08040 has one order, and it was late, so its encoding is 1.0, made from the very answer we are predicting. The answer has leaked into the feature: target-encoding leakage, a kind of target leakage. It's like an exam paper with the answers printed beside the questions: full marks, nothing learned. On test orders there are no printed answers.
The fix: cross fitting.
- Split the training rows into 5 folds (equal parts).
- Encode the rows of fold 1 using late rates computed only from folds 2-5; repeat for every fold. No row's encoding uses its own label.
- Encode test rows with the table learned from all training rows.
scikit-learn's TargetEncoder (added in version 1.3) does this in fit_transform, with cv=5 folds by default. Its documentation states that fit(X, y).transform(X) does not equal fit_transform(X, y), because only fit_transform uses cross fitting, and recommends fit_transform for training data. Unseen categories are encoded with the global target mean (target_mean_). We pass a fixed StratifiedKFold as cv so the numbers are the same on every run (in scikit-learn 1.9, the shuffle and random_state parameters are deprecated in favour of passing a splitter as cv).
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import TargetEncoder; from sklearn.model_selection import StratifiedKFold
cols = ["restaurant", "customer"]; enc = TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0))
Xtr = pd.DataFrame(enc.fit_transform(df[cols][tr], y[tr]), columns=cols).assign(km=df.distance_km[tr].values)
Xte = pd.DataFrame(enc.transform(df[cols][te]), columns=cols).assign(km=df.distance_km[te].values)
m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, y[tr])
print(f"TargetEncoder restaurant+customer train {m.score(Xtr, y[tr]):.3f} test {m.score(Xte, y[te]):.3f}")
Output:
TargetEncoder restaurant+customer train 0.774 test 0.756
Train and test now agree (0.774 vs 0.756), and test is the best so far.

Smoothing
Tiny groups give wild rates: one late order → 100% late. Smoothing mixes the group's own rate with the global rate (the late rate over all training orders):
encoded = (n · rate + m · global) / (n + m)
n = rows in this category, m = the smoothing strength (smooth in scikit-learn), like m imaginary rows at the global rate. This is the formula scikit-learn uses when smooth is a number. With one late order, global rate 0.434 and m = 10: (1 + 4.34) / 11 = 0.486. A restaurant with 1,697 orders barely moves. The default smooth="auto" picks the amount from the data with an empirical Bayes estimate; the documentation says a larger smooth puts more weight on the global mean.
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.preprocessing import TargetEncoder
R = df[["restaurant"]][tr]; table = pd.DataFrame({"orders": R.restaurant.value_counts(),
"late_rate": y[tr].groupby(R.restaurant).mean()})
for s in ["auto", 10]: # fit only to LOOK at the encoding table
enc = TargetEncoder(smooth=s).fit(R, y[tr])
table[f"smooth={s}"] = pd.Series(enc.encodings_[0], index=enc.categories_[0])
print("global late rate:", round(enc.target_mean_, 3))
print(table.loc[["R0801", "R0082", "R0011", "R0004"]].round(3))
Output:
global late rate: 0.434
orders late_rate smooth=auto smooth=10
restaurant
R0801 1697 0.608 0.607 0.607
R0082 29 0.759 0.750 0.675
R0011 2 0.000 0.000 0.362
R0004 1 1.000 1.000 0.486
With auto, the one-order and two-order restaurants keep their raw rates in this run: a single order has no spread inside its group for the estimate to measure. With smooth=10 they are pulled towards 0.434. When many groups are tiny, try a few numbers for smooth and compare them with cross-validation (Practice 2). Note that leakage is fixed by cross fitting; smoothing fixes noisy small groups. You usually want both.
The hashing trick
When names never stop coming (millions of restaurants, dishes, users), even storing a name → column table is costly. Real-life picture: a post office with a fixed wall of 1,000 pigeonholes, where a letter goes to the box given by the last three digits of its pincode. No lookup list, just a rule; sometimes two towns share a box.
- A hash function turns any text into a number, the same number every time.
- The hashing trick (feature hashing) uses
hash(name) mod n_featuresas the column. Described in the 2009 paper Feature Hashing for Large Scale Multitask Learning (Weinberger et al.). - A collision is two names landing in the same column.
- scikit-learn's
FeatureHasheruses the signed 32-bit MurmurHash3; defaultn_features = 2**20(1,048,576);alternate_sign=Truegives some names −1 instead of +1 so collisions tend to cancel out. It stores no vocabulary, so a brand-new name just works. The result is a sparse matrix: a table that stores only its non-zero entries.
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.feature_extraction import FeatureHasher; from sklearn.linear_model import LogisticRegression
from scipy.sparse import hstack; hasher = lambda k: FeatureHasher(n_features=k, input_type="string")
for r in ["R0001", "R0002", "R1234", "R9999"]: # R9999 opens tomorrow: no problem
v = hasher(8).transform([[r]]).toarray()[0]; print(r, "-> column", v.nonzero()[0][0], " value", int(v.sum()))
for k in [64, 1024, 16384]: # hash 'restaurant' into k columns
X = hstack([hasher(k).transform(df[["restaurant"]].values), df[["distance_km"]].values]).tocsr()
m = LogisticRegression(max_iter=1000).fit(X[:22000], y[tr])
print(f"{k:>6} columns: test {m.score(X[22000:], y[te]):.3f}")
Output:
R0001 -> column 3 value 1
R0002 -> column 7 value 1
R1234 -> column 0 value -1
R9999 -> column 7 value 1
64 columns: test 0.677
1024 columns: test 0.730
16384 columns: test 0.760
R9999 (a restaurant that doesn't exist yet) gets a column with no error, and collides with R0002 in column 7. With 64 columns there are too many collisions (0.677); 16,384 columns give 0.760. Rule of thumb: many more columns than distinct names.
Date-time features
A timestamp like 2026-03-22 02:06:00 is text to a model:
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.linear_model import LogisticRegression
try: LogisticRegression().fit(df[["distance_km", "ordered_at"]][tr], y[tr]) # the timestamp, as it is
except ValueError as e: print("ValueError:", e)
ts = pd.to_datetime(df.ordered_at) # text -> real date-time
F = pd.DataFrame({"hour": ts.dt.hour, "weekday": ts.dt.dayofweek, "weekend": (ts.dt.dayofweek >= 5).astype(int),
"month": ts.dt.month, "distance_km": df.distance_km})
print(pd.concat([df.ordered_at, F.drop(columns="distance_km")], axis=1).head(4))
print("late rate by weekend:", y.groupby(F.weekend).mean().round(3).to_dict())
Output:
ValueError: could not convert string to float: '2026-03-22 02:06:00'
ordered_at hour weekday weekend month
0 2026-03-22 02:06:00 2 6 1 3
1 2026-01-13 19:06:00 19 1 0 1
2 2026-03-01 10:14:00 10 6 1 3
3 2026-01-26 21:27:00 21 0 0 1
late rate by weekend: {0: 0.406, 1: 0.504}
pd.to_datetime turns text into real date-times; the .dt accessor gives date-time features: hour, dayofweek (Monday = 0, Sunday = 6), a weekend flag, month, and also day, dayofyear, minutes since midnight, "days until a festival", and so on. Weekend orders were late 50.4% of the time vs 40.6% on weekdays.
Cyclical encoding with sine and cosine
As a plain number, 23:00 is 23 and 00:00 is 0: 23 apart, though they are one hour apart in real life (same for December and January, Sunday and Monday). Cyclical encoding puts the value on a circle:
angle = 2π · hour / 24
hour_sin = sin(angle) hour_cos = cos(angle)
Sine and cosine are simply the across and up positions of a point going round a circle of radius 1. Use both: sine alone gives the same value (0) for 0:00 and 12:00, so you need the pair to tell every hour apart.

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.linear_model import LogisticRegression
ts = pd.to_datetime(df.ordered_at); hour = ts.dt.hour + ts.dt.minute / 60 # 21:30 -> 21.5
sin, cos = lambda h: np.sin(2 * np.pi * h / 24), lambda h: np.cos(2 * np.pi * h / 24) # one lap = 24 hours
for h in [23, 0, 12]: print(f"hour {h:>2}: sin {sin(h):+.3f} cos {cos(h):+.3f}")
print("23:00 vs 00:00 raw gap: 23 circle gap:", round(np.hypot(sin(23) - sin(0), cos(23) - cos(0)), 3))
base = pd.DataFrame({"weekend": (ts.dt.dayofweek >= 5).astype(int), "distance_km": df.distance_km})
for name, X in [("raw hour", base.assign(hour=hour)), ("sin + cos", base.assign(sin=sin(hour), cos=cos(hour)))]:
m = LogisticRegression(max_iter=1000).fit(X[tr], y[tr]); print(f"{name:<10} test {m.score(X[te], y[te]):.3f}")
Output:
hour 23: sin -0.259 cos +0.966
hour 0: sin +0.000 cos +1.000
hour 12: sin +0.000 cos -1.000
23:00 vs 00:00 raw gap: 23 circle gap: 0.261
raw hour test 0.659
sin + cos test 0.714
The circle gap between 23:00 and 00:00 is 0.261 instead of 23. Logistic regression goes from 0.659 to 0.714: lateness peaks around dinner and stays high after midnight, a shape a straight line in the raw hour cannot follow. Tree models cope better with raw hours, but the circle rarely hurts.
Everything together
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier as HGB; from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import StratifiedKFold; cols = ["restaurant", "customer"]
ts = pd.to_datetime(df.ordered_at); a = 2 * np.pi * (ts.dt.hour + ts.dt.minute / 60) / 24 # hour -> angle
X = pd.DataFrame({"hour_sin": np.sin(a), "hour_cos": np.cos(a), "weekend": (ts.dt.dayofweek >= 5).astype(int),
"distance_km": df.distance_km, "restaurant": 0.0, "customer": 0.0})
enc = TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0)) # same folds every run
X.iloc[tr, -2:] = enc.fit_transform(df[cols][tr], y[tr]); X.iloc[te, -2:] = enc.transform(df[cols][te])
m = HGB(random_state=0).fit(X[tr], y[tr]); print("all together: test", round(m.score(X[te], y[te]), 3))
Output:
all together: test 0.858
From 0.697 (label codes) to 0.858: same raw data, better features.
TF-IDF features for text
Kiran also wants to flag complaints in written reviews (6,000 made-up reviews, each labelled 1 = complaint or 0).
- A token is one unit of text, usually a word.
CountVectorizer/TfidfVectorizerby default lowercase the text and use words of 2 or more letters or digits (token_pattern=r"(?u)\b\w\w+\b"). - The vocabulary is the list of every token seen when fitting.
- Bag of words (Episode 28): one count column per token. Problem: "the" counts as much as "cold".
Real-life picture: in a family group chat, "good morning" appears every day and tells you nothing; "flood" appears once in months and you read that message.
TF-IDF = term frequency × inverse document frequency. A document is one piece of text (one review).
tf(t, d) = count of word t in document d
idf(t) = ln((1 + n) / (1 + df(t))) + 1 # scikit-learn default, smooth_idf=True
tf-idf(t, d) = tf(t, d) × idf(t) # then each row scaled to L2 length 1 (norm='l2')
n = number of documents, df(t) (document frequency) = number of documents containing t. A word in every document gets the minimum idf, 1. Scaling each row to length 1 means long reviews don't get bigger numbers just for being long. TfidfVectorizer = CountVectorizer followed by this weighting, in one object.
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
reviews = ["the food was cold and late", # 4 made-up reviews of Kiran's app
"the food was hot and tasty",
"late again, the dal spilled",
"tasty biryani, the rider was polite"]
cv, tf = CountVectorizer(), TfidfVectorizer() # tokens: words of 2+ letters, lowercased
counts, weights = cv.fit_transform(reviews), tf.fit_transform(reviews) # sparse matrices
print("vocabulary:", len(tf.vocabulary_), "words ->", sorted(tf.vocabulary_)[:8], "...")
row = pd.DataFrame({"count": counts[0].toarray()[0], "tf-idf": weights[0].toarray()[0].round(2),
"idf": tf.idf_.round(2)}, index=cv.get_feature_names_out())
print("review 0:", reviews[0]); print(row[row["count"] > 0].sort_values("tf-idf"))
Output:
vocabulary: 14 words -> ['again', 'and', 'biryani', 'cold', 'dal', 'food', 'hot', 'late'] ...
review 0: the food was cold and late
count tf-idf idf
the 1 0.28 1.00
was 1 0.34 1.22
and 1 0.42 1.51
food 1 0.42 1.51
late 1 0.42 1.51
cold 1 0.53 1.92
Check by hand: n = 4; "the" is in all 4 reviews → ln(5/5) + 1 = 1.00; "cold" is in 1 → ln(5/2) + 1 = 1.92. Counts say all six words matter equally; TF-IDF gives "the" the lowest weight (0.28) and "cold" the highest (0.53).
TF-IDF on 6,000 reviews
We classify each test review by a vote of its 15 most similar training reviews (kNN):
import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))] # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
"the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c): # c = 1: a complaint
a, b = (bad, good) if c else (good, bad)
say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y]) # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None) # 8% mislabelled
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier
print(docs[0], "->", y[0]); print(docs[1], "->", y[1]) # 1 = complaint
for name, vec in [("counts", CountVectorizer()), ("tf-idf", TfidfVectorizer())]:
Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te]) # vocabulary from TRAIN only
m = KNeighborsClassifier(15).fit(Xtr, y[tr]) # vote of the 15 most similar reviews
print(f"{name:<7} vocabulary {Xtr.shape[1]} 15-NN test {m.score(Xte, y[te]):.3f}")
Output:
the rider called twice the paneer was not fresh -> 1
second order this week the biryani was not tasty ordered for my family -> 1
counts vocabulary 36 15-NN test 0.693
tf-idf vocabulary 36 15-NN test 0.789
With counts, two reviews look similar just because both contain "the order was"; TF-IDF turns those filler words down: 0.693 → 0.789.
n-grams
Single words have a blind spot: "fresh" appears in "fresh" (good) and "not fresh" (complaint). An n-gram is a run of n words in a row: a unigram is one word, a bigram two ("not fresh"), a trigram three. ngram_range=(1, 2) uses words and pairs.
import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))] # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
"the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c): # c = 1: a complaint
a, b = (bad, good) if c else (good, bad)
say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y]) # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None) # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier; from sklearn.linear_model import LogisticRegression
vec = TfidfVectorizer(ngram_range=(1, 2)) # words AND word pairs
Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te])
print("vocabulary:", Xtr.shape[1], " e.g.", [w for w in vec.get_feature_names_out() if w.startswith("not ")][:4])
for M in [KNeighborsClassifier(15), LogisticRegression()]:
print(f"tf-idf + bigrams, {type(M).__name__:<20} test {M.fit(Xtr, y[tr]).score(Xte, y[te]):.3f}")
Output:
vocabulary: 146 e.g. ['not cold', 'not fresh', 'not hot', 'not late']
tf-idf + bigrams, KNeighborsClassifier test 0.889
tf-idf + bigrams, LogisticRegression test 0.922
About 8% of these made-up labels were flipped on purpose, so 0.922 is close to the best possible.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| One-hot, new categories tomorrow | ValueError: Found unknown categories [...] in column 0 during transform |
No column for unseen names | Target/frequency encoding, hashing, or handle_unknown |
| Label codes for names | test 0.697 | Fake order | Frequency or target encoding |
| Naive target encoding (own label included) | train 0.916, test 0.628 | Target-encoding leakage | TargetEncoder().fit_transform (cross fitting) |
TargetEncoder().fit(X, y).transform(X) on train |
train 0.916, test 0.630 | Skips cross fitting | fit_transform on train, transform on test |
| Raw timestamp into a model | ValueError: could not convert string to float: '2026-03-22 02:06:00' |
Text, not numbers | pd.to_datetime + .dt.hour etc. |
| Raw hour for a cyclical pattern | 0.659 vs 0.714 | 23 and 0 look far apart | sin and cos |
| Too few hashed columns | 64 columns: 0.677 | Many collisions | Many more columns than names |
| New vectorizer fitted on new text | ValueError: X has 26 features, but LogisticRegression is expecting 146 features as input. |
Different vocabulary | Fit once on train; use a pipeline |
The two live mistakes from the video:
import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))] # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
"the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c): # c = 1: a complaint
a, b = (bad, good) if c else (good, bad)
say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y]) # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None) # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer as Tfidf
from sklearn.linear_model import LogisticRegression as LR
m = LR().fit(Tfidf(ngram_range=(1, 2)).fit_transform(docs[tr]), y[tr])
new = ["the dosa was soggy", "rider was polite", "the thali was not fresh", "biryani was not cold, loved it"]
try: print(m.predict(Tfidf(ngram_range=(1, 2)).fit_transform(new))) # MISTAKE: a brand-new vectorizer
except ValueError as e: print("ValueError:", e)
from sklearn.pipeline import make_pipeline; model = make_pipeline(Tfidf(ngram_range=(1, 2)), LR()).fit(docs[tr], y[tr])
for text, p in zip(new, model.predict(new)): print(p, "<-", text) # FIX: one pipeline, fitted once
Output:
ValueError: X has 26 features, but LogisticRegression is expecting 146 features as input.
1 <- the dosa was soggy
0 <- rider was polite
1 <- the thali was not fresh
0 <- biryani was not cold, loved it
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier; from sklearn.preprocessing import TargetEncoder
cols = ["restaurant", "customer"]; enc = TargetEncoder().fit(df[cols][tr], y[tr]) # fit on train ...
Xtr = pd.DataFrame(enc.transform(df[cols][tr]), columns=cols).assign(km=df.distance_km[tr].values) # MISTAKE
Xte = pd.DataFrame(enc.transform(df[cols][te]), columns=cols).assign(km=df.distance_km[te].values)
m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, y[tr])
print(f"fit, then transform(train) train {m.score(Xtr, y[tr]):.3f} test {m.score(Xte, y[te]):.3f}")
Output:
fit, then transform(train) train 0.916 test 0.630
Mini project: one leakage-safe pipeline
Put everything into a single Pipeline with a ColumnTransformer, so cross_val_score re-fits the encoders inside each fold (no leakage), and the same object can score tomorrow's raw orders.
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import TargetEncoder, FunctionTransformer
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
def time_features(X): # timestamp text -> weekend + hour on a circle
ts = pd.to_datetime(X.iloc[:, 0]); a = 2 * np.pi * (ts.dt.hour + ts.dt.minute / 60) / 24
return np.c_[(ts.dt.dayofweek >= 5).astype(int), np.sin(a), np.cos(a)]
prep = ColumnTransformer([
("ids", TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0)), ["restaurant", "customer"]),
("time", FunctionTransformer(time_features), ["ordered_at"]),
("num", "passthrough", ["distance_km"])])
model = make_pipeline(prep, HistGradientBoostingClassifier(random_state=0))
scores = cross_val_score(model, df[tr], y[tr], cv=5)
print("5-fold CV accuracy:", scores.round(3), "mean", scores.mean().round(3))
model.fit(df[tr], y[tr]); print("test accuracy:", round(model.score(df[te], y[te]), 3))
Output:
5-fold CV accuracy: [0.842 0.858 0.855 0.842 0.839] mean 0.847
test accuracy: 0.858
Inside the pipeline, fit calls TargetEncoder.fit_transform on each training part (cross fitting), and predict/score call transform. The cross-validated score (0.847) agrees with the test score (0.858 is within the spread of the folds).
Practice
1. Encode the month with sine and cosine. How far apart are December and January? Does it help on Kiran's data?
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
month = pd.to_datetime(df.ordered_at).dt.month # 1..12
msin, mcos = np.sin(2 * np.pi * month / 12), np.cos(2 * np.pi * month / 12)
for m in [12, 1, 6]:
print(f"month {m:>2}: sin {np.sin(2*np.pi*m/12):+.3f} cos {np.cos(2*np.pi*m/12):+.3f}")
gap = np.hypot(np.sin(2*np.pi*12/12) - np.sin(2*np.pi*1/12), np.cos(2*np.pi*12/12) - np.cos(2*np.pi*1/12))
print("December vs January: raw gap 11, circle gap", round(gap, 3))
print("months in the data:", sorted(month.unique().tolist()))
Output:
month 12: sin -0.000 cos +1.000
month 1: sin +0.500 cos +0.866
month 6: sin +0.000 cos -1.000
December vs January: raw gap 11, circle gap 0.518
months in the data: [1, 2, 3]
On a circle of 12 months, December and January are 0.518 apart instead of 11. But Kiran's orders only cover January to March, so the month cannot show a yearly cycle here; cyclical month features pay off when the data spans more than a year.
2. TargetEncoder with smooth = "auto", 1, 10 and 100: compare with 5-fold cross-validation on the training rows.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import TargetEncoder; from sklearn.model_selection import StratifiedKFold, cross_val_score
cols = ["restaurant", "customer"]
for s in ["auto", 1, 10, 100]:
enc = TargetEncoder(smooth=s, cv=StratifiedKFold(5, shuffle=True, random_state=0))
Xtr = pd.DataFrame(enc.fit_transform(df[cols][tr], y[tr]), columns=cols).assign(km=df.distance_km[tr].values)
cv = cross_val_score(HistGradientBoostingClassifier(random_state=0), Xtr, y[tr], cv=5)
print(f"smooth={s!s:<5} 5-fold CV accuracy on train {cv.mean():.3f} (+/- {cv.std():.3f})")
Output:
smooth=auto 5-fold CV accuracy on train 0.756 (+/- 0.004)
smooth=1 5-fold CV accuracy on train 0.760 (+/- 0.005)
smooth=10 5-fold CV accuracy on train 0.758 (+/- 0.004)
smooth=100 5-fold CV accuracy on train 0.753 (+/- 0.005)
All values are within about one spread of each other (0.753-0.760, spread ≈ 0.005), so here smoothing hardly matters, while cross fitting was the big fix. A very large smooth (100) pulls even useful restaurants towards the global mean and is slightly worse.
3. FeatureHasher: what is the smallest power-of-two n_features within 0.01 of the 16,384-column score (0.760)?
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000 # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum()) # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
"customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
"distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1) # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int) # 1 = late
tr, te = slice(0, 22000), slice(22000, None) # 22,000 train · 8,000 test
from sklearn.feature_extraction import FeatureHasher; from sklearn.linear_model import LogisticRegression
from scipy.sparse import hstack
for k in [2**p for p in range(6, 16)]:
H = FeatureHasher(n_features=k, input_type="string").transform(df[["restaurant"]].values)
X = hstack([H, df[["distance_km"]].values]).tocsr()
print(f"{k:>6} columns: test {LogisticRegression(max_iter=1000).fit(X[:22000], y[tr]).score(X[22000:], y[te]):.3f}")
Output:
64 columns: test 0.677
128 columns: test 0.695
256 columns: test 0.707
512 columns: test 0.723
1024 columns: test 0.730
2048 columns: test 0.747
4096 columns: test 0.752
8192 columns: test 0.757
16384 columns: test 0.760
32768 columns: test 0.762
4,096 columns (0.752) is within 0.01 of 0.760. With 2,823 names, about 1.5 columns per name is already close; fewer columns lose accuracy to collisions.
4. Add min_df=5 (and 50) to the TF-IDF + bigrams model. What changes?
Solution
import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))] # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
"the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c): # c = 1: a complaint
a, b = (bad, good) if c else (good, bad)
say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y]) # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None) # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer; from sklearn.linear_model import LogisticRegression
for md in [1, 5, 50]:
vec = TfidfVectorizer(ngram_range=(1, 2), min_df=md) # ignore tokens in fewer than md reviews
Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te])
print(f"min_df={md:<3} vocabulary {Xtr.shape[1]:>4} test {LogisticRegression().fit(Xtr, y[tr]).score(Xte, y[te]):.3f}")
Output:
min_df=1 vocabulary 146 test 0.922
min_df=5 vocabulary 146 test 0.922
min_df=50 vocabulary 144 test 0.922
min_df drops tokens that appear in fewer than min_df documents. This made-up vocabulary is tiny and every pair is common, so almost nothing is dropped and the score stays 0.922. On real reviews with typos and rare words, min_df shrinks the vocabulary a lot and often helps.
5. (No code) A column pincode has 19,000 values and new ones appear weekly. Which encodings would you try, and how would you check them?
Solution
Target encoding with cross fitting (TargetEncoder.fit_transform, maybe with a numeric smooth), frequency encoding as a cheap extra column, or feature hashing with many more columns than pincodes. Compare them with cross-validation inside a pipeline, and keep the test set for the final check only.
Quick quiz
- 2,823 names hashed into 8 columns: about how many names share a column? About 350 (2,823 / 8 ≈ 353).
- Why did naive target encoding of the customer column give train 0.916 but test 0.628? Each one-order customer's encoding was its own label: target-encoding leakage.
- What does
TargetEncoder.fit_transformdo thatfit().transform()doesn't? Cross fitting: each fold is encoded with statistics from the other folds. - With smoothing m = 10 and global rate 0.434, what is a one-order, late restaurant encoded as? (1 + 4.34) / 11 ≈ 0.486.
- Why encode the hour with both sine and cosine? So 23:00 and 00:00 are neighbours, and every hour gets a unique point.
- Which word gets the lowest idf? One that appears in every document (idf = 1).
- What does
ngram_range=(1, 2)add? Word pairs (bigrams) such as "not fresh".
Interview questions
- What is a high-cardinality categorical feature, and why is one-hot a problem? Many distinct values (thousands); one-hot gives thousands of sparse columns, few rows per column, and no column for unseen values.
- Explain target encoding and its main risk. Replace a category with the mean target of that category; the risk is target leakage, especially for rare categories, causing overfitting.
- How do you prevent target-encoding leakage? Cross fitting (out-of-fold encodings) for training rows, encoders fitted inside each CV fold (pipelines), and smoothing towards the global mean.
- What does smoothing do in target encoding? Blends the category mean with the global mean, weighted by the category's count: (n·mean + m·global) / (n + m).
- What is the hashing trick? Pros and cons? Map features to columns with a hash function modulo n_features. Pros: fixed memory, no vocabulary, handles new values, streaming-friendly. Cons: collisions, no way to map a column back to a name.
- Why use sine and cosine for time features? Cyclical values (hour, weekday, month) wrap around; a circle keeps the wrap-around neighbours close. Both are needed for a unique position.
- What date-time features would you extract from a timestamp? Hour, minute, weekday, weekend flag, month, day of month, holiday/festival flags, time since an event; all computed without looking into the future.
- Define TF-IDF. Why does it beat raw counts? Term frequency × inverse document frequency; it down-weights words common to all documents and up-weights rare, informative ones.
- What is an n-gram, and why use bigrams? A sequence of n consecutive tokens; bigrams capture phrases and negation ("not good").
- Why must a text vectorizer be fitted only on training data? The vocabulary and idf values are learned; fitting on test data leaks information and creates a different column set (feature-count mismatch errors).
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Categorical feature | A column whose values are names from a list | restaurant, customer |
| Cardinality | The number of different values in a column | restaurant: 2,823 |
| High-cardinality category | A categorical column with hundreds or thousands of values | 2,823 restaurants, 12,980 customers |
| One-hot encoding | One 0/1 column per category | 2,667 columns |
| Unknown (unseen) category | A value not seen during fitting | tomorrow's new restaurants |
| Label codes | Replacing names with arbitrary integers | R0370 → 370 (fake order) |
| Frequency encoding | Replacing a category with how many training rows have it | busy restaurant → 1,697 |
| Target encoding | Replacing a category with the average target of that category | R0082 → 22/29 = 0.759 |
| Global mean (global rate) | The average target over all training rows | late rate 0.434 |
| Target-encoding leakage | A row's own label inside its encoding | one-order customer → 1.0 |
| Cross fitting | Encoding each fold with statistics from the other folds | TargetEncoder.fit_transform, cv=5 |
| Fold | One of k equal parts of the training rows | 5 folds |
| Smoothing | Blending a category's mean with the global mean by count | (1 + 4.34) / 11 = 0.486 |
smooth |
scikit-learn's smoothing strength m ("auto" or a number) | smooth=10 |
| Empirical Bayes estimate | Choosing the amount of smoothing from the data itself | smooth="auto" |
TargetEncoder |
scikit-learn's target encoder with cross fitting (since 1.3) | test 0.756 |
| Hash function | A rule that turns any text into a number, the same every time | MurmurHash3 |
| Hashing trick (feature hashing) | Using hash(name) mod n_features as the column | R9999 → column 7 |
| Collision | Two names landing in the same hashed column | R9999 and R0002 |
FeatureHasher |
scikit-learn's feature hashing transformer | n_features=16384 |
| Sparse matrix | A table that stores only its non-zero entries | hashed and TF-IDF features |
| Timestamp | A date and time of day | 2026-03-22 02:06:00 |
| Date-time features | Parts of a timestamp used as features | hour, weekday, weekend, month |
| Cyclical feature | A value that wraps around | hour 23 → 0, December → January |
| Cyclical encoding | Placing a cyclical value on a circle with sine and cosine | hour 23 → (−0.259, 0.966) |
| Sine, cosine | The across and up positions of a point on a circle of radius 1 | sin(2π·h/24), cos(2π·h/24) |
| Token | One unit of text, usually a word | "cold" |
| Tokenisation | Splitting text into tokens | default: words of 2+ characters, lowercased |
| Vocabulary | Every token seen when fitting the vectorizer | 14 words; 146 with bigrams |
| Document | One piece of text | one review |
| Bag of words | One count column per vocabulary word | CountVectorizer |
| Term frequency (tf) | How often a word appears in a document | "the": 1 |
| Document frequency (df) | How many documents contain the word | "the": 4 of 4 |
| Inverse document frequency (idf) | ln((1+n)/(1+df)) + 1: higher for rare words | "the" 1.00, "cold" 1.92 |
| TF-IDF | tf × idf, then each row scaled to length 1 | "the" 0.28, "cold" 0.53 |
TfidfVectorizer |
scikit-learn object: tokenise, count and TF-IDF weight in one step | TfidfVectorizer(ngram_range=(1, 2)) |
| n-gram | A run of n consecutive tokens | "not fresh" (n = 2) |
| Unigram, bigram | One-word and two-word n-grams | "fresh", "not fresh" |
min_df |
Ignore tokens found in fewer than this many documents | min_df=5 |
| Pipeline | Preprocessing + model in one object, fitted together | make_pipeline(TfidfVectorizer(), LogisticRegression()) |
Summary
- High-cardinality names break one-hot: thousands of sparse columns, and no column for new values.
- Frequency encoding uses the count; target encoding uses the category's average target.
- Naive target encoding leaks each row's own label (train 0.916, test 0.628); cross fitting with
TargetEncoder.fit_transformfixes it (0.774 / 0.756), and smoothing tames tiny groups. - The hashing trick gives a fixed number of columns for endless names; use many more columns than names to keep collisions rare.
- Split timestamps into date-time features, and put cyclical ones on a circle with sine and cosine (0.659 → 0.714). All together: 0.858.
- TF-IDF weights words by how informative they are (kNN 0.693 → 0.789); bigrams catch "not fresh" (0.922). Fit vectorizers and encoders on training data only, ideally inside a pipeline.
- Next episode: feature selection, keeping only the features that help.