SyntaxPilot
Courses/Machine Learning/Lesson 39
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 39 of 90

Feature engineering 2: categories, dates and text

high-cardinality categoriesfrequency encodingtarget encodingtarget-encoding leakage and smoothinghashing trickdate-time featurescyclical encoding (sine and cosine)TF-IDF featuresTfidfVectorizer

Machine Learning Full Course · Episode 39. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Kiran's 30,000 delivery orders and the 6,000 reviews are made up for teaching.

The big idea: models only eat numbers, but real tables are full of names (restaurants, customers), timestamps and free text. A restaurant column with thousands of values is a high-cardinality category: one-hot encoding makes thousands of mostly-zero columns and breaks on tomorrow's new restaurants. Frequency encoding replaces a name with its count; target encoding replaces it with the average target of that name (its late rate). Done naively, target encoding leaks each row's own answer into its feature (train 0.916, test 0.628 in this episode); cross fitting (scikit-learn's TargetEncoder.fit_transform) and smoothing fix it. When names never stop coming, the hashing trick maps any name to one of a fixed number of columns. Timestamps become date-time features (hour, weekday, weekend), and cyclical ones go on a circle with sine and cosine, so 23:00 and 00:00 are neighbours. Text becomes numbers with TF-IDF, which weights a word by how often it appears in this review and how rare it is overall; n-grams catch phrases like "not fresh".

Sources for the product facts in this episode: scikit-learn's TargetEncoder documentation, FeatureHasher documentation and TfidfTransformer documentation (the formulas used by TfidfVectorizer), and the feature hashing paper by Weinberger, Dasgupta, Attenberg, Langford and Smola (arXiv 0902.2206, 2009).

Words you need first

  • Feature, target (label): an input column, and the thing we predict (Episode 1). Here the target is late (1 = delivered late).
  • Categorical feature: a column whose values are names from a list (Episode 13).
  • One-hot encoding, handle_unknown: one 0/1 column per category (Episode 16).
  • Train / test split, target leakage: Episode 15. Cross-validation, folds: Episode 26.
  • Gradient boosting (HistGradientBoostingClassifier): Episodes 35-36. Logistic regression: Episodes 22-23.
  • k-nearest neighbours (kNN): classify by a vote of the most similar training rows (Episode 27).
  • Bag of words, CountVectorizer: one count column per word (Episode 28).
  • Pipeline: preprocessing and model chained into one object (Episode 17). L2 norm: a vector's length (Episode 3).

The problem: thousands of categories

Kiran runs a food-delivery app. Each of 30,000 made-up orders has a restaurant (2,823 different ones), a customer (12,980), a distance and an order time. Question: will this order be late? The first 22,000 orders are for training, the last 8,000 for testing (they stand for "tomorrow").

The number of different values in a column is its cardinality. A plan column with veg, nonveg and jain has cardinality 3; the restaurant column has cardinality 2,823. A column with hundreds or thousands of values is a high-cardinality category.

The obvious attempt is one-hot encoding:

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.preprocessing import OneHotEncoder
print(df.head(3)); c = df.restaurant[tr].value_counts()
print("restaurants:", df.restaurant.nunique(), "  customers:", df.customer.nunique())
print("median orders per restaurant:", int(c.median()))
oh = OneHotEncoder().fit(df[["restaurant"]][tr])                     # one 0/1 column per restaurant
print("one-hot columns:", oh.transform(df[["restaurant"]][tr]).shape[1])
try: oh.transform(df[["restaurant"]][te])                          # tomorrow's orders
except ValueError as e: print("ValueError:", str(e)[:35], "...", str(e)[-30:])

Output (the list of unknown names differs from run to run, because Python lists a set in a different order each time):

  restaurant customer  distance_km           ordered_at
0      R2508   C04879     5.552547  2026-03-22 02:06:00
1      R2228   C06195     2.742027  2026-01-13 19:06:00
2      R1683   C06265     7.705933  2026-03-01 10:14:00
restaurants: 2823   customers: 12980
median orders per restaurant: 3
one-hot columns: 2667
ValueError: Found unknown categories ['R1581',  ... ] in column 0 during transform

Why one-hot struggles here:

  • 2,667 columns, one per restaurant seen in training, almost all zeros.
  • A typical restaurant has only 3 orders: too little to learn a weight per column.
  • New restaurants arrive every day and have no column: ValueError: Found unknown categories [...] in column 0 during transform. With handle_unknown="ignore" there is no error, but the model knows nothing about them.

The goal: replace each restaurant with one number that means something about lateness.

Label codes and frequency encoding

  • Label codes: use the number in the name (R0370 → 370). The order is fake: restaurant 370 is not "more" than 369.
  • Frequency encoding: replace each category with how many training rows have it. New restaurant → 0. It works when "how common" matters (busy kitchens are late more often in Kiran's data).
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
codes = df.restaurant.str[1:].astype(int)                             # 'R0370' -> 370
counts = df.restaurant[tr].value_counts()                             # orders per restaurant, from TRAIN rows only
freq = df.restaurant.map(counts).fillna(0)                            # frequency encoding (new restaurant -> 0)
for name, col in [("label codes", codes), ("frequency", freq)]:
    X = pd.DataFrame({"restaurant": col, "distance_km": df.distance_km})
    m = HistGradientBoostingClassifier(random_state=0).fit(X[tr], y[tr])
    print(f"{name:<12} train {m.score(X[tr], y[tr]):.3f}   test {m.score(X[te], y[te]):.3f}")

Output:

label codes  train 0.724   test 0.697
frequency    train 0.734   test 0.715

Frequency beats the fake codes, but two restaurants with the same count look identical.

Target encoding

Target encoding replaces each category with the average target of that category in the training data. For a 0/1 target that is the category's rate (here: its late rate); for a number target (delivery minutes) it is the category's average. New categories get the global average.

Example: restaurant R0082 had 29 training orders, 22 of them late → 22 / 29 = 0.759.

Real-life picture: before ordering, you ask a friend "how often has this place been late before?", its track record.

Naive version with pandas groupby:

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
def naive_target(col):                                                # late rate per category, ALL train rows
    rate = y[tr].groupby(df[col][tr]).mean(); return df[col].map(rate).fillna(y[tr].mean())
for cols in [["restaurant"], ["restaurant", "customer"]]:
    X = pd.DataFrame({c: naive_target(c) for c in cols}).assign(distance_km=df.distance_km)
    m = HistGradientBoostingClassifier(random_state=0).fit(X[tr], y[tr])
    print(f"naive target {'+'.join(cols):<20} train {m.score(X[tr], y[tr]):.3f}   test {m.score(X[te], y[te]):.3f}")

Output:

naive target restaurant           train 0.810   test 0.752
naive target restaurant+customer  train 0.916   test 0.628

Restaurant alone: test 0.752, the best so far. But adding the customer makes train 0.916 and test 0.628, worse than the fake label codes.

Target-encoding leakage and cross fitting

Why it failed. Most customers have one or two orders; 5,040 have exactly one training order. Customer C08040 has one order, and it was late, so its encoding is 1.0, made from the very answer we are predicting. The answer has leaked into the feature: target-encoding leakage, a kind of target leakage. It's like an exam paper with the answers printed beside the questions: full marks, nothing learned. On test orders there are no printed answers.

The fix: cross fitting.

  1. Split the training rows into 5 folds (equal parts).
  2. Encode the rows of fold 1 using late rates computed only from folds 2-5; repeat for every fold. No row's encoding uses its own label.
  3. Encode test rows with the table learned from all training rows.

scikit-learn's TargetEncoder (added in version 1.3) does this in fit_transform, with cv=5 folds by default. Its documentation states that fit(X, y).transform(X) does not equal fit_transform(X, y), because only fit_transform uses cross fitting, and recommends fit_transform for training data. Unseen categories are encoded with the global target mean (target_mean_). We pass a fixed StratifiedKFold as cv so the numbers are the same on every run (in scikit-learn 1.9, the shuffle and random_state parameters are deprecated in favour of passing a splitter as cv).

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import TargetEncoder; from sklearn.model_selection import StratifiedKFold
cols = ["restaurant", "customer"]; enc = TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0))
Xtr = pd.DataFrame(enc.fit_transform(df[cols][tr], y[tr]), columns=cols).assign(km=df.distance_km[tr].values)
Xte = pd.DataFrame(enc.transform(df[cols][te]), columns=cols).assign(km=df.distance_km[te].values)
m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, y[tr])
print(f"TargetEncoder restaurant+customer   train {m.score(Xtr, y[tr]):.3f}   test {m.score(Xte, y[te]):.3f}")

Output:

TargetEncoder restaurant+customer   train 0.774   test 0.756

Train and test now agree (0.774 vs 0.756), and test is the best so far.

Train vs test accuracy for each encoding

Smoothing

Tiny groups give wild rates: one late order → 100% late. Smoothing mixes the group's own rate with the global rate (the late rate over all training orders):

encoded = (n · rate + m · global) / (n + m)

n = rows in this category, m = the smoothing strength (smooth in scikit-learn), like m imaginary rows at the global rate. This is the formula scikit-learn uses when smooth is a number. With one late order, global rate 0.434 and m = 10: (1 + 4.34) / 11 = 0.486. A restaurant with 1,697 orders barely moves. The default smooth="auto" picks the amount from the data with an empirical Bayes estimate; the documentation says a larger smooth puts more weight on the global mean.

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.preprocessing import TargetEncoder
R = df[["restaurant"]][tr]; table = pd.DataFrame({"orders": R.restaurant.value_counts(),
                                                  "late_rate": y[tr].groupby(R.restaurant).mean()})
for s in ["auto", 10]:                                             # fit only to LOOK at the encoding table
    enc = TargetEncoder(smooth=s).fit(R, y[tr])
    table[f"smooth={s}"] = pd.Series(enc.encodings_[0], index=enc.categories_[0])
print("global late rate:", round(enc.target_mean_, 3))
print(table.loc[["R0801", "R0082", "R0011", "R0004"]].round(3))

Output:

global late rate: 0.434
            orders  late_rate  smooth=auto  smooth=10
restaurant                                           
R0801         1697      0.608        0.607      0.607
R0082           29      0.759        0.750      0.675
R0011            2      0.000        0.000      0.362
R0004            1      1.000        1.000      0.486

With auto, the one-order and two-order restaurants keep their raw rates in this run: a single order has no spread inside its group for the estimate to measure. With smooth=10 they are pulled towards 0.434. When many groups are tiny, try a few numbers for smooth and compare them with cross-validation (Practice 2). Note that leakage is fixed by cross fitting; smoothing fixes noisy small groups. You usually want both.

The hashing trick

When names never stop coming (millions of restaurants, dishes, users), even storing a name → column table is costly. Real-life picture: a post office with a fixed wall of 1,000 pigeonholes, where a letter goes to the box given by the last three digits of its pincode. No lookup list, just a rule; sometimes two towns share a box.

  • A hash function turns any text into a number, the same number every time.
  • The hashing trick (feature hashing) uses hash(name) mod n_features as the column. Described in the 2009 paper Feature Hashing for Large Scale Multitask Learning (Weinberger et al.).
  • A collision is two names landing in the same column.
  • scikit-learn's FeatureHasher uses the signed 32-bit MurmurHash3; default n_features = 2**20 (1,048,576); alternate_sign=True gives some names −1 instead of +1 so collisions tend to cancel out. It stores no vocabulary, so a brand-new name just works. The result is a sparse matrix: a table that stores only its non-zero entries.
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.feature_extraction import FeatureHasher; from sklearn.linear_model import LogisticRegression
from scipy.sparse import hstack; hasher = lambda k: FeatureHasher(n_features=k, input_type="string")
for r in ["R0001", "R0002", "R1234", "R9999"]:                     # R9999 opens tomorrow: no problem
    v = hasher(8).transform([[r]]).toarray()[0]; print(r, "-> column", v.nonzero()[0][0], " value", int(v.sum()))
for k in [64, 1024, 16384]:                                        # hash 'restaurant' into k columns
    X = hstack([hasher(k).transform(df[["restaurant"]].values), df[["distance_km"]].values]).tocsr()
    m = LogisticRegression(max_iter=1000).fit(X[:22000], y[tr])
    print(f"{k:>6} columns: test {m.score(X[22000:], y[te]):.3f}")

Output:

R0001 -> column 3  value 1
R0002 -> column 7  value 1
R1234 -> column 0  value -1
R9999 -> column 7  value 1
    64 columns: test 0.677
  1024 columns: test 0.730
 16384 columns: test 0.760

R9999 (a restaurant that doesn't exist yet) gets a column with no error, and collides with R0002 in column 7. With 64 columns there are too many collisions (0.677); 16,384 columns give 0.760. Rule of thumb: many more columns than distinct names.

Date-time features

A timestamp like 2026-03-22 02:06:00 is text to a model:

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.linear_model import LogisticRegression
try: LogisticRegression().fit(df[["distance_km", "ordered_at"]][tr], y[tr])   # the timestamp, as it is
except ValueError as e: print("ValueError:", e)
ts = pd.to_datetime(df.ordered_at)                                     # text -> real date-time
F = pd.DataFrame({"hour": ts.dt.hour, "weekday": ts.dt.dayofweek, "weekend": (ts.dt.dayofweek >= 5).astype(int),
                  "month": ts.dt.month, "distance_km": df.distance_km})
print(pd.concat([df.ordered_at, F.drop(columns="distance_km")], axis=1).head(4))
print("late rate by weekend:", y.groupby(F.weekend).mean().round(3).to_dict())

Output:

ValueError: could not convert string to float: '2026-03-22 02:06:00'
            ordered_at  hour  weekday  weekend  month
0  2026-03-22 02:06:00     2        6        1      3
1  2026-01-13 19:06:00    19        1        0      1
2  2026-03-01 10:14:00    10        6        1      3
3  2026-01-26 21:27:00    21        0        0      1
late rate by weekend: {0: 0.406, 1: 0.504}

pd.to_datetime turns text into real date-times; the .dt accessor gives date-time features: hour, dayofweek (Monday = 0, Sunday = 6), a weekend flag, month, and also day, dayofyear, minutes since midnight, "days until a festival", and so on. Weekend orders were late 50.4% of the time vs 40.6% on weekdays.

Cyclical encoding with sine and cosine

As a plain number, 23:00 is 23 and 00:00 is 0: 23 apart, though they are one hour apart in real life (same for December and January, Sunday and Monday). Cyclical encoding puts the value on a circle:

angle = 2π · hour / 24
hour_sin = sin(angle)      hour_cos = cos(angle)

Sine and cosine are simply the across and up positions of a point going round a circle of radius 1. Use both: sine alone gives the same value (0) for 0:00 and 12:00, so you need the pair to tell every hour apart.

Hours on a line vs on a circle

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.linear_model import LogisticRegression
ts = pd.to_datetime(df.ordered_at); hour = ts.dt.hour + ts.dt.minute / 60            # 21:30 -> 21.5
sin, cos = lambda h: np.sin(2 * np.pi * h / 24), lambda h: np.cos(2 * np.pi * h / 24)   # one lap = 24 hours
for h in [23, 0, 12]: print(f"hour {h:>2}:  sin {sin(h):+.3f}   cos {cos(h):+.3f}")
print("23:00 vs 00:00   raw gap: 23   circle gap:", round(np.hypot(sin(23) - sin(0), cos(23) - cos(0)), 3))
base = pd.DataFrame({"weekend": (ts.dt.dayofweek >= 5).astype(int), "distance_km": df.distance_km})
for name, X in [("raw hour", base.assign(hour=hour)), ("sin + cos", base.assign(sin=sin(hour), cos=cos(hour)))]:
    m = LogisticRegression(max_iter=1000).fit(X[tr], y[tr]); print(f"{name:<10} test {m.score(X[te], y[te]):.3f}")

Output:

hour 23:  sin -0.259   cos +0.966
hour  0:  sin +0.000   cos +1.000
hour 12:  sin +0.000   cos -1.000
23:00 vs 00:00   raw gap: 23   circle gap: 0.261
raw hour   test 0.659
sin + cos  test 0.714

The circle gap between 23:00 and 00:00 is 0.261 instead of 23. Logistic regression goes from 0.659 to 0.714: lateness peaks around dinner and stays high after midnight, a shape a straight line in the raw hour cannot follow. Tree models cope better with raw hours, but the circle rarely hurts.

Everything together

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier as HGB; from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import StratifiedKFold; cols = ["restaurant", "customer"]
ts = pd.to_datetime(df.ordered_at); a = 2 * np.pi * (ts.dt.hour + ts.dt.minute / 60) / 24       # hour -> angle
X = pd.DataFrame({"hour_sin": np.sin(a), "hour_cos": np.cos(a), "weekend": (ts.dt.dayofweek >= 5).astype(int),
                  "distance_km": df.distance_km, "restaurant": 0.0, "customer": 0.0})
enc = TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0))       # same folds every run
X.iloc[tr, -2:] = enc.fit_transform(df[cols][tr], y[tr]); X.iloc[te, -2:] = enc.transform(df[cols][te])
m = HGB(random_state=0).fit(X[tr], y[tr]); print("all together: test", round(m.score(X[te], y[te]), 3))

Output:

all together: test 0.858

From 0.697 (label codes) to 0.858: same raw data, better features.

TF-IDF features for text

Kiran also wants to flag complaints in written reviews (6,000 made-up reviews, each labelled 1 = complaint or 0).

  • A token is one unit of text, usually a word. CountVectorizer/TfidfVectorizer by default lowercase the text and use words of 2 or more letters or digits (token_pattern=r"(?u)\b\w\w+\b").
  • The vocabulary is the list of every token seen when fitting.
  • Bag of words (Episode 28): one count column per token. Problem: "the" counts as much as "cold".

Real-life picture: in a family group chat, "good morning" appears every day and tells you nothing; "flood" appears once in months and you read that message.

TF-IDF = term frequency × inverse document frequency. A document is one piece of text (one review).

tf(t, d)     = count of word t in document d
idf(t)       = ln((1 + n) / (1 + df(t))) + 1          # scikit-learn default, smooth_idf=True
tf-idf(t, d) = tf(t, d) × idf(t)                      # then each row scaled to L2 length 1 (norm='l2')

n = number of documents, df(t) (document frequency) = number of documents containing t. A word in every document gets the minimum idf, 1. Scaling each row to length 1 means long reviews don't get bigger numbers just for being long. TfidfVectorizer = CountVectorizer followed by this weighting, in one object.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
reviews = ["the food was cold and late",                         # 4 made-up reviews of Kiran's app
           "the food was hot and tasty",
           "late again, the dal spilled",
           "tasty biryani, the rider was polite"]
cv, tf = CountVectorizer(), TfidfVectorizer()                   # tokens: words of 2+ letters, lowercased
counts, weights = cv.fit_transform(reviews), tf.fit_transform(reviews)   # sparse matrices
print("vocabulary:", len(tf.vocabulary_), "words ->", sorted(tf.vocabulary_)[:8], "...")
row = pd.DataFrame({"count": counts[0].toarray()[0], "tf-idf": weights[0].toarray()[0].round(2),
                    "idf": tf.idf_.round(2)}, index=cv.get_feature_names_out())
print("review 0:", reviews[0]); print(row[row["count"] > 0].sort_values("tf-idf"))

Output:

vocabulary: 14 words -> ['again', 'and', 'biryani', 'cold', 'dal', 'food', 'hot', 'late'] ...
review 0: the food was cold and late
      count  tf-idf   idf
the       1    0.28  1.00
was       1    0.34  1.22
and       1    0.42  1.51
food      1    0.42  1.51
late      1    0.42  1.51
cold      1    0.53  1.92

Check by hand: n = 4; "the" is in all 4 reviews → ln(5/5) + 1 = 1.00; "cold" is in 1 → ln(5/2) + 1 = 1.92. Counts say all six words matter equally; TF-IDF gives "the" the lowest weight (0.28) and "cold" the highest (0.53).

TF-IDF on 6,000 reviews

We classify each test review by a vote of its 15 most similar training reviews (kNN):

import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))]   # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
        "the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c):                                             # c = 1: a complaint
    a, b = (bad, good) if c else (good, bad)
    say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
    return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y])                  # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None)   # 8% mislabelled
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier
print(docs[0], "->", y[0]); print(docs[1], "->", y[1])                 # 1 = complaint
for name, vec in [("counts", CountVectorizer()), ("tf-idf", TfidfVectorizer())]:
    Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te])   # vocabulary from TRAIN only
    m = KNeighborsClassifier(15).fit(Xtr, y[tr])                       # vote of the 15 most similar reviews
    print(f"{name:<7} vocabulary {Xtr.shape[1]}   15-NN test {m.score(Xte, y[te]):.3f}")

Output:

the rider called twice the paneer was not fresh -> 1
second order this week the biryani was not tasty ordered for my family -> 1
counts  vocabulary 36   15-NN test 0.693
tf-idf  vocabulary 36   15-NN test 0.789

With counts, two reviews look similar just because both contain "the order was"; TF-IDF turns those filler words down: 0.693 → 0.789.

n-grams

Single words have a blind spot: "fresh" appears in "fresh" (good) and "not fresh" (complaint). An n-gram is a run of n words in a row: a unigram is one word, a bigram two ("not fresh"), a trigram three. ngram_range=(1, 2) uses words and pairs.

import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))]   # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
        "the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c):                                             # c = 1: a complaint
    a, b = (bad, good) if c else (good, bad)
    say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
    return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y])                  # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None)   # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier; from sklearn.linear_model import LogisticRegression
vec = TfidfVectorizer(ngram_range=(1, 2))                              # words AND word pairs
Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te])
print("vocabulary:", Xtr.shape[1], " e.g.", [w for w in vec.get_feature_names_out() if w.startswith("not ")][:4])
for M in [KNeighborsClassifier(15), LogisticRegression()]:
    print(f"tf-idf + bigrams, {type(M).__name__:<20} test {M.fit(Xtr, y[tr]).score(Xte, y[te]):.3f}")

Output:

vocabulary: 146  e.g. ['not cold', 'not fresh', 'not hot', 'not late']
tf-idf + bigrams, KNeighborsClassifier test 0.889
tf-idf + bigrams, LogisticRegression   test 0.922

About 8% of these made-up labels were flipped on purpose, so 0.922 is close to the best possible.

Common mistakes

Mistake Exact message / symptom Why Fix
One-hot, new categories tomorrow ValueError: Found unknown categories [...] in column 0 during transform No column for unseen names Target/frequency encoding, hashing, or handle_unknown
Label codes for names test 0.697 Fake order Frequency or target encoding
Naive target encoding (own label included) train 0.916, test 0.628 Target-encoding leakage TargetEncoder().fit_transform (cross fitting)
TargetEncoder().fit(X, y).transform(X) on train train 0.916, test 0.630 Skips cross fitting fit_transform on train, transform on test
Raw timestamp into a model ValueError: could not convert string to float: '2026-03-22 02:06:00' Text, not numbers pd.to_datetime + .dt.hour etc.
Raw hour for a cyclical pattern 0.659 vs 0.714 23 and 0 look far apart sin and cos
Too few hashed columns 64 columns: 0.677 Many collisions Many more columns than names
New vectorizer fitted on new text ValueError: X has 26 features, but LogisticRegression is expecting 146 features as input. Different vocabulary Fit once on train; use a pipeline

The two live mistakes from the video:

import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))]   # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
        "the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c):                                             # c = 1: a complaint
    a, b = (bad, good) if c else (good, bad)
    say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
    return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y])                  # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None)   # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer as Tfidf
from sklearn.linear_model import LogisticRegression as LR
m = LR().fit(Tfidf(ngram_range=(1, 2)).fit_transform(docs[tr]), y[tr])
new = ["the dosa was soggy", "rider was polite", "the thali was not fresh", "biryani was not cold, loved it"]
try: print(m.predict(Tfidf(ngram_range=(1, 2)).fit_transform(new)))            # MISTAKE: a brand-new vectorizer
except ValueError as e: print("ValueError:", e)
from sklearn.pipeline import make_pipeline; model = make_pipeline(Tfidf(ngram_range=(1, 2)), LR()).fit(docs[tr], y[tr])
for text, p in zip(new, model.predict(new)): print(p, "<-", text)            # FIX: one pipeline, fitted once

Output:

ValueError: X has 26 features, but LogisticRegression is expecting 146 features as input.
1 <- the dosa was soggy
0 <- rider was polite
1 <- the thali was not fresh
0 <- biryani was not cold, loved it
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier; from sklearn.preprocessing import TargetEncoder
cols = ["restaurant", "customer"]; enc = TargetEncoder().fit(df[cols][tr], y[tr])      # fit on train ...
Xtr = pd.DataFrame(enc.transform(df[cols][tr]), columns=cols).assign(km=df.distance_km[tr].values)  # MISTAKE
Xte = pd.DataFrame(enc.transform(df[cols][te]), columns=cols).assign(km=df.distance_km[te].values)
m = HistGradientBoostingClassifier(random_state=0).fit(Xtr, y[tr])
print(f"fit, then transform(train)   train {m.score(Xtr, y[tr]):.3f}   test {m.score(Xte, y[te]):.3f}")

Output:

fit, then transform(train)   train 0.916   test 0.630

Mini project: one leakage-safe pipeline

Put everything into a single Pipeline with a ColumnTransformer, so cross_val_score re-fits the encoders inside each fold (no leakage), and the same object can score tomorrow's raw orders.

import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import TargetEncoder, FunctionTransformer
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

def time_features(X):                                   # timestamp text -> weekend + hour on a circle
    ts = pd.to_datetime(X.iloc[:, 0]); a = 2 * np.pi * (ts.dt.hour + ts.dt.minute / 60) / 24
    return np.c_[(ts.dt.dayofweek >= 5).astype(int), np.sin(a), np.cos(a)]

prep = ColumnTransformer([
    ("ids", TargetEncoder(cv=StratifiedKFold(5, shuffle=True, random_state=0)), ["restaurant", "customer"]),
    ("time", FunctionTransformer(time_features), ["ordered_at"]),
    ("num", "passthrough", ["distance_km"])])
model = make_pipeline(prep, HistGradientBoostingClassifier(random_state=0))
scores = cross_val_score(model, df[tr], y[tr], cv=5)
print("5-fold CV accuracy:", scores.round(3), "mean", scores.mean().round(3))
model.fit(df[tr], y[tr]); print("test accuracy:", round(model.score(df[te], y[te]), 3))

Output:

5-fold CV accuracy: [0.842 0.858 0.855 0.842 0.839] mean 0.847
test accuracy: 0.858

Inside the pipeline, fit calls TargetEncoder.fit_transform on each training part (cross fitting), and predict/score call transform. The cross-validated score (0.847) agrees with the test score (0.858 is within the spread of the folds).

Practice

1. Encode the month with sine and cosine. How far apart are December and January? Does it help on Kiran's data?

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
month = pd.to_datetime(df.ordered_at).dt.month                       # 1..12
msin, mcos = np.sin(2 * np.pi * month / 12), np.cos(2 * np.pi * month / 12)
for m in [12, 1, 6]:
    print(f"month {m:>2}: sin {np.sin(2*np.pi*m/12):+.3f}  cos {np.cos(2*np.pi*m/12):+.3f}")
gap = np.hypot(np.sin(2*np.pi*12/12) - np.sin(2*np.pi*1/12), np.cos(2*np.pi*12/12) - np.cos(2*np.pi*1/12))
print("December vs January: raw gap 11, circle gap", round(gap, 3))
print("months in the data:", sorted(month.unique().tolist()))

Output:

month 12: sin -0.000  cos +1.000
month  1: sin +0.500  cos +0.866
month  6: sin +0.000  cos -1.000
December vs January: raw gap 11, circle gap 0.518
months in the data: [1, 2, 3]

On a circle of 12 months, December and January are 0.518 apart instead of 11. But Kiran's orders only cover January to March, so the month cannot show a yearly cycle here; cyclical month features pay off when the data spans more than a year.

2. TargetEncoder with smooth = "auto", 1, 10 and 100: compare with 5-fold cross-validation on the training rows.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import TargetEncoder; from sklearn.model_selection import StratifiedKFold, cross_val_score
cols = ["restaurant", "customer"]
for s in ["auto", 1, 10, 100]:
    enc = TargetEncoder(smooth=s, cv=StratifiedKFold(5, shuffle=True, random_state=0))
    Xtr = pd.DataFrame(enc.fit_transform(df[cols][tr], y[tr]), columns=cols).assign(km=df.distance_km[tr].values)
    cv = cross_val_score(HistGradientBoostingClassifier(random_state=0), Xtr, y[tr], cv=5)
    print(f"smooth={s!s:<5} 5-fold CV accuracy on train {cv.mean():.3f} (+/- {cv.std():.3f})")

Output:

smooth=auto  5-fold CV accuracy on train 0.756 (+/- 0.004)
smooth=1     5-fold CV accuracy on train 0.760 (+/- 0.005)
smooth=10    5-fold CV accuracy on train 0.758 (+/- 0.004)
smooth=100   5-fold CV accuracy on train 0.753 (+/- 0.005)

All values are within about one spread of each other (0.753-0.760, spread ≈ 0.005), so here smoothing hardly matters, while cross fitting was the big fix. A very large smooth (100) pulls even useful restaurants towards the global mean and is slightly worse.

3. FeatureHasher: what is the smallest power-of-two n_features within 0.01 of the 16,384-column score (0.760)?

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(39); n = 30000                            # Kiran's delivery app (made up)
pop = 1 / np.arange(1, 3001) ** 0.9; rid = rng.choice(3000, n, p=pop / pop.sum())   # 3,000 restaurants
t = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 90 * 1440, n), "min")
df = pd.DataFrame({"restaurant": [f"R{i:04d}" for i in rng.permutation(3000)[rid]],
                   "customer": [f"C{i:05d}" for i in rng.integers(0, 15000, n)],
                   "distance_km": rng.uniform(.5, 9, n), "ordered_at": t.astype(str)})
h = t.hour + t.minute / 60; busy = -.4 * np.log(rid + 1)                # popular kitchens are busier
s = rng.normal(0, 1.3, 3000)[rid] + busy + .25 * df.distance_km + 1.4 * np.cos((h - 21) * np.pi / 12)
y = (s + rng.normal(0, .7, n) + .5 * (t.dayofweek >= 5) > np.quantile(s, .6)).astype(int)  # 1 = late
tr, te = slice(0, 22000), slice(22000, None)                          # 22,000 train · 8,000 test
from sklearn.feature_extraction import FeatureHasher; from sklearn.linear_model import LogisticRegression
from scipy.sparse import hstack
for k in [2**p for p in range(6, 16)]:
    H = FeatureHasher(n_features=k, input_type="string").transform(df[["restaurant"]].values)
    X = hstack([H, df[["distance_km"]].values]).tocsr()
    print(f"{k:>6} columns: test {LogisticRegression(max_iter=1000).fit(X[:22000], y[tr]).score(X[22000:], y[te]):.3f}")

Output:

    64 columns: test 0.677
   128 columns: test 0.695
   256 columns: test 0.707
   512 columns: test 0.723
  1024 columns: test 0.730
  2048 columns: test 0.747
  4096 columns: test 0.752
  8192 columns: test 0.757
 16384 columns: test 0.760
 32768 columns: test 0.762

4,096 columns (0.752) is within 0.01 of 0.760. With 2,823 names, about 1.5 columns per name is already close; fewer columns lose accuracy to collisions.

4. Add min_df=5 (and 50) to the TF-IDF + bigrams model. What changes?

Solution
import numpy as np
rng = np.random.default_rng(39); n = 6000; pick = lambda xs: xs[rng.integers(len(xs))]   # made-up reviews
good, bad = ["tasty", "hot", "fresh", "polite", "quick"], ["cold", "late", "soggy", "rude", "stale"]
dish = ["biryani", "dal", "paneer", "dosa", "thali"]; extra = ["ordered for my family", "second order this week",
        "the packaging was okay", "paid by upi", "the rider called twice", "it was raining", ""]
def review(c):                                             # c = 1: a complaint
    a, b = (bad, good) if c else (good, bad)
    say = pick([f"the {pick(dish)} was {pick(a)}", f"the {pick(dish)} was not {pick(b)}", f"rider was {pick(a)}"])
    return f"{pick(extra)} {say} {pick(extra)}".strip()
y = rng.integers(0, 2, n); docs = np.array([review(c) for c in y])                  # 1 = complaint
y = np.where(rng.random(n) < .08, 1 - y, y); tr, te = slice(0, 4500), slice(4500, None)   # 8% mislabelled
from sklearn.feature_extraction.text import TfidfVectorizer; from sklearn.linear_model import LogisticRegression
for md in [1, 5, 50]:
    vec = TfidfVectorizer(ngram_range=(1, 2), min_df=md)              # ignore tokens in fewer than md reviews
    Xtr = vec.fit_transform(docs[tr]); Xte = vec.transform(docs[te])
    print(f"min_df={md:<3} vocabulary {Xtr.shape[1]:>4}   test {LogisticRegression().fit(Xtr, y[tr]).score(Xte, y[te]):.3f}")

Output:

min_df=1   vocabulary  146   test 0.922
min_df=5   vocabulary  146   test 0.922
min_df=50  vocabulary  144   test 0.922

min_df drops tokens that appear in fewer than min_df documents. This made-up vocabulary is tiny and every pair is common, so almost nothing is dropped and the score stays 0.922. On real reviews with typos and rare words, min_df shrinks the vocabulary a lot and often helps.

5. (No code) A column pincode has 19,000 values and new ones appear weekly. Which encodings would you try, and how would you check them?

Solution

Target encoding with cross fitting (TargetEncoder.fit_transform, maybe with a numeric smooth), frequency encoding as a cheap extra column, or feature hashing with many more columns than pincodes. Compare them with cross-validation inside a pipeline, and keep the test set for the final check only.

Quick quiz

  1. 2,823 names hashed into 8 columns: about how many names share a column? About 350 (2,823 / 8 ≈ 353).
  2. Why did naive target encoding of the customer column give train 0.916 but test 0.628? Each one-order customer's encoding was its own label: target-encoding leakage.
  3. What does TargetEncoder.fit_transform do that fit().transform() doesn't? Cross fitting: each fold is encoded with statistics from the other folds.
  4. With smoothing m = 10 and global rate 0.434, what is a one-order, late restaurant encoded as? (1 + 4.34) / 11 ≈ 0.486.
  5. Why encode the hour with both sine and cosine? So 23:00 and 00:00 are neighbours, and every hour gets a unique point.
  6. Which word gets the lowest idf? One that appears in every document (idf = 1).
  7. What does ngram_range=(1, 2) add? Word pairs (bigrams) such as "not fresh".

Interview questions

  1. What is a high-cardinality categorical feature, and why is one-hot a problem? Many distinct values (thousands); one-hot gives thousands of sparse columns, few rows per column, and no column for unseen values.
  2. Explain target encoding and its main risk. Replace a category with the mean target of that category; the risk is target leakage, especially for rare categories, causing overfitting.
  3. How do you prevent target-encoding leakage? Cross fitting (out-of-fold encodings) for training rows, encoders fitted inside each CV fold (pipelines), and smoothing towards the global mean.
  4. What does smoothing do in target encoding? Blends the category mean with the global mean, weighted by the category's count: (n·mean + m·global) / (n + m).
  5. What is the hashing trick? Pros and cons? Map features to columns with a hash function modulo n_features. Pros: fixed memory, no vocabulary, handles new values, streaming-friendly. Cons: collisions, no way to map a column back to a name.
  6. Why use sine and cosine for time features? Cyclical values (hour, weekday, month) wrap around; a circle keeps the wrap-around neighbours close. Both are needed for a unique position.
  7. What date-time features would you extract from a timestamp? Hour, minute, weekday, weekend flag, month, day of month, holiday/festival flags, time since an event; all computed without looking into the future.
  8. Define TF-IDF. Why does it beat raw counts? Term frequency × inverse document frequency; it down-weights words common to all documents and up-weights rare, informative ones.
  9. What is an n-gram, and why use bigrams? A sequence of n consecutive tokens; bigrams capture phrases and negation ("not good").
  10. Why must a text vectorizer be fitted only on training data? The vocabulary and idf values are learned; fitting on test data leaks information and creates a different column set (feature-count mismatch errors).

Glossary

Term Meaning (plain words) Example from this episode
Categorical feature A column whose values are names from a list restaurant, customer
Cardinality The number of different values in a column restaurant: 2,823
High-cardinality category A categorical column with hundreds or thousands of values 2,823 restaurants, 12,980 customers
One-hot encoding One 0/1 column per category 2,667 columns
Unknown (unseen) category A value not seen during fitting tomorrow's new restaurants
Label codes Replacing names with arbitrary integers R0370 → 370 (fake order)
Frequency encoding Replacing a category with how many training rows have it busy restaurant → 1,697
Target encoding Replacing a category with the average target of that category R0082 → 22/29 = 0.759
Global mean (global rate) The average target over all training rows late rate 0.434
Target-encoding leakage A row's own label inside its encoding one-order customer → 1.0
Cross fitting Encoding each fold with statistics from the other folds TargetEncoder.fit_transform, cv=5
Fold One of k equal parts of the training rows 5 folds
Smoothing Blending a category's mean with the global mean by count (1 + 4.34) / 11 = 0.486
smooth scikit-learn's smoothing strength m ("auto" or a number) smooth=10
Empirical Bayes estimate Choosing the amount of smoothing from the data itself smooth="auto"
TargetEncoder scikit-learn's target encoder with cross fitting (since 1.3) test 0.756
Hash function A rule that turns any text into a number, the same every time MurmurHash3
Hashing trick (feature hashing) Using hash(name) mod n_features as the column R9999 → column 7
Collision Two names landing in the same hashed column R9999 and R0002
FeatureHasher scikit-learn's feature hashing transformer n_features=16384
Sparse matrix A table that stores only its non-zero entries hashed and TF-IDF features
Timestamp A date and time of day 2026-03-22 02:06:00
Date-time features Parts of a timestamp used as features hour, weekday, weekend, month
Cyclical feature A value that wraps around hour 23 → 0, December → January
Cyclical encoding Placing a cyclical value on a circle with sine and cosine hour 23 → (−0.259, 0.966)
Sine, cosine The across and up positions of a point on a circle of radius 1 sin(2π·h/24), cos(2π·h/24)
Token One unit of text, usually a word "cold"
Tokenisation Splitting text into tokens default: words of 2+ characters, lowercased
Vocabulary Every token seen when fitting the vectorizer 14 words; 146 with bigrams
Document One piece of text one review
Bag of words One count column per vocabulary word CountVectorizer
Term frequency (tf) How often a word appears in a document "the": 1
Document frequency (df) How many documents contain the word "the": 4 of 4
Inverse document frequency (idf) ln((1+n)/(1+df)) + 1: higher for rare words "the" 1.00, "cold" 1.92
TF-IDF tf × idf, then each row scaled to length 1 "the" 0.28, "cold" 0.53
TfidfVectorizer scikit-learn object: tokenise, count and TF-IDF weight in one step TfidfVectorizer(ngram_range=(1, 2))
n-gram A run of n consecutive tokens "not fresh" (n = 2)
Unigram, bigram One-word and two-word n-grams "fresh", "not fresh"
min_df Ignore tokens found in fewer than this many documents min_df=5
Pipeline Preprocessing + model in one object, fitted together make_pipeline(TfidfVectorizer(), LogisticRegression())

Summary

  • High-cardinality names break one-hot: thousands of sparse columns, and no column for new values.
  • Frequency encoding uses the count; target encoding uses the category's average target.
  • Naive target encoding leaks each row's own label (train 0.916, test 0.628); cross fitting with TargetEncoder.fit_transform fixes it (0.774 / 0.756), and smoothing tames tiny groups.
  • The hashing trick gives a fixed number of columns for endless names; use many more columns than names to keep collisions rare.
  • Split timestamps into date-time features, and put cyclical ones on a circle with sine and cosine (0.659 → 0.714). All together: 0.858.
  • TF-IDF weights words by how informative they are (kNN 0.693 → 0.789); bigrams catch "not fresh" (0.922). Fit vectorizers and encoders on training data only, ideally inside a pipeline.
  • Next episode: feature selection, keeping only the features that help.