SyntaxPilot
Courses/Machine Learning/Lesson 40
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 40 of 90

Feature selection

feature selectionfilter methodsvariance thresholdcorrelation filtermutual informationchi-square selectionwrapper methodsrecursive feature elimination (RFE)embedded methodsvariance inflation factor (VIF)

Machine Learning Full Course · Episode 40. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Kiran's 3,000 delivery orders are made up for teaching, which is the only reason we know which four columns truly matter.

The big idea: more columns are not always better. Useless columns (noise), constant columns and near-duplicate columns make a model slower, harder to explain and often less accurate: with all 201 columns, a 15-nearest-neighbour model scored 0.691 on test; with the 4 columns that matter it scored 0.855. Feature selection means keeping a useful subset of the columns you already have. There are three families. Filter methods score each column on its own, quickly (variance threshold, correlation filter, mutual information, chi-square). Wrapper methods train the real model on different subsets (forward selection, backward selection, recursive feature elimination). Embedded methods select while the model trains (L1 penalty, tree importance with SelectFromModel). The variance inflation factor (VIF) finds a column hidden inside a combination of others. And because selection learns from the labels, it must happen inside the pipeline, refitted in every cross-validation fold.

Sources for the product facts in this episode: scikit-learn's feature_selection API list and SequentialFeatureSelector page (greedy forward or backward selection by cross-validation score, added in version 0.24), the docstrings of the installed scikit-learn 1.9.1 (mutual information is non-negative and zero only for independent variables, estimated from k-nearest-neighbour distances; chi2 needs non-negative values such as booleans or counts; RFE prunes the least important features recursively; VarianceThreshold with the default threshold 0 removes constant features), and the statsmodels variance_inflation_factor page ("if VIF is greater than 5, then the explanatory variable ... is highly collinear").

Words you need first

  • Feature (column), target (label): an input column, and the thing we predict (Episode 1). Here the target is late (1 = delivered late).
  • Train / test split, data leakage: Episode 15. Cross-validation, folds: Episode 26.
  • k-nearest neighbours (kNN): classify a row by a vote of the k most similar training rows (Episode 27). Curse of dimensionality: with many columns, "nearest" stops meaning much (Episode 27).
  • Overfitting: learning noise that does not repeat on new data (Episode 20).
  • Correlation: how closely two columns move together in a straight line, from −1 to 1 (Episode 11). p-value: Episode 11.
  • Entropy: uncertainty, measured in bits or nats (Episode 12).
  • L1 penalty (lasso): a penalty that pushes weights to exactly zero (Episode 21).
  • Random forest, impurity importance: Episode 34. Pipeline: preprocessing and model chained into one object (Episode 17).
  • Multicollinearity, R²: Episode 19.

The problem: when more columns hurt

Kiran's delivery app logged everything it could for 3,000 made-up orders: distance, hour, rain, number of items, kitchen preparation time, food bill, delivery fee, total bill, distance in metres, the app version, a city code, and 190 technical log columns. 201 columns in all. Only four of them (distance, hour, rain, preparation time) drive lateness. The obvious idea is "more information must be better", so give the model every column:

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
print(X.shape); print(list(X.columns[:6])); print(list(X.columns[6:12]), "... log_190")
real = ["distance_km", "hour", "raining", "prep_min"]                 # what drives lateness (we made the data)
for name, cols in [("all 201 columns", X.columns), ("the 4 that matter", real)]:
    m = knn().fit(X[cols][tr], y[tr])
    print(f"{name:<18} train {m.score(X[cols][tr], y[tr]):.3f}   test {m.score(X[cols][te], y[te]):.3f}")

Output:

(3000, 201)
['distance_km', 'hour', 'raining', 'items', 'prep_min', 'food_bill']
['delivery_fee', 'total_bill', 'distance_m', 'app_version', 'city_code', 'log_001'] ... log_190
all 201 columns    train 0.755   test 0.691
the 4 that matter  train 0.878   test 0.855

All 201 columns: test 0.691. Only the 4 that matter: 0.855. Same model, same rows.

Why? kNN measures distance across all columns at once. 197 useless columns add random differences that drown the 4 real ones: two orders with the same distance and hour no longer look similar. With many columns and few rows, models also find fake patterns in noise (overfitting). And every extra column costs training time, storage and explanation.

Feature selection is choosing a subset of the columns you already have and dropping the rest. It is different from feature engineering (Episodes 38-39), which creates new columns.

Three families, like picking a cricket eleven

Picking a team Picking features Speed Sees columns together?
Rank players by their own batting average, take the top 11 (might get 11 openers) Filter method: score each column on its own Fast No
Try line-ups in practice matches, keep the one that wins Wrapper method: train the real model on different subsets Slow Yes
Bench whoever doesn't contribute during the season Embedded method: the model drops columns as part of training One fit Yes

Filter methods 1: constant columns and twins

Variance measures how much a column's values spread out (Episode 10). A column with variance 0 has the same value in every row, like a quiz question every student answers the same way: it cannot separate anyone. VarianceThreshold(threshold=0) removes features whose training-set variance is at or below the threshold; with the default 0 it removes only constant columns. It never looks at the target, so it is safe even before a split (but fit it on training rows anyway, as a habit).

A correlation filter removes near-duplicates: if two columns have |correlation| near 1 they carry the same information, so keep one of each pair.

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import VarianceThreshold
vt = VarianceThreshold(threshold=0).fit(X[tr])                          # columns that never change
const = list(X.columns[~vt.get_support()]); print("constant columns:", const)
r = X[tr].drop(columns=const).corr().abs()                              # |correlation| of every pair
pairs = r.where(np.triu(np.ones(r.shape, bool), k=1)).stack()          # each pair once
print(pairs[pairs > 0.95].round(4))
twins = sorted({b for (a, b), v in pairs.items() if v > 0.95})          # keep the first of each twin pair
clean = X.drop(columns=const + twins); print("drop:", twins, "  columns left:", clean.shape[1])

Output:

constant columns: ['app_version', 'city_code']
distance_km   delivery_fee    0.9801
              distance_m      0.9999
food_bill     total_bill      0.9972
delivery_fee  distance_m      0.9799
dtype: float64
drop: ['delivery_fee', 'distance_m', 'total_bill']   columns left: 196
  • app_version and city_code never change.
  • distance_m is distance_km × 1000 (correlation 0.9999); total_bill is food_bill + delivery_fee (0.9972); delivery_fee is set from the distance (0.9801).
  • After dropping 2 constant columns and 3 twins, 196 columns are left.

Filter methods 2: mutual information

Can we rank columns by their correlation with the target and keep the best? Look at the hour: orders around lunch (11:00 to 15:00) are late about 80% of the time, night orders about 20%. A huge pattern, but it goes up and then down like a hill, so the best straight line through it is almost flat: correlation 0.09. Ranking by correlation would throw away one of the four columns that matter.

Mutual information (MI) asks a more general question: how much does knowing this column tell me about the target? Knowing the sky is full of dark clouds tells you a lot about rain; knowing today's date digit tells you nothing. MI is 0 when the two are independent (one tells you nothing about the other), and bigger when knowing the column removes more uncertainty (entropy, Episode 12) about the target. It does not care about the shape of the relationship: hills, steps and lines all count. scikit-learn's mutual_info_classif estimates it from the data with k-nearest-neighbour distances (n_neighbors=3 by default); mutual_info_regression is the version for a numeric target.

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import mutual_info_classif
cols = ["distance_km", "hour", "raining", "prep_min", "items", "log_001"]
corr = X[cols][tr].corrwith(y[tr]).abs()                                 # straight-line relationship only
mi = mutual_info_classif(X[cols][tr], y[tr], random_state=0)            # any kind of relationship
print(pd.DataFrame({"|correlation|": corr, "mutual info": mi}, index=cols).round(3))
print("late rate by hour:", y[tr].groupby(X.hour[tr] // 4 * 4).mean().round(2).to_dict())

Output:

             |correlation|  mutual info
distance_km          0.485        0.138
hour                 0.092        0.080
raining              0.291        0.051
prep_min             0.328        0.048
items                0.000        0.000
log_001              0.008        0.000
late rate by hour: {0: 0.26, 4: 0.43, 8: 0.68, 12: 0.8, 16: 0.55, 20: 0.34}
Column |correlation| Mutual information What it means
distance_km 0.485 0.138 Strong, and roughly a straight line
hour 0.092 0.080 Strong hill: correlation misses it, MI sees it
raining, prep_min 0.29, 0.33 0.05 Real effects
items, log_001 0.000 0.000 Nothing

Filter methods 3: the chi-square test

The chi-square test (χ², "chi" rhymes with "sky") is a classic filter for count and 0/1 columns against a class. It asks: if rain had nothing to do with lateness, how many late orders would we expect on rainy days? and compares that with the number actually observed. The bigger the gap between expected and observed counts, the bigger the chi-square statistic and the smaller the p-value (the chance of a gap that big if the column were truly unrelated). scikit-learn's chi2 needs non-negative values such as booleans or frequencies (for example word counts in text classification).

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import chi2
counts = X[["items", "raining", "hour"]][tr]                             # non-negative counts and 0/1 flags
stat, p = chi2(counts, y[tr])
print(pd.DataFrame({"chi2": stat.round(1), "p-value": p.round(4)}, index=counts.columns))
try: chi2(X[["distance_km", "log_001"]][tr], y[tr])                      # log_001 has negative numbers
except ValueError as e: print("ValueError:", e)

Output:

         chi2  p-value
items     0.0   0.9849
raining  85.5   0.0000
hour     72.2   0.0000
ValueError: Input X must be non-negative.

Raining (85.5) and hour (72.2) are strongly related to lateness; the number of items (0.0, p-value 0.98) is not. A column with negative numbers gives ValueError: Input X must be non-negative. For ordinary numeric columns, use mutual information, or bin them into counts first (for example with KBinsDiscretizer).

Why filters fail: twins

SelectKBest(score, k) keeps the k columns with the highest filter scores. With mutual information on all 201 columns:

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import SelectKBest, mutual_info_classif
mi = lambda A, b: mutual_info_classif(A, b, random_state=0)
sel = SelectKBest(mi, k=4).fit(X[tr], y[tr])                            # filter: the 4 best scores, one by one
cols = sel.get_feature_names_out(); print("filter picked:", list(cols))
m = knn().fit(X[cols][tr], y[tr]); print("15-NN test:", round(m.score(X[cols][te], y[te]), 3))

Output:

filter picked: ['distance_km', 'hour', 'delivery_fee', 'distance_m']
15-NN test: 0.753

It picked distance in kilometres, the hour, the delivery fee and distance in metres: three copies of the same information, test 0.753. A filter scores each column alone; every copy of distance looks great on its own, so it takes all three, like the selector who picked eleven openers. Rain and preparation time never got a seat. (Running the correlation filter first would have removed the twins: filters work best as a combination of cheap checks.)

Wrapper methods: forward selection

A wrapper method wraps the selection around the real model: it judges subsets of columns by how well the model does with them. Forward selection starts with no columns, tries each column alone (scored by cross-validation), keeps the best, then tries every remaining column alongside it, keeps the best pair, and so on. A second copy of distance adds nothing once distance is in, so it never wins a slot. It is greedy: at each step it takes the best move now and never undoes a choice. Backward selection is the mirror image: start with everything and remove the least useful column each step. scikit-learn's SequentialFeatureSelector (added in 0.24) does both (direction="forward" or "backward", default 5-fold CV).

The cost: on all 201 columns, picking 4 means 201 + 200 + 199 + 198 = 798 candidate subsets × 3 folds = 2,394 model fits. The practical recipe is to filter to a shortlist first, then run the wrapper on the shortlist:

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import SequentialFeatureSelector, mutual_info_classif
mi = pd.Series(mutual_info_classif(X[tr], y[tr], random_state=0), X.columns)
short = mi.nlargest(20).index; print("shortlist of 20 by MI:", list(short[:7]), "...")   # cheap filter first
sfs = SequentialFeatureSelector(knn(), n_features_to_select=4, direction="forward", cv=3)   # then the wrapper
cols = sfs.fit(X[short][tr], y[tr]).get_feature_names_out(); print("forward selection picked:", list(cols))
print("models trained:", 3 * (20 + 19 + 18 + 17), "  on all 201 columns it would be", 3 * (201 + 200 + 199 + 198))
m = knn().fit(X[cols][tr], y[tr]); print("15-NN test:", round(m.score(X[cols][te], y[te]), 3))

Output:

shortlist of 20 by MI: ['distance_km', 'distance_m', 'delivery_fee', 'hour', 'raining', 'prep_min', 'log_073'] ...
forward selection picked: ['distance_km', 'hour', 'raining', 'prep_min']
models trained: 222   on all 201 columns it would be 2394
15-NN test: 0.855

The shortlist still contains all three copies of distance, but forward selection picked exactly the four columns that matter (test 0.855) in 222 fits.

Recursive feature elimination (RFE)

Recursive feature elimination is a cheaper backward idea. Fit the model once on all columns, read how important each column was, drop the weakest, refit on the rest, and repeat until the number you want remains. The importance comes from the model itself: coef_ (the weights of a linear model) or feature_importances_ (tree models). step=0.2 drops 20% of the starting columns (39 of 196) per round.

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import RFE; from sklearn.ensemble import RandomForestClassifier
clean = X.drop(columns=["app_version", "city_code", "delivery_fee", "total_bill", "distance_m"])   # after the filters
forest = RandomForestClassifier(50, random_state=0)                    # 50 trees, episode 34
rfe = RFE(forest, n_features_to_select=4, step=0.2).fit(clean[tr], y[tr])   # drop the weakest 20%, refit, repeat
cols = rfe.get_feature_names_out(); print("RFE kept:", list(cols))
print("columns left after each round:", [196 - 39 * k for k in range(5)], "-> 4")
m = knn().fit(clean[cols][tr], y[tr]); print("15-NN test:", round(m.score(clean[cols][te], y[te]), 3))

Output:

RFE kept: ['distance_km', 'hour', 'raining', 'prep_min']
columns left after each round: [196, 157, 118, 79, 40] -> 4
15-NN test: 0.855

196 → 157 → 118 → 79 → 40 → 4 columns in six fits, and again exactly the four real columns. One warning: RFE is only as good as the importance it reads. Impurity importance in random forests favours columns with many distinct values (Episode 34), so a two-valued column like raining can be ranked low; always check the result with a validation score. RFECV (Practice 2) lets cross-validation choose how many columns to keep.

Embedded methods: L1 and SelectFromModel

Embedded methods select during one ordinary training run.

  • L1 penalty (lasso, Episode 21): pushes the weights of useless columns to exactly 0; a zero weight means the column is out. In scikit-learn 1.9, LogisticRegression(l1_ratio=1, solver="liblinear") is a pure L1 penalty (the old penalty="l1" argument is deprecated since 1.8).
  • Tree models: a column never used for a split has zero importance.
  • SelectFromModel(model, threshold=...) turns either into a selector: it fits the model, reads coef_ or feature_importances_, and keeps columns at or above the threshold. threshold="5*mean" keeps columns with at least five times the average importance ("mean", "median" and factors like "1.25*mean" are all allowed).
import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import SelectFromModel; from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
l1 = make_pipeline(StandardScaler(), LogisticRegression(l1_ratio=1, C=0.02, solver="liblinear")).fit(X[tr], y[tr])
w = pd.Series(l1[-1].coef_[0], X.columns); kept = w[w != 0]             # L1 pushes useless weights to exactly 0
print(f"L1 kept {len(kept)} of {len(w)}:", kept.round(2).to_dict())
sfm = SelectFromModel(RandomForestClassifier(100, random_state=0), threshold="5*mean").fit(X[tr], y[tr])
cols = sfm.get_feature_names_out(); print("forest kept:", list(cols))
m = knn().fit(X[cols][tr], y[tr]); print("15-NN test:", round(m.score(X[cols][te], y[te]), 3))

Output:

L1 kept 5 of 201: {'distance_km': 1.16, 'hour': 0.07, 'raining': 0.63, 'prep_min': 0.79, 'log_091': 0.0}
forest kept: ['distance_km', 'hour', 'raining', 'prep_min', 'delivery_fee', 'distance_m']
15-NN test: 0.861

L1 kept 5 of 201 columns in one fit (the fifth, log_091, has a weight so small it rounds to 0.0). Notice the hour's weight is tiny: a linear model cannot use a hill either, so an embedded method inherits its model's blind spots (Practice 3 fixes this with sine and cosine). The forest kept the four real columns plus two copies of distance, and the 15-NN model reached 0.861.

Method Columns kept 15-NN test
All 201 columns 201 0.691
Filter: MI top 4 3 copies of distance + hour 0.753
Wrapper: forward selection (shortlist of 20) the 4 real ones 0.855
Wrapper: RFE (forest, after the filters) the 4 real ones 0.855
Embedded: SelectFromModel (forest) 4 real + 2 copies of distance 0.861
The 4 true columns (known only because the data is made up) 4 0.855

The variance inflation factor (VIF)

Correlation only checks pairs. But total_bill = food_bill + delivery_fee: no single pair shows it perfectly, yet together the other two rebuild it exactly. When columns can be predicted from other columns, that is multicollinearity (Episode 19). It makes a linear model's weights unstable and hard to interpret.

The test: predict each column from all the other columns with linear regression and look at R² (0 = not at all, 1 = perfectly). The variance inflation factor is

VIF = 1 / (1 − R²)

It says how much the uncertainty (variance) in that column's weight is inflated by its look-alikes. R² = 0 gives VIF 1 (no problem), R² = 0.9 gives 10, R² = 1 gives infinity. The statsmodels documentation suggests that a VIF above 5 means the column is highly collinear (statsmodels.stats.outliers_influence.variance_inflation_factor computes it; here we build it from scratch).

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.linear_model import LinearRegression
def vif(D):                                    # VIF of a column = 1 / (1 - R^2 of predicting it from the others)
    out = {}
    for c in D.columns:
        rest = D.drop(columns=c); r2 = LinearRegression().fit(rest, D[c]).score(rest, D[c])
        out[c] = np.inf if r2 > 0.999999 else 1 / (1 - r2)
    return pd.Series(out).round(1).to_dict()
money = X[["distance_km", "delivery_fee", "food_bill", "total_bill", "prep_min"]][tr]
print("all five:          ", vif(money))
print("without total_bill:", vif(money.drop(columns="total_bill")))
print("and without fee:   ", vif(money.drop(columns=["total_bill", "delivery_fee"])))

Output:

all five:           {'distance_km': 25.4, 'delivery_fee': inf, 'food_bill': inf, 'total_bill': inf, 'prep_min': 1.0}
without total_bill: {'distance_km': 25.4, 'delivery_fee': 25.5, 'food_bill': 1.0, 'prep_min': 1.0}
and without fee:    {'distance_km': 1.0, 'food_bill': 1.0, 'prep_min': 1.0}
  • delivery_fee, food_bill and total_bill are infinite: each is rebuilt exactly from the other two.
  • Dropping total_bill frees food_bill (VIF 1), but distance_km and delivery_fee are still about 25, because the fee is set from the distance.
  • Dropping the fee too brings everything to 1. Drop one column at a time and recompute (Practice 4).

The leakage trap: selection before cross-validation

Imagine 5,000 quiz questions whose answers are coin flips, and 200 students. If you look at the answer key and keep the 20 questions that happen to match it best, using every student, those questions will look predictive on the same students. Selecting features on all rows before cross-validation is exactly that:

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif; from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline; from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(0); X = rng.normal(size=(200, 5000)); y = rng.integers(0, 2, 200)   # PURE noise
picked = SelectKBest(f_classif, k=20).fit_transform(X, y)              # MISTAKE: choose columns using ALL rows
print("select first, then CV: ", cross_val_score(LogisticRegression(), picked, y, cv=5).mean().round(3))
safe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())   # choose again inside every fold
print("select inside each fold:", cross_val_score(safe, X, y, cv=5).mean().round(3))

Output:

select first, then CV:  0.795
select inside each fold: 0.51

On pure noise (true accuracy 50%), selecting first and then cross-validating reports 0.795. Selecting inside each fold (a pipeline) gives the honest 0.51. Feature selection learns from the labels, so it is part of training and belongs inside the pipeline (Episode 17).

The final leak-proof pipeline

import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import VarianceThreshold, SelectFromModel
from sklearn.ensemble import RandomForestClassifier; from sklearn.model_selection import cross_val_score
model = make_pipeline(VarianceThreshold(), SelectFromModel(RandomForestClassifier(50, random_state=0),
                      threshold="5*mean"), StandardScaler(), KNeighborsClassifier(15))   # select INSIDE the pipeline
print("3-fold CV on train:", cross_val_score(model, X[tr], y[tr], cv=3).mean().round(3))
model.fit(X[tr], y[tr]); print("kept:", list(model[1].get_feature_names_out(model[0].get_feature_names_out())))
print("test:", round(model.score(X[te], y[te]), 3))

Output:

3-fold CV on train: 0.841
kept: ['distance_km', 'hour', 'raining', 'prep_min', 'delivery_fee', 'distance_m']
test: 0.861

Variance threshold → SelectFromModel (forest) → scaling → kNN, all inside one pipeline: 3-fold CV on train 0.841, test 0.861 (from 0.691 with every column). The two extra copies of distance would also go with a correlation filter or a VIF check.

Common mistakes

Mistake Exact message / symptom Why Fix
Feed every logged column test 0.691 vs 0.855 Noise drowns the signal Select features
Rank only by correlation hour: |r| = 0.092 Correlation only sees straight lines Mutual information
chi2 on negative numbers ValueError: Input X must be non-negative. Chi-square is for counts MI, or bin first
Top-k filter with twin columns 3 copies of distance: 0.753 Filters score columns alone Correlation filter first, or a wrapper
Wrapper on hundreds of columns 2,394 fits for 4 columns Every step tries every column Shortlist with a filter first
Trusting RFE's ranking blindly binary columns ranked low Impurity importance favours many-valued columns Check with a validation score
Select on all rows, then cross-validate pure noise scores 0.795 Labels leak into the selection Select inside a pipeline
penalty="l1" in scikit-learn 1.9 FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10 API change l1_ratio=1

Mini project: a selection report for any table

Write a function select_report(X, y) that, on the training rows only: (1) lists constant columns, (2) lists twin pairs with |correlation| > 0.95, (3) prints the top 10 columns by mutual information, (4) runs forward selection on that top 10 for 4 columns, and (5) compares the cross-validated score of all columns vs the selected ones inside a pipeline. Run it on Kiran's data; every step above is in this page's code, so you only need to combine them.

Practice

1. Run backward selection (direction="backward") and compare it with forward selection.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import SequentialFeatureSelector, mutual_info_classif
mi = pd.Series(mutual_info_classif(X[tr], y[tr], random_state=0), X.columns)
short = list(mi.nlargest(10).index)                                    # backward selection is costly: start from 10
for direction in ["forward", "backward"]:
    sfs = SequentialFeatureSelector(knn(), n_features_to_select=4, direction=direction, cv=3).fit(X[short][tr], y[tr])
    cols = sfs.get_feature_names_out(); m = knn().fit(X[cols][tr], y[tr])
    print(f"{direction:<8} picked {list(cols)}  test {m.score(X[cols][te], y[te]):.3f}")

Output:

forward  picked ['distance_km', 'hour', 'raining', 'prep_min']  test 0.855
backward picked ['distance_m', 'hour', 'raining', 'prep_min']  test 0.854

Backward selection is costlier (it starts from 10 columns and removes one at a time), and it kept distance_m instead of distance_km: the same information, so the score is almost the same.

2. Use RFECV to let cross-validation choose how many columns to keep.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.feature_selection import RFECV; from sklearn.ensemble import RandomForestClassifier
clean = X.drop(columns=["app_version", "city_code", "delivery_fee", "total_bill", "distance_m"])
rfecv = RFECV(RandomForestClassifier(50, random_state=0), step=0.2, cv=3, min_features_to_select=2).fit(clean[tr], y[tr])
cols = rfecv.get_feature_names_out(); print("RFECV chose", len(cols), "columns:", list(cols)[:8])
m = knn().fit(clean[cols][tr], y[tr]); print("15-NN test:", round(m.score(clean[cols][te], y[te]), 3))

Output:

RFECV chose 40 columns: ['distance_km', 'hour', 'raining', 'prep_min', 'log_005', 'log_014', 'log_032', 'log_040']
15-NN test: 0.739

RFECV kept 40 columns: the forest's cross-validated score barely changes between 4 and 40 columns, because forests ignore noise well. But the kNN model that uses those columns suffers (0.739). The lesson: select with the model you will actually use, or check the selection with that model.

3. Give L1 the hour as sine and cosine (Episode 39) and redo the L1 selection.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.linear_model import LogisticRegression
X2 = X.assign(hour_sin=np.sin(2 * np.pi * X.hour / 24), hour_cos=np.cos(2 * np.pi * X.hour / 24))   # episode 39
l1 = make_pipeline(StandardScaler(), LogisticRegression(l1_ratio=1, C=0.02, solver="liblinear")).fit(X2[tr], y[tr])
w = pd.Series(l1[-1].coef_[0], X2.columns); kept = w[w != 0]
print(f"L1 kept {len(kept)} of {len(w)}:", kept.round(2).to_dict())
print("test:", round(l1.score(X2[te], y[te]), 3))

Output:

L1 kept 4 of 203: {'distance_km': 1.43, 'raining': 0.78, 'prep_min': 0.96, 'hour_cos': -1.16}
test: 0.872

Now the linear model can see the lunch hill through hour_cos, L1 keeps exactly 4 columns, and plain logistic regression on all of them scores 0.872 on test.

4. Write a loop that drops the column with the highest VIF until every VIF is below 5.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(40); n = 3000                            # Kiran's app: 3,000 made-up orders
X = pd.DataFrame({"distance_km": rng.uniform(.5, 9, n), "hour": rng.integers(0, 24, n),
                  "raining": rng.integers(0, 2, n), "items": rng.integers(1, 8, n),
                  "prep_min": rng.normal(15, 5, n).round(1), "food_bill": rng.gamma(3, 120, n).round()})
X["delivery_fee"] = (20 + 6 * X.distance_km + rng.normal(0, 3, n)).round()
X["total_bill"] = X.food_bill + X.delivery_fee                         # the sum of two other columns
X["distance_m"] = (X.distance_km * 1000 + rng.normal(0, 30, n)).round()  # the same distance, in metres
X["app_version"] = 5.2; X["city_code"] = 1                             # never change in this city
X = X.join(pd.DataFrame(rng.normal(size=(n, 190)), columns=[f"log_{i:03d}" for i in range(1, 191)]))  # 190 noise logs
s = .45 * X.distance_km + 1.3 * np.cos((X.hour - 12) * np.pi / 12) + 1.2 * X.raining + .15 * X.prep_min
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int)              # 1 = late
tr, te = slice(0, 2000), slice(2000, None)                             # 2,000 train · 1,000 test
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(15))  # episode 27: 15 nearest orders vote
from sklearn.linear_model import LinearRegression
def vif(D):
    out = {}
    for c in D.columns:
        rest = D.drop(columns=c); r2 = LinearRegression().fit(rest, D[c]).score(rest, D[c])
        out[c] = np.inf if r2 > 0.999999 else 1 / (1 - r2)
    return pd.Series(out)
D = X[["distance_km", "delivery_fee", "distance_m", "food_bill", "total_bill", "prep_min", "items"]][tr]
while True:
    v = vif(D)
    if v.max() < 5: break
    print("drop", v.idxmax(), "VIF", round(v.max(), 1)); D = D.drop(columns=v.idxmax())
print("kept:", vif(D).round(2).to_dict())

Output:

drop delivery_fee VIF inf
drop distance_km VIF 7257.5
drop total_bill VIF 4580.3
kept: {'distance_m': 1.0, 'food_bill': 1.0, 'prep_min': 1.0, 'items': 1.0}

Which twin survives depends on the order the loop meets them; here it kept distance_m and food_bill. Both choices carry the same information.

Quick quiz

Predicting delivery_fee from the other columns gives R² = 0.96. What is its VIF? Answer: 25, because 1 / (1 − 0.96) = 1 / 0.04 = 25. Well above 5, so delivery_fee is highly collinear (it is set from the distance).

Interview questions

  1. What are filter, wrapper and embedded methods? Give one example of each. Filter: score each column alone (mutual information, chi-square, variance threshold). Wrapper: search subsets with the real model (forward selection, RFE). Embedded: selection inside training (L1, tree importance).
  2. Why can adding features make a model worse? Noise columns dilute distances (kNN), give more chances to overfit, and slow everything down.
  3. When would you use mutual information instead of correlation? When the relationship may be non-linear (a hill, a step); correlation only measures straight-line relationships.
  4. What does chi-square feature selection require? Non-negative values (counts or 0/1) and a class target.
  5. Why is forward selection called greedy, and what does it cost? It adds the best column at each step and never reconsiders; picking k of p columns costs about k·p model fits per fold.
  6. How does RFE work? Fit, rank columns by coef_ or feature_importances_, drop the weakest, refit, repeat.
  7. What is VIF and what value is a concern? 1 / (1 − R²) of predicting a column from the others; above 5 (statsmodels) or 10 (another common rule of thumb) signals multicollinearity.
  8. Why must feature selection happen inside cross-validation? It uses the labels; selecting on all rows leaks validation labels and inflates scores (noise scored 0.795 instead of 0.51).
  9. Is L1 selection reliable with correlated columns? It tends to keep one of a correlated group and zero the rest, somewhat arbitrarily; the choice can flip between samples.
  10. Feature selection vs dimensionality reduction (PCA)? Selection keeps original columns (interpretable); PCA (Episode 5's SVD, later in the course) builds new combined columns.

Glossary

Term Meaning, with an example
Feature selection Keeping a useful subset of existing columns: 4 of Kiran's 201
Feature engineering Creating new columns from existing ones (Episodes 38-39)
Filter method Scores each column on its own, without the final model: mutual information
Wrapper method Searches subsets of columns using the real model's validation score: forward selection
Embedded method Selection happens inside one training run: L1 zeroes weights
Variance How much a column's values spread out; 0 means constant
Variance threshold Drops columns whose variance is at or below a threshold: app_version
Constant column Same value in every row; useless for prediction
Correlation filter Drops one column of each pair with |correlation| above a cut-off (0.95 here)
Twin (redundant) columns Columns carrying the same information: distance_km and distance_m
Mutual information (MI) How much knowing a column reduces uncertainty about the target; 0 = independent; sees any shape
Independent Knowing one tells you nothing about the other
Chi-square (χ²) test Compares observed counts with the counts expected if a column and the class were unrelated
Expected vs observed counts What independence predicts vs what actually happened
p-value Chance of a result at least this extreme if the column were truly unrelated
SelectKBest Keeps the k columns with the highest filter scores
Forward selection Start empty, add the column that helps most, repeat
Backward selection Start with all, remove the column that helps least, repeat
Greedy algorithm Takes the best step now and never undoes it
SequentialFeatureSelector scikit-learn's forward/backward selection by cross-validation score
Shortlist A small set of candidates from a cheap filter, before an expensive wrapper
Recursive feature elimination (RFE) Fit, drop the least important columns, refit, repeat
step (RFE) How many columns to drop per round (a fraction like 0.2 = 20% of the start)
RFECV RFE that chooses the number of columns by cross-validation
L1 penalty (lasso) Adds the sum of absolute weights to the loss; pushes useless weights to exactly 0
SelectFromModel Keeps columns whose model importance is at or above a threshold like "5*mean"
Impurity importance How much a column's splits reduce impurity in a tree model (Episode 34)
Multicollinearity A column can be predicted from other columns
R² Share of a column's variation explained by a regression: 0 none, 1 perfect
Variance inflation factor (VIF) 1 / (1 − R²); above 5 = highly collinear (statsmodels)
Selection leakage Choosing features using rows that are later used to validate
Pipeline Chain of steps fitted together, refitted inside every CV fold

Summary

  • More columns can hurt: 201 columns gave 0.691, the right 4 gave 0.855.
  • Filters are fast first passes: drop constant columns (variance threshold) and twins (correlation filter); rank with mutual information (any shape) or chi-square (counts only).
  • Filters score columns alone, so they pick twins; wrappers (forward selection, RFE) judge columns together with the real model, at a higher cost, so shortlist first.
  • Embedded methods select in one fit (L1, SelectFromModel with trees) but inherit their model's blind spots.
  • VIF = 1 / (1 − R²) finds columns hidden in combinations of others; drop one at a time.
  • Selection learns from labels: always do it inside the pipeline, refitted in every fold.

Next episode: imbalanced data, when the thing you care about is only 2 rows in 100.