Machine Learning Full Course · Episode 30. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The kirana customers and chai sales are made up for teaching; the mini project uses the diabetes dataset bundled with scikit-learn.
The big idea: an unlimited tree memorises noise. Stop it early with pre-pruning (max_depth, min_samples_split, min_samples_leaf) or grow it fully and cut it back with post-pruning (cost-complexity pruning, ccp_alpha), choosing the strength by cross-validation. A pruned tree can be drawn and read. Its feature importances are easy to get but biased towards features with many unique values. Trees are unstable (high variance), which ensembles fix later. Regression trees split by variance reduction and predict the mean of a leaf, so their predictions are steps that cannot extrapolate.
Words you need first
- Decision tree, root, node, leaf, split, depth, Gini impurity, CART: Episode 29.
- Overfitting / underfitting; variance: memorising noise / missing the pattern; how much a model changes with different training data (Episode 20).
- Cross-validation (CV): scoring on several train/validation splits and averaging (Episode 26).
- Hyperparameter: a setting we choose, not learned (Episode 8).
- Regression: predicting a number (Episode 1). Variance of numbers: average squared distance from their mean (Episode 10). MSE / MAE: mean squared / mean absolute error (Episodes 18-19).
The problem: the tree that never stops asking
Same 2,000 made-up customers as Episode 29 (income in thousand rupees, years Raju has known them, repaid or not). Unlimited, the tree grows 284 leaves, some holding a single customer: a memory, not a pattern.
Pre-pruning
- Pruning: reducing a tree's size to stop overfitting.
- Pre-pruning (early stopping): stop growing before the tree is complete.
max_depth: maximum number of questions on any root-to-leaf path.min_samples_split: a node may split only if it holds at least this many training rows.min_samples_leaf: every leaf must keep at least this many rows (a split that would create a smaller leaf is not allowed).- Others:
max_leaf_nodes(cap on leaves),min_impurity_decrease(a split must reduce weighted impurity by at least this much).
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
def report(name, t): # pre-pruning: stop the tree growing early
tr, te = t.score(X_tr, y_tr), t.score(X_te, y_te)
print(f"{name:<22} leaves {t.get_n_leaves():>3} train {tr:.3f} test {te:.3f}")
report("no limits", DT(random_state=0).fit(X_tr, y_tr))
report("max_depth=3", DT(max_depth=3, random_state=0).fit(X_tr, y_tr))
report("min_samples_split=100", DT(min_samples_split=100, random_state=0).fit(X_tr, y_tr))
report("min_samples_leaf=50", DT(min_samples_leaf=50, random_state=0).fit(X_tr, y_tr))
Output:
no limits leaves 284 train 0.917 test 0.818
max_depth=3 leaves 8 train 0.903 test 0.880
min_samples_split=100 leaves 30 train 0.903 test 0.880
min_samples_leaf=50 leaves 23 train 0.903 test 0.880
All three brakes lift test accuracy from 0.818 to 0.880 with far fewer leaves. min_samples_leaf is often the most useful: a leaf's vote is only trustworthy with enough rows. Pre-pruning's risk: a split that looks useless may enable a great split below it (e.g. XOR-like patterns), and early stopping can't know that.
Post-pruning: cost-complexity pruning
Real-life example: a mali (gardener) lets the hedge grow, then trims weak branches. To decide which, think of Raju's shelves: every product pays shelf rent and stays only if it earns more than the rent.
- Post-pruning: grow the full tree, then remove branches that don't pay for themselves.
- Cost-complexity pruning (CCP): minimise
R_α(T) = R(T) + α × |leaves(T)|where R(T) is the tree's total training impurity (weighted by samples) and α is the "rent" per leaf. - α = 0 keeps the full tree; growing α removes the weakest link first (the branch whose impurity reduction per extra leaf is smallest); a huge α leaves a single leaf (the majority class).
cost_complexity_pruning_path(X, y)returns every α at which a branch is cut (ccp_alphas) and the matching impurities.ccp_alphais a hyperparameter: pick it with cross-validation.
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import cross_val_score
path = DT(random_state=0).cost_complexity_pruning_path(X_tr, y_tr) # grow fully, then prune
alphas = path.ccp_alphas
print(len(alphas), "pruning levels, alpha from", alphas[0], "to", alphas[-1].round(3))
for a in [0.0, 0.0005, 0.001, 0.002, 0.005, 0.05, 0.15]:
t = DT(ccp_alpha=a, random_state=0)
cv = cross_val_score(t, X_tr, y_tr, cv=5).mean(); t.fit(X_tr, y_tr)
print(f"ccp_alpha {a:<6} leaves {t.get_n_leaves():>3} CV {cv:.3f} test {t.score(X_te, y_te):.3f}")
Output:
119 pruning levels, alpha from 0.0 to 0.206
ccp_alpha 0.0 leaves 284 CV 0.827 test 0.818
ccp_alpha 0.0005 leaves 25 CV 0.878 test 0.876
ccp_alpha 0.001 leaves 8 CV 0.894 test 0.882
ccp_alpha 0.002 leaves 4 CV 0.901 test 0.880
ccp_alpha 0.005 leaves 4 CV 0.902 test 0.880
ccp_alpha 0.05 leaves 3 CV 0.855 test 0.856
ccp_alpha 0.15 leaves 2 CV 0.837 test 0.822
- 119 pruning levels between α = 0 and 0.206.
- α = 0.0005 already cuts 284 leaves to 25; α ≈ 0.002 gives 4 leaves, CV 0.901, test 0.880.
- α = 0.15 leaves just 2 leaves (one question): underfitting.
Pre- vs post-pruning
| Pre-pruning | Post-pruning (CCP) | |
|---|---|---|
| When | while growing | after growing fully |
| Knobs | max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes |
ccp_alpha |
| Speed | fast (never grows the big tree) | slower (grows it, then prunes) |
| Risk | may stop before a useful deeper split | none of that; needs CV for α |
Reading a tree: visualisation
from sklearn.tree import plot_tree, export_text
t = DT(ccp_alpha=0.002, random_state=0).fit(X_tr, y_tr)
plot_tree(t, feature_names=["income", "years"], class_names=["not repaid", "repaid"], filled=True, rounded=True)
print(export_text(t, feature_names=["income", "years"]))

Each box shows: the question; gini (impurity of the node); samples (training rows reaching it); value = [not repaid, repaid] counts; class = the majority. Left = True (yes), right = False. Colour: orange = mostly not repaid, blue = mostly repaid, darker = purer.
Read as rules: income ≤ 29.5 → no udhaar; above that, known ≥ 2 years → udhaar; known < 2 years → only if income > 59.5. This is exactly the hidden rule used to make the data, and the useless split from Episode 29 is gone. A small tree is one of the few models you can explain line by line to a customer or a regulator.
Feature importance
- Impurity-based feature importance (
feature_importances_): for each feature, the total weighted impurity decrease from all splits on it, normalised to sum to 1. Also called mean decrease in impurity (MDI).
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
phone = rng.integers(0, 10**6, n) # last 6 digits of phone: pure noise
X3 = np.c_[X, phone]; names = ["income", "years", "phone"]
models = [("no limits", DT(random_state=0)), ("ccp_alpha=0.002", DT(ccp_alpha=0.002, random_state=0))]
for name, t in models:
t.fit(X3[:1500], y_tr)
imp = ", ".join(f"{f} {v:.2f}" for f, v in zip(names, t.feature_importances_))
print(f"{name:<16} test {t.score(X3[1500:], y_te):.3f}")
print(f" importance: {imp}")
Output:
no limits test 0.790
importance: income 0.60, years 0.17, phone 0.23
ccp_alpha=0.002 test 0.880
importance: income 0.82, years 0.18, phone 0.00
- The phone digits are pure noise, yet the full tree gives them 0.23, more than years known (0.17), and test accuracy drops to 0.790.
- Why: a feature with many unique values offers many thresholds; deep in the tree, with a handful of rows left, some threshold always happens to separate them. Pure luck, counted as importance. And the importance is computed on training data, so memorised noise gets credit.
- The pruned tree gives phone 0.00. Prune first; for a reliable check use permutation importance on held-out data (Episode 34).
Why trees are unstable
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import export_text
for name, kw in [("no limits", {}), ("max_depth=3", {"max_depth": 3})]:
preds = []
for seed in [1, 2]: # two shops, slightly different 80% of data
i = np.random.default_rng(seed).choice(1500, 1200, replace=False)
t = DT(random_state=0, **kw).fit(X_tr[i], y_tr[i]); preds.append(t.predict(X_te))
print(f"{name:<11} sample {seed}: leaves {t.get_n_leaves():>3} root:",
export_text(t, feature_names=["income", "years"]).splitlines()[0])
print(f"{name:<11} the two trees disagree on {np.mean(preds[0] != preds[1]):.1%} of test customers")
Output:
no limits sample 1: leaves 231 root: |--- income <= 29.50
no limits sample 2: leaves 247 root: |--- income <= 29.50
no limits the two trees disagree on 6.6% of test customers
max_depth=3 sample 1: leaves 8 root: |--- income <= 29.50
max_depth=3 sample 2: leaves 8 root: |--- income <= 29.50
max_depth=3 the two trees disagree on 0.0% of test customers
- Two trees trained on two random 80% samples of the same data: full trees (231 and 247 leaves) disagree on 6.6% of new customers; depth-3 trees agree on all of them.
- Instability (high variance): a small change in the training data changes the tree, because each split decides what every node below it sees.
- Fixes: prune or limit depth (less variance, maybe more bias), or average many trees trained on different samples: bagging and random forests (Episodes 33-34).
Regression trees
Real-life example: Raju's chai stall: 300 made-up days, temperature vs cups sold. Hot days sell fewer cups. Gini needs classes; for numbers, a "messy" group is one whose values are spread out.
- Regression tree: a tree whose leaves predict numbers (
DecisionTreeRegressor). - Variance reduction: split quality = variance of the parent − size-weighted variance of the children. scikit-learn calls this criterion
"squared_error"(variance is the MSE around the mean)."absolute_error"(MAE around the median) and"poisson"also exist. - Leaf prediction: the mean of the leaf's training targets.
import numpy as np
from sklearn.tree import DecisionTreeRegressor, export_text
rng = np.random.default_rng(30) # 300 made-up days at Raju's chai stall
temp = rng.uniform(10, 45, 300).round(1) # temperature, degrees C
cups = (260 - 4 * temp + rng.normal(0, 15, 300)).round() # cups of chai sold
def var_drop(t): # variance reduction of "temp <= t"
m = temp <= t
return cups.var() - (m.mean() * cups[m].var() + (~m).mean() * cups[~m].var())
for t in [15, 27.5, 35]:
print(f"temp <= {t:<4}: variance {cups.var():.0f} -> drop {var_drop(t):.0f}")
tree = DecisionTreeRegressor(max_depth=2).fit(temp.reshape(-1, 1), cups)
print(export_text(tree, feature_names=["temp"], decimals=1).strip())
temps = [[5], [12], [22], [32], [42], [50]] # 5 and 50 C: outside the data
print("5, 12, 22, 32, 42, 50 C ->", tree.predict(temps).round(1), "cups")
Output:
temp <= 15 : variance 2003 -> drop 604
temp <= 27.5: variance 2003 -> drop 1310
temp <= 35 : variance 2003 -> drop 1147
|--- temp <= 27.2
| |--- temp <= 18.9
| | |--- value: [204.4]
| |--- temp > 18.9
| | |--- value: [167.4]
|--- temp > 27.2
| |--- temp <= 35.8
| | |--- value: [136.6]
| |--- temp > 35.8
| | |--- value: [93.5]
5, 12, 22, 32, 42, 50 C -> [204.4 204.4 167.4 136.6 93.5 93.5] cups
- The variance of cups is 2,003. Splitting at 27.5 °C removes 1,310 of it, more than at 15 or 35; the tree picks 27.2 °C.
- Leaves predict 204.4, 167.4, 136.6 and 93.5 cups: a staircase.
- 5 °C predicts the same as 12 °C, and 50 °C the same as 42 °C: a tree cannot extrapolate beyond the range of its training data. For a trend that continues, a linear model extrapolates; a tree just repeats the edge leaf.

Depth 2: 4 flat steps. No limit: 196 leaves, a jagged line through every point (overfitting). Prune regression trees with the same knobs.
Decision trees at a glance
| Strengths | readable rules, no scaling, handles mixed features and interactions, fast prediction, captures non-linear patterns |
| Weaknesses | overfits without limits, unstable (high variance), axis-parallel boundaries, no extrapolation, importance biased to many-valued features |
| Key hyperparameters | max_depth, min_samples_leaf, min_samples_split, ccp_alpha, criterion |
| Classifier / regressor | DecisionTreeClassifier (Gini/entropy, majority leaf) / DecisionTreeRegressor (squared error, mean leaf) |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| No limits on the tree | 284 leaves, test 0.818 | Memorises noise | Pre- or post-prune |
Guessing ccp_alpha |
α = 0.15: CV 0.837 | Too much rent: underfits | Pruning path + CV |
| Trusting a full tree's importances | phone 0.23 > years 0.17 | Many values = lucky splits; training data | Prune; permutation importance |
| Betting on one deep tree | Two trees disagree on 6.6% | High variance | Prune, or ensembles |
| Extrapolating with a tree | 50 °C → 93.5, same as 42 °C | Leaf = mean of seen rows | Linear model for trends |
Passing a 1-D array to fit |
ValueError: Expected 2D array, got 1D array instead: |
scikit-learn wants (rows, features) | temp.reshape(-1, 1) |
Negative ccp_alpha |
InvalidParameterError: The 'ccp_alpha' parameter of DecisionTreeClassifier must be a float in the range [0.0, inf). Got -1 instead. |
α is a rent ≥ 0 | Use values from the pruning path |
The 1-D mistake, reproduced:
import numpy as np
from sklearn.tree import DecisionTreeRegressor
temp = np.array([12.0, 22.0, 32.0, 42.0]) # 1-D: shape (4,)
cups = np.array([205, 167, 137, 94])
DecisionTreeRegressor().fit(temp, cups)
Output:
Traceback (most recent call last):
... (library frames) ...
ValueError: Expected 2D array, got 1D array instead:
array=[12. 22. 32. 42.].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Mini project: a regression tree on real data
The diabetes dataset (bundled with scikit-learn): 10 numeric measurements (age, sex, body mass index, blood pressure and six blood serum measurements) for 442 patients; the target is a quantitative measure of disease progression one year after baseline.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error as mae
data = load_diabetes() # bundled with scikit-learn
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.25, random_state=0)
print(X_tr.shape, "target: disease progression one year after baseline")
full = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
grid = GridSearchCV(DecisionTreeRegressor(random_state=0), {"max_depth": [2, 3, 4, 5, 6],
"min_samples_leaf": [5, 10, 20, 40]}, cv=5, scoring="neg_mean_absolute_error").fit(X_tr, y_tr)
lin = LinearRegression().fit(X_tr, y_tr)
print(f"full tree leaves {full.get_n_leaves():>3} test MAE {mae(y_te, full.predict(X_te)):.1f}")
print(f"tuned tree leaves {grid.best_estimator_.get_n_leaves():>3} test MAE {mae(y_te, grid.predict(X_te)):.1f} {grid.best_params_}")
print(f"linear regression test MAE {mae(y_te, lin.predict(X_te)):.1f}")
top = np.argsort(grid.best_estimator_.feature_importances_)[::-1][:3]
print("top features of the tuned tree:", [data.feature_names[i] for i in top])
Output:
(331, 10) target: disease progression one year after baseline
full tree leaves 318 test MAE 60.4
tuned tree leaves 8 test MAE 53.5 {'max_depth': 3, 'min_samples_leaf': 10}
linear regression test MAE 45.1
top features of the tuned tree: ['s5', 'bmi', 'bp']
- The full tree (318 leaves) has the worst error; tuning
max_depthandmin_samples_leafby CV helps (8 leaves). - Plain linear regression still wins here: this target changes smoothly with the features, and a single tree can only make steps. Trees shine as parts of ensembles (Episodes 33-36).
- The tuned tree's top features are
s5(a serum measurement),bmi(body mass index) andbp(average blood pressure).
Practice
1. Grid-search max_depth × min_samples_leaf with 5-fold CV.
Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import GridSearchCV
grid = {"max_depth": [2, 3, 4, 6, 10, None], "min_samples_leaf": [1, 5, 20, 50, 100]}
search = GridSearchCV(DT(random_state=0), grid, cv=5).fit(X_tr, y_tr)
print("best settings:", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"test accuracy {search.score(X_te, y_te):.3f} leaves {search.best_estimator_.get_n_leaves()}")
Output:
best settings: {'max_depth': 3, 'min_samples_leaf': 1} CV 0.902
test accuracy 0.880 leaves 8
2. Plot the CV score against ccp_alpha for every level of the pruning path.
Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import cross_val_score
import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
alphas = DT(random_state=0).cost_complexity_pruning_path(X_tr, y_tr).ccp_alphas
cv = [cross_val_score(DT(ccp_alpha=a, random_state=0), X_tr, y_tr, cv=5).mean() for a in alphas]
best = alphas[int(np.argmax(cv))]
print(f"{len(alphas)} alphas tried; best ccp_alpha {best:.5f}, CV {max(cv):.3f}")
plt.semilogx(alphas[1:], cv[1:], marker="o"); plt.xlabel("ccp_alpha"); plt.ylabel("5-fold CV accuracy")
plt.savefig("ccp_curve.png", dpi=90)
Output:
119 alphas tried; best ccp_alpha 0.00126, CV 0.899
The curve rises steeply from α = 0, stays flat around 0.0005-0.005, then drops as α removes useful splits. Any α on the plateau is fine (in the episode, α = 0.005 gave CV 0.902; different α between two path levels give the same tree, so CV values cluster).
3. Add a unique customer-ID column. What importance does it get?
Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000 # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)] # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int) # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
cid = np.arange(n) # customer ID: 0, 1, 2, ... unique for everyone
X3 = np.c_[X, cid]
for name, t in [("no limits", DT(random_state=0)), ("min_samples_leaf=50", DT(min_samples_leaf=50, random_state=0))]:
t.fit(X3[:1500], y_tr)
print(f"{name:<20} importance income {t.feature_importances_[0]:.2f} years {t.feature_importances_[1]:.2f}"
f" customer_id {t.feature_importances_[2]:.2f} test {t.score(X3[1500:], y_te):.3f}")
Output:
no limits importance income 0.60 years 0.17 customer_id 0.22 test 0.824
min_samples_leaf=50 importance income 0.81 years 0.18 customer_id 0.01 test 0.880
An ID is unique for every customer, so the full tree uses it to separate the last few rows: importance 0.22 for a meaningless number. With min_samples_leaf=50 it almost disappears.
4. Prune the chai regression tree with ccp_alpha and compare cross-validated MAE.
Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(30) # the same 300 made-up chai days
temp = rng.uniform(10, 45, 300).round(1).reshape(-1, 1)
cups = (260 - 4 * temp[:, 0] + rng.normal(0, 15, 300)).round()
for a in [0, 10, 50, 200, 1000]: # alpha is in squared cups here
r = DecisionTreeRegressor(ccp_alpha=a, random_state=0)
mae = -cross_val_score(r, temp, cups, cv=5, scoring="neg_mean_absolute_error").mean()
print(f"ccp_alpha {a:<5} leaves {r.fit(temp, cups).get_n_leaves():>3} CV MAE {mae:.1f} cups")
Output:
ccp_alpha 0 leaves 196 CV MAE 16.0 cups
ccp_alpha 10 leaves 6 CV MAE 13.1 cups
ccp_alpha 50 leaves 4 CV MAE 14.5 cups
ccp_alpha 200 leaves 3 CV MAE 17.8 cups
ccp_alpha 1000 leaves 2 CV MAE 22.9 cups
For regression trees α is measured in squared target units (cups²), so useful values are much larger than for classification. About 6 leaves gives the lowest error; too much pruning underfits.
5. By hand: a regression leaf holds 120, 130, 140, 150 cups. Prediction? Its variance?
Solution
Mean = 540 / 4 = 135. Variance = ((−15)² + (−5)² + 5² + 15²) / 4 = (225 + 25 + 25 + 225) / 4 = 125.
Quick quiz
- Which knob guarantees every leaf has at least 50 rows?
min_samples_leaf=50. - Bigger
ccp_alphameans a bigger or smaller tree? Smaller. - A full tree gives a random ID column high importance. Why? Many unique values give lucky splits, measured on training data.
- What does a regression tree's leaf predict? The mean of its training targets.
- Prediction at 50 °C if training data stops at 45 °C? The same as the edge leaf (no extrapolation).
Interview questions
- How do you prevent a decision tree from overfitting? Pre-pruning (
max_depth,min_samples_leaf,min_samples_split,max_leaf_nodes) or post-pruning (cost-complexity,ccp_alpha), tuned by cross-validation; or use ensembles. - Explain cost-complexity pruning. Minimise training impurity + α × number of leaves; increasing α removes the weakest-link branches first; choose α by CV.
- Pre-pruning vs post-pruning? Pre stops early (fast, may miss deeper useful splits); post grows fully then prunes (slower, usually safer).
- How is feature importance computed in a tree, and what's its bias? Total weighted impurity decrease per feature, normalised; biased towards high-cardinality features and computed on training data. Use permutation importance on held-out data to check.
- Why are decision trees unstable? Small data changes alter early splits, which change the whole subtree: high variance. Bagging/random forests average it out.
- How does a regression tree choose splits and make predictions? Maximum variance (squared error) reduction; leaf predicts the mean.
- Can a tree extrapolate? No: predictions are always one of the leaf means seen in training.
- How do you read
valueinplot_tree? The per-class counts (or the mean, for regression) of the training rows reaching that node. - When would you prefer a single tree over a more accurate model? When the decision must be explained as rules (credit, medical triage, compliance) and a small accuracy loss is acceptable.
- Do trees need scaling? No; splits compare one feature at a time against a threshold.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Pruning | Shrinking a tree to stop overfitting | 284 → 4 leaves |
| Pre-pruning | Stopping growth early | max_depth=3 |
max_depth |
Most questions on any path | 3 |
min_samples_split |
Rows a node needs before it may split | 100 |
min_samples_leaf |
Rows every leaf must keep | 50 |
| Post-pruning | Grow fully, then cut weak branches | cost-complexity pruning |
| Cost-complexity pruning | Minimise impurity + α × leaves | ccp_alpha=0.002 |
ccp_alpha (α) |
"Rent" per leaf; bigger = smaller tree | 0.15 → 2 leaves |
| Pruning path | Every α where a branch is cut | 119 levels |
| Weakest link | The branch that gains least per extra leaf | cut first as α grows |
plot_tree / export_text |
Draw / print a fitted tree | the 4-leaf tree figure |
value |
Class counts (or mean) at a node | [578, 922] at the root |
| Feature importance (MDI) | Share of total impurity decrease from a feature | income 0.82 |
| High cardinality | Many unique values | phone digits, customer ID |
| Permutation importance | Score drop when a feature is shuffled (Episode 34) | the reliable check |
| Instability / high variance | Small data change → different model | 6.6% disagreement |
| Regression tree | Tree predicting numbers | chai cups vs temperature |
| Variance reduction | Parent variance − weighted children variance | 1,310 at 27.5 °C |
squared_error |
scikit-learn's variance-reduction criterion | default for DecisionTreeRegressor |
| Extrapolation | Predicting outside the training range | 50 °C → 93.5 (flat) |
| Step function | Prediction that jumps between flat levels | 4 steps at depth 2 |
Summary
- Stop overgrowth with pre-pruning (
max_depth,min_samples_split,min_samples_leaf) or post-pruning (ccp_alpha), tuned by CV. - A pruned tree is readable: question, gini, samples, value, class.
- Feature importances of overgrown trees credit noise; prune first, check with permutation importance.
- Trees are unstable; pruning and ensembles fix it.
- Regression trees split by variance reduction, predict leaf means in steps, and can't extrapolate.
- Next episode: support vector machines.