SyntaxPilot
Courses/Machine Learning/Lesson 36
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 36 of 90

XGBoost, LightGBM and CatBoost

gradient boosted decision trees (GBDT)XGBoostregularised objectivehistogram-based splittingLightGBMleaf-wise growthCatBoostnative categorical handlingearly stoppinglearning rate vs n_estimatorssubsample and colsample

Machine Learning Full Course · Episode 36. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas, scikit-learn 1.9.1, XGBoost 3.4.1, LightGBM 4.7.0 and CatBoost 1.2.10; the output shown is the real output (fit times will differ on your machine). Sunita's 20,000 app customers are made up for teaching; the mini project uses the breast cancer dataset bundled with scikit-learn.

The big idea: the gradient boosting we built in Episode 35 is a GBDT (gradient boosted decision trees). On real business tables it hits three problems: it is slow (every split tries every unique value), it can't read text columns, and we must guess the number of trees. The three libraries people use fix these. All of them use histogram-based splitting (bin each feature once, try only bin edges). XGBoost adds a regularised objective (a penalty per leaf and on leaf values). LightGBM grows trees leaf-wise (always split the leaf that helps most; size knob num_leaves). CatBoost handles text categories natively with ordered target statistics. With early stopping, a validation set decides when to stop adding trees, so the learning rate and n_estimators stop being a guessing game. subsample and colsample bring the bagging and random-forest ideas into boosting.

Sources for the product facts in this episode: the XGBoost paper (Chen and Guestrin, KDD 2016, arXiv 1603.02754), the LightGBM features documentation, the CatBoost paper (Prokhorenkova et al., NeurIPS 2018) and the CatBoost documentation on categorical features.

Words you need first

  • Gradient boosting, pseudo-residuals, learning rate (shrinkage): Episode 35.
  • Categorical feature: a column whose values are names from a fixed list (area, plan) (Episode 13).
  • One-hot encoding: one 0/1 column per category (Episode 16).
  • Train / validation / test split, target leakage: Episode 15.
  • Regularisation, ridge (L2) penalty: Episode 21.
  • Log loss: the loss for predicted probabilities; lower is better (Episodes 12, 22).
  • Bagging, random forest, max_features: Episodes 33-34.

Real data breaks Episode 35

Sunita's tiffin service moved to an app: 20,000 made-up customers with six number columns and two text columns, area (30 neighbourhoods) and plan (veg, nonveg, jain). Some areas like Sunita much more than others. Split: 12,000 train, 4,000 validation, 4,000 test.

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
from sklearn.ensemble import GradientBoostingClassifier
print(X[["missed_meals", "rating", "area", "plan"]].head(3))
try: GradientBoostingClassifier(n_estimators=300).fit(X_tr, y_tr)      # ep35's way
except ValueError as e: print("ValueError:", e)                           # the real error, without the long traceback

Output:

   missed_meals    rating     area    plan
0             3  1.609491  area_21  nonveg
1             4  2.436860  area_00  nonveg
2             1  3.630504  area_15     veg
ValueError: could not convert string to float: 'area_21'

A tree split asks "is x ≤ t?"; "area_21" is not a number, so scikit-learn's classic GradientBoostingClassifier refuses it. (The programs catch the error with try/except and print its type and message, so the real message is visible without the long library traceback.)

One-hot, and the speed problem

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import time, xgboost, lightgbm
from sklearn.ensemble import GradientBoostingClassifier
Xd = pd.get_dummies(X, dtype=float)                 # one-hot: area_00 ... area_29, plan_jain ...
print("columns after one-hot:", Xd.shape[1], "  unique ratings:", X.rating.nunique())
for name, m in [("sklearn GradientBoosting", GradientBoostingClassifier(n_estimators=100, max_depth=3)),
                ("XGBoost (hist)", xgboost.XGBClassifier(n_estimators=100, max_depth=3, n_jobs=1)),
                ("LightGBM", lightgbm.LGBMClassifier(n_estimators=100, num_leaves=8, n_jobs=1, verbose=-1))]:
    t = time.perf_counter(); m.fit(Xd[:12000], y_tr); secs = time.perf_counter() - t
    print(f"{name:<25} fit {secs:5.2f} s   test {m.score(Xd[16000:], y_te):.3f}")

Output:

columns after one-hot: 39   unique ratings: 20000
sklearn GradientBoosting  fit  1.89 s   test 0.882
XGBoost (hist)            fit  0.11 s   test 0.900
LightGBM                  fit  0.08 s   test 0.892
  • Each model runs on one CPU core (n_jobs=1) so the times are comparable. One-hot turns 8 columns into 39; rating alone has 20,000 different values.
  • The classic implementation is more than ten times slower than XGBoost and LightGBM (100 trees each), and their accuracy is similar or better (XGBoost's default learning rate is 0.3, LightGBM's and scikit-learn's 0.1, so the scores differ). Exact times change from run to run. On millions of rows, that is minutes versus hours.

Histogram-based splitting

Real-life picture: to split a class by height, the exact way measures everyone to the millimetre and tries a line between every pair of students; the quick way puts students into height groups (140-145 cm, 145-150 cm, …), counts each group, and tries lines only between groups.

  • Exact split search: for every feature, sort the values and try every unique value as a threshold (20,000 candidates for rating).
  • Histogram-based: bin each feature once into a few hundred bins; at each split, add up the gradient statistics per bin (a histogram) and try only the bin edges. Much less work, almost the same splits. The LightGBM documentation describes exactly this ("bucket continuous feature values into discrete bins"); the XGBoost paper proposes candidate split points from feature percentiles and maps values into the buckets between them.
  • GBDT (gradient boosted decision trees) is the general name for gradient boosting with trees; XGBoost, LightGBM and CatBoost are all GBDT libraries.

XGBoost: a regularised objective

From the XGBoost paper (Chen and Guestrin, KDD 2016), training minimises

objective = Σ loss(yᵢ, ŷᵢ) + Σₖ Ω(treeₖ), Ω(tree) = γ·T + ½·λ·Σⱼ wⱼ²

  • T = number of leaves in the tree; wⱼ = the value leaf j adds to the prediction.
  • γ (gamma, gamma / min_split_loss): a fee per leaf; a split is kept only if it lowers the loss by more than γ.
  • λ (lambda, reg_lambda): an L2 (ridge) penalty on leaf values, pulling them towards 0 (Episode 21). reg_alpha adds an L1 penalty.
  • The paper also uses shrinkage (the learning rate eta / learning_rate), column subsampling borrowed from random forests, and a default direction learned for missing values at every node, so rows with gaps still reach a leaf.

A small worked example of how λ shrinks a leaf: for squared loss, the best value of a leaf holding rows with residuals r is w = Σr / (n + λ). Residuals 10, 12 and 8 (n = 3): λ = 0 → w = 30/3 = 10; λ = 3 → w = 30/6 = 5. A leaf with few rows is pulled towards 0 the most.

Early stopping

Real-life picture: tasting dal while it cooks; it improves, then gets worse if you wait; stop when it hasn't improved for a while and keep the best moment.

Early stopping: after each tree, score the model on a validation set; if the score hasn't improved for early_stopping_rounds trees in a row, stop and keep the best number of trees (best_iteration). Set n_estimators high and let early stopping choose.

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
for lr in [0.3, 0.1, 0.05]:
    m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=lr, max_depth=4, enable_categorical=True,
                              early_stopping_rounds=20, n_jobs=1)
    m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)                # watch the validation loss
    print(f"lr {lr:<5} early stop: best {m.best_iteration + 1:>4} trees   test {m.score(X_te, y_te):.3f}")
m = xgboost.XGBClassifier(n_estimators=500, learning_rate=0.1, max_depth=4, enable_categorical=True, n_jobs=1)
print(f"lr 0.1   no early stop: all 500 trees    test {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")

Output:

lr 0.3   early stop: best   55 trees   test 0.896
lr 0.1   early stop: best  168 trees   test 0.898
lr 0.05  early stop: best  332 trees   test 0.901
lr 0.1   no early stop: all 500 trees    test 0.892
  • Learning rate 0.3 → best at 55 trees; 0.1 → 168; 0.05 → 332. Smaller learning rate, more trees, slightly better test accuracy (0.896 → 0.898 → 0.901).
  • 500 trees without stopping: 0.892, worse than every early-stopped model: overfit.
  • In XGBoost 3, predict and score automatically use the best iteration after early stopping.

Validation log loss while XGBoost adds trees (learning rates 0.3, 0.1, 0.05)

Recipe: pick the smallest learning rate you can afford in time, set n_estimators very high, use early stopping on a validation set (never the test set), then report the test score once.

LightGBM: leaf-wise growth

Level-wise vs leaf-wise growth

  • Level-wise (depth-wise) growth: split every node of one level before going deeper; most decision tree learners grow this way.
  • Leaf-wise (best-first) growth: always split the leaf with the largest loss reduction, wherever it is. LightGBM does this; with the same number of leaves it usually reaches a lower loss.
  • Size knob: num_leaves (maximum leaves per tree, default 31). The LightGBM documentation warns that leaf-wise growth may over-fit when the data is small and offers max_depth to limit it; min_child_samples (minimum rows per leaf) also helps.
  • LightGBM is an open-source library maintained under Microsoft's GitHub organisation (its documentation is © Microsoft Corporation).
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import lightgbm
for rows in [12000, 1000]:
    for leaves in [8, 31, 255]:
        m = lightgbm.LGBMClassifier(n_estimators=200, num_leaves=leaves, n_jobs=1, verbose=-1)
        m.fit(X_tr[:rows], y_tr[:rows])                   # area and plan: native categories
        print(f"{rows:>5} rows, num_leaves {leaves:>3}: test {m.score(X_te, y_te):.3f}")

Output:

12000 rows, num_leaves   8: test 0.904
12000 rows, num_leaves  31: test 0.896
12000 rows, num_leaves 255: test 0.886
 1000 rows, num_leaves   8: test 0.873
 1000 rows, num_leaves  31: test 0.861
 1000 rows, num_leaves 255: test 0.863
  • With 12,000 rows, more leaves is slightly worse here (0.904 → 0.886); with 1,000 rows the drop is bigger (0.873 → 0.861). Small data, small num_leaves.

Categorical features

Giving XGBoost the text columns directly:

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
m = xgboost.XGBClassifier(n_estimators=300, n_jobs=1)
try: m.fit(df[:12000], y_tr)                          # df: area and plan are still text
except ValueError as e: print("ValueError:", e)

Output:

ValueError: DataFrame.dtypes for data must be int, float, bool or category. When categorical type is supplied, the experimental DMatrix parameter`enable_categorical` must be set to `True`.  Invalid columns:area: str, plan: str

The fix is in the message: convert the text to the pandas category dtype (each name stored once with a small code) and pass enable_categorical=True. That is what every other program here does (df.astype({"area": "category", "plan": "category"})).

Native categorical handling vs one-hot:

  • One-hot: 30 areas → 30 columns; a split can separate only one area from all the others.
  • Native: a split can send any group of areas left and the rest right. The LightGBM documentation describes sorting the categories by their accumulated gradient statistics and then finding the best cut in that order.
  • XGBoost: enable_categorical=True with category dtype. LightGBM: category dtype is enough. CatBoost: name the columns in cat_features; plain text is fine.

CatBoost and ordered target statistics

CatBoost turns each category into a number: roughly the average label of earlier rows with that category (target statistics, a form of target encoding; Episode 39 covers target encoding in depth). If a row's own label went into its category's average, the model would see part of the answer during training: target leakage. CatBoost's documentation describes the fix:

  1. Shuffle the rows into a random order (several random permutations are used).
  2. Encode each row using only the rows before it: value = (countInClass + prior) / (totalCount + 1), where countInClass counts earlier rows of that category with label 1, totalCount counts earlier rows of that category, and the prior is a starting constant.

Example from the documentation with prior 0.05: the first "rock" row gets 0.05; a later "rock" row whose only earlier "rock" had label 0 gets (0 + 0.05)/2 = 0.025. The CatBoost paper (NeurIPS 2018) presents this categorical processing together with ordered boosting, both designed to fight a prediction shift caused by target leakage.

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import catboost
cb = catboost.CatBoostClassifier(iterations=3000, learning_rate=0.2, depth=4, cat_features=["area", "plan"],
                                 early_stopping_rounds=20, thread_count=1, verbose=0)
cb.fit(df[:12000], y_tr, eval_set=(df[12000:16000], y_va))             # df: area and plan as plain text
print(f"CatBoost on raw text: best {cb.best_iteration_ + 1} trees   test {cb.score(df[16000:], y_te):.3f}")

Output:

CatBoost on raw text: best 219 trees   test 0.902
  • Raw text in, no encoding by hand: best at 219 trees (learning rate 0.2), test 0.902, close to the best XGBoost below (0.905). CatBoost calls the number of trees iterations and the tree depth depth.

subsample and colsample

  • subsample: each tree trains on a random fraction of the rows (without replacement). LightGBM: bagging_fraction (with bagging_freq).
  • colsample_bytree: each tree sees a random fraction of the columns. LightGBM: feature_fraction. (XGBoost also has colsample_bylevel and colsample_bynode.)
  • These bring the bagging (Episode 33) and random forest (Episode 34) ideas into boosting: trees differ more, chase less noise, and train faster.
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
for ss in [1.0, 0.8, 0.5]:                                    # row and column sampling per tree
    m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=0.1, max_depth=4, subsample=ss, colsample_bytree=ss,
                              enable_categorical=True, early_stopping_rounds=20, random_state=0, n_jobs=1)
    m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
    print(f"XGBoost subsample = colsample_bytree = {ss}: test {m.score(X_te, y_te):.3f}")

Output:

XGBoost subsample = colsample_bytree = 1.0: test 0.898
XGBoost subsample = colsample_bytree = 0.8: test 0.905
XGBoost subsample = colsample_bytree = 0.5: test 0.905
  • 0.898 → 0.905 with 80% (and also 50%) of rows and columns. Differences under about a point on 4,000 test rows are close to noise; confirm with cross-validation (Episode 26).

Which library when?

XGBoost LightGBM CatBoost
Tree growth (default) level-wise (max_depth) leaf-wise (num_leaves) depth-limited (depth)
Categories category dtype + enable_categorical=True category dtype raw text in cat_features
Main size knob max_depth num_leaves (+ max_depth, min_child_samples) depth
Number of trees n_estimators n_estimators iterations
Row / column sampling subsample, colsample_bytree bagging_fraction, feature_fraction subsample, rsm
Good first choice when general tabular data large data, speed matters many categorical columns

All three are strong; on a given dataset the differences are usually small compared with good features (Episodes 38-40) and honest validation. Try two, tune lightly with early stopping, and compare with cross-validation.

Common mistakes

import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
m = xgboost.XGBClassifier(n_estimators=3000, early_stopping_rounds=20, enable_categorical=True, n_jobs=1)
try: m.fit(X_tr, y_tr)                               # forgot eval_set
except ValueError as e: print("ValueError:", e)

Output:

ValueError: Must have at least 1 validation dataset for early stopping.
Mistake What you see Why Fix
Text columns into classic GradientBoostingClassifier ValueError: could not convert string to float: 'area_21' Splits need numbers Encode, or use a GBDT library with native categories
Raw text into XGBoost ValueError: DataFrame.dtypes for data must be int, float, bool or category. ... Invalid columns:area: str, plan: str Text, not category dtype astype("category") + enable_categorical=True
A fixed, large number of trees 500 trees: test 0.892 Overfitting Early stopping on a validation set
Early stopping on the test set an optimistic test score The test set chose the model A separate validation set
Large num_leaves on small data 0.873 → 0.861 Deep leaf-wise branches fit a few rows Small num_leaves, max_depth, min_child_samples
early_stopping_rounds without eval_set ValueError: Must have at least 1 validation dataset for early stopping. Nothing to watch Pass eval_set=[(X_va, y_va)]
Lowering the learning rate but not raising n_estimators stops at the cap, underfit ~10× smaller rate needs ~10× more trees Raise n_estimators; let early stopping decide

Mini project: LightGBM with early stopping on real data

import lightgbm as lgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
m = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.02, num_leaves=8, min_child_samples=10,
                       subsample=0.8, subsample_freq=1, colsample_bytree=0.8, verbose=-1, random_state=0)
m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], callbacks=[lgb.early_stopping(100, verbose=False)])
print(f"best trees: {m.best_iteration_}   validation log loss {m.best_score_['valid_0']['binary_logloss']:.3f}")
print(f"test accuracy {m.score(X_te, y_te):.3f}")
imp = sorted(zip(m.booster_.feature_importance("gain"), X.columns), reverse=True)[:3]
print("top 3 by gain:", [name for _, name in imp])

Output:

best trees: 176   validation log loss 0.124
test accuracy 0.937
top 3 by gain: ['worst concave points', 'worst area', 'mean concave points']
  • LightGBM's early stopping is a callback (lgb.early_stopping(rounds)); subsample needs subsample_freq ≥ 1 to take effect.
  • feature_importance("gain") is LightGBM's impurity-style importance (total loss reduction); remember Episode 34's warning and confirm with permutation importance before trusting it.

Practice

1. XGBoost with gamma = 0, 1 and 5 (early stopping). What happens to the tree size?

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost, json
for g in [0, 1, 5]:
    m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=0.1, max_depth=6, gamma=g,
                              enable_categorical=True, early_stopping_rounds=50)
    m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
    trees = m.get_booster().get_dump()[: m.best_iteration + 1]
    leaves = sum(t.count("leaf=") for t in trees) / len(trees)
    print(f"gamma {g}: {m.best_iteration + 1:>3} trees, {leaves:5.1f} leaves per tree, test {m.score(X_te, y_te):.3f}")

Output:

gamma 0:  90 trees,  53.7 leaves per tree, test 0.894
gamma 1:  96 trees,  47.2 leaves per tree, test 0.897
gamma 5:  94 trees,  24.2 leaves per tree, test 0.901

A larger γ charges more per leaf, so splits that only help a little are dropped and trees get smaller: a regularisation knob for deep trees.

2. LightGBM with num_leaves 8 vs 31, with and without max_depth=4, on 1,000 rows.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import lightgbm
for leaves in [8, 31]:
    for depth in [-1, 4]:
        m = lightgbm.LGBMClassifier(n_estimators=300, num_leaves=leaves, max_depth=depth, verbose=-1)
        m.fit(X_tr[:1000], y_tr[:1000])
        print(f"num_leaves {leaves:>2}, max_depth {depth:>2}: test {m.score(X_te, y_te):.3f}")

Output:

num_leaves  8, max_depth -1: test 0.872
num_leaves  8, max_depth  4: test 0.871
num_leaves 31, max_depth -1: test 0.855
num_leaves 31, max_depth  4: test 0.870

max_depth=-1 means no limit. A depth limit of 4 caps a tree at 16 leaves, so it tames num_leaves=31 on small data.

3. CatBoost vs one-hot XGBoost on only 1,000 training rows.

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import catboost, xgboost
cb = catboost.CatBoostClassifier(iterations=500, learning_rate=0.05, depth=4, verbose=0,
                                 cat_features=["area", "plan"], allow_writing_files=False)
cb.fit(df[:1000], y_tr[:1000])
Xd = pd.get_dummies(X, dtype=float)
xg = xgboost.XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=4).fit(Xd[:1000], y_tr[:1000])
print(f"CatBoost (raw text): {cb.score(df[16000:], y_te):.3f}   XGBoost (one-hot): {xg.score(Xd[16000:], y_te):.3f}")

Output:

CatBoost (raw text): 0.873   XGBoost (one-hot): 0.874

A tie (0.873 vs 0.874). With only 30 areas and about 33 customers per area, one-hot columns work fine. CatBoost's advantage is convenience (raw text in, no encoding code) and robustness when a column has hundreds or thousands of categories, where one-hot becomes huge and sparse; on a small, low-cardinality table like this, don't expect a gain. Always compare on your own data.

4. Choose the number of trees with 5-fold cross-validation (xgboost.cv).

Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)]   # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
    "price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
     + .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4))                  # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int)                                     # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"})                     # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
d = xgboost.DMatrix(X_tr, label=y_tr, enable_categorical=True)
params = {"objective": "binary:logistic", "eta": 0.1, "max_depth": 4, "eval_metric": "logloss"}
cv = xgboost.cv(params, d, num_boost_round=3000, nfold=5, early_stopping_rounds=50, seed=0)
print("trees chosen by 5-fold CV:", len(cv))
print("CV log loss at that point:", round(cv["test-logloss-mean"].iloc[-1], 4))

Output:

trees chosen by 5-fold CV: 156
CV log loss at that point: 0.2335

xgboost.cv stops when the mean validation loss across the 5 folds stops improving; the returned table has one row per kept tree. Close to the 168 trees picked by the single validation split, but steadier because it averages 5 splits.

5. By hand: with learning rate 0.1 early stopping kept 168 trees. Roughly how many trees would you expect at learning rate 0.02?

Solution

About 5 times as many: ≈ 800-850 trees (each tree adds a fifth as much). Set n_estimators well above that, e.g. 5,000, so early stopping can find the point.

Quick quiz

  1. Learning rate 0.1 → best at 168 trees. At 0.01? Roughly 10× more trees.
  2. What does histogram-based splitting try as thresholds? Bin edges, not every unique value.
  3. In Ω = γT + ½λΣw², what is T? The number of leaves.
  4. What does LightGBM grow at each step? The leaf with the largest loss reduction (leaf-wise).
  5. Why does CatBoost encode each row using only earlier rows? To avoid target leakage from the row's own label.
  6. What does early stopping need besides early_stopping_rounds? A validation set (eval_set).

Interview questions

  1. XGBoost vs classic gradient boosting? Regularised objective (γT + ½λΣw²), second-order (Newton) leaf values, histogram/approximate splits, missing-value default directions, column subsampling, fast parallel implementation.
  2. What is histogram-based split finding? Bin features once; accumulate gradient statistics per bin; evaluate only bin edges. Faster and less memory, almost no accuracy loss.
  3. Leaf-wise vs level-wise growth? Leaf-wise splits the best leaf anywhere (lower loss for a given leaf count, risk of overfitting on small data); level-wise splits all nodes of a level (more balanced trees).
  4. How do you set learning rate and number of trees? Small learning rate, large n_estimators, early stopping on validation data.
  5. How does CatBoost handle categorical features? Ordered target statistics: encode each row with the average label of earlier rows (random permutation) plus a prior, to avoid target leakage.
  6. What do subsample and colsample_bytree do? Random row/column sampling per tree; reduces overfitting and speeds up training.
  7. Name regularisation knobs in XGBoost. gamma, reg_lambda, reg_alpha, max_depth, min_child_weight, subsample, colsample_*, learning_rate.
  8. Why not early-stop on the test set? The test set would then help choose the model, so its score becomes optimistic.
  9. Do GBDT libraries need feature scaling? No; splits depend only on value order (bins are per feature).
  10. When might a random forest be preferred over GBDT? Very little tuning time, noisy small data, or when you want something robust with few knobs; GBDT usually wins with tuning.

Glossary

Term Meaning (plain words) Example from this episode
GBDT (gradient boosted decision trees) Gradient boosting where each learner is a tree XGBoost, LightGBM, CatBoost
XGBoost A GBDT library with a regularised objective (KDD 2016 paper) XGBClassifier
LightGBM A GBDT library with histogram bins and leaf-wise growth LGBMClassifier
CatBoost A GBDT library with native text-category handling (NeurIPS 2018 paper) CatBoostClassifier, 0.902
Exact split search Trying every unique value as a threshold 20,000 candidates for rating
Bin A range of values grouped together rating 3.10-3.12
Histogram-based splitting Binning features and trying only bin edges a few hundred candidates
Objective What training minimises loss + penalties
Regularised objective Loss plus a complexity penalty Ω = γT + ½λΣw²
γ (gamma, min_split_loss) Fee per leaf; minimum gain for a split gamma=5 → smaller trees
λ (reg_lambda) L2 penalty on leaf values w = Σr/(n + λ)
Leaf value (w) What a leaf adds to the prediction 10 → 5 with λ = 3
Level-wise (depth-wise) growth Split every node of a level first most tree learners
Leaf-wise (best-first) growth Split the leaf with the biggest loss drop LightGBM
num_leaves LightGBM's maximum leaves per tree 8, 31, 255
Early stopping Stop adding trees when validation stops improving best at 168 trees
early_stopping_rounds How many non-improving trees to allow 20
eval_set The validation data early stopping watches [(X_va, y_va)]
best_iteration Index of the best tree count (0-based in XGBoost) 167 → 168 trees
Validation log loss Log loss on the validation set the curves' minimum
Native categorical handling Splitting on groups of categories directly enable_categorical=True
pandas category dtype Text stored once with integer codes astype("category")
Target statistics (target encoding) Replacing a category by the average label area_21 → ≈ 0.8
Ordered target statistics Using only earlier rows in a random order, plus a prior CatBoost
Prior Starting value in the encoding formula 0.05 in the docs' example
Target leakage Training on information that contains the answer a row's own label in its encoding
Prediction shift The bias that leakage causes, named in the CatBoost paper fought by ordered boosting
subsample Fraction of rows per tree 0.8
colsample_bytree Fraction of columns per tree 0.8
iterations CatBoost's name for the number of trees 3000 cap, 219 kept

Summary

  • Episode 35's gradient boosting is a GBDT; on real tables it is slow, can't read text and needs a tree count.
  • Histogram-based splitting bins features once and tries only bin edges: same accuracy, much faster.
  • XGBoost minimises loss + γT + ½λΣw²; LightGBM grows leaf-wise (num_leaves); CatBoost reads text categories with ordered target statistics.
  • Early stopping on a validation set picks the number of trees; smaller learning rates need more trees.
  • Native categories beat one-hot for many-valued columns; subsample/colsample_bytree add healthy randomness.
  • Next episode: stacking and blending, a model that learns how to combine other models.