SyntaxPilot
Courses/Machine Learning/Lesson 33
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 33 of 90

Ensembles: voting and bagging

ensemble learningwisdom of crowdshard votingsoft votingbootstrap samplingbaggingout-of-bag (OOB) errorvariance reductionBaggingClassifier

Machine Learning Full Course · Episode 33. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The mela guesses and Mohan's customers are made up for teaching; the mini project uses the diabetes dataset bundled with scikit-learn.

The big idea: one model can be unstable or wrong in its own way. If several models are each better than guessing and their mistakes are independent, their combined answer (an ensemble) is better than a typical member: the wisdom of crowds. Copies of one model trained on the same data make the same mistakes, so they don't help. Different models can vote (hard voting counts labels, soft voting averages probabilities). To get many different versions of one model from one dataset, train each on a bootstrap sample (n rows drawn with replacement): that is bagging. Averaging cancels the random wobble of deep trees (variance reduction), and the ~36.8% of rows each model never saw give a free validation score, the out-of-bag (OOB) score. Bagging helps unstable, low-bias models; it does nothing for stable, simple ones.

Words you need first

  • Classifier, predict_proba: a model that predicts a class; the predicted probability of each class (Episodes 22-23).
  • Decision tree, depth, pruning, instability: Episodes 29-30.
  • Bias and variance: error from a model that is too simple; error from a model that changes a lot with the training data (Episode 20).
  • Binomial distribution: the number of successes in n independent yes/no trials (Episode 10).
  • Logistic regression, k-nearest neighbours (kNN): Episodes 22-23, 27.
  • Validation set, cross-validation: data held back to score a model; scoring on several splits (Episodes 15, 26).

Wisdom of crowds

At a village mela, 500 made-up visitors guess the number of laddoos in a jar (412). Each guess = truth + random error.

import numpy as np
from math import comb
rng = np.random.default_rng(33)                            # mela game: guess the laddoos in a jar
true = 412
guesses = true + rng.normal(0, 80, 500).round()            # 500 visitors, each off by about 80
print("typical single guess is off by:", np.median(np.abs(guesses - true)))
print("the crowd's average:", guesses.mean().round(1), " off by:", abs(guesses.mean() - true).round(1))
def majority(n, p):                                        # n independent voters, each right with prob p
    return sum(comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n // 2 + 1, n + 1))
for n in [1, 5, 25, 101]:
    print(f"{n:>3} independent voters, each 70% right -> majority right {majority(n, 0.7):.3f}")

Output:

typical single guess is off by: 55.0
the crowd's average: 415.2  off by: 3.2
  1 independent voters, each 70% right -> majority right 0.700
  5 independent voters, each 70% right -> majority right 0.837
 25 independent voters, each 70% right -> majority right 0.983
101 independent voters, each 70% right -> majority right 1.000
  • Random errors point in different directions, so averaging cancels them: one guess is off by 55, the average by 3.2.
  • For voting: the number of correct votes among n independent voters is binomial, so the chance that more than half are right is Σ over k > n/2 of C(n, k) pᵏ (1 − p)ⁿ⁻ᵏ. If p > 0.5, it rises towards 1 as n grows. If p < 0.5, it falls towards 0: a crowd of bad voters is worse than one.
  • Ensemble learning: combining the predictions of several models. The models are the ensemble's members (or base learners).

Majority vote of independent voters

Why copies don't help

Mohan's made-up khata: 2,000 customers with income (thousand rupees, 10-80) and years known (0-10); the hidden rule is curved (the more you earn, the fewer years of trust you need), with 3% of labels flipped as noise. 1,500 rows to train, 500 to test.

import numpy as np
rng = np.random.default_rng(34); n = 2000                  # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]   # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9  # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)     # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import VotingClassifier
trees = [DT(random_state=s).fit(X_tr, y_tr) for s in range(3)]   # 3 helpers, the SAME data
votes = np.mean([t.predict(X_te) for t in trees], axis=0) > 0.5
print(f"one tree {trees[0].score(X_te, y_te):.3f}   vote of 3 same-data trees {np.mean(votes == y_te):.3f}")
models = [("logreg", LogisticRegression(max_iter=1000)), ("knn", KNeighborsClassifier(15)),
          ("tree", DT(max_depth=5, random_state=0))]
for name, m in models: print(f"{name:<6} test {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
for vote in ["hard", "soft"]:
    v = VotingClassifier(models, voting=vote).fit(X_tr, y_tr)
    print(f"{vote} voting: test {v.score(X_te, y_te):.3f}")

Output:

one tree 0.908   vote of 3 same-data trees 0.906
logreg test 0.870
knn    test 0.922
tree   test 0.928
hard voting: test 0.922
soft voting: test 0.930
  • Three full trees on the same data (only random_state differs, which only breaks ties) are nearly identical: 0.908 alone, 0.906 voting. Same data, same mistakes, no independence, no gain.
  • Requirement for an ensemble to help: members that are accurate (better than guessing) and diverse (make different mistakes).

Hard and soft voting

  • Hard voting: each model predicts a class; the majority wins.
  • Soft voting: average the models' predict_proba; pick the class with the highest average. Uses each model's confidence; needs every member to have predict_proba.
  • Example: P(repaid) = 0.9, 0.4, 0.45. Hard: 1 of 3 says yes → no. Soft: average 0.583 → yes.
  • VotingClassifier([("name", model), ...], voting="hard" | "soft", weights=[...]); VotingRegressor averages numbers.
  • Result above: logistic regression 0.870 (a straight boundary on a curved rule), kNN 0.922, depth-5 tree 0.928; hard voting 0.922, soft voting 0.930. A small gain: three voters is a small crowd and they share many mistakes. Voting is a safe way to combine models when you don't know in advance which is best.

Bootstrap sampling

  • Sampling with replacement: pick a row, record it, put it back, repeat.
  • Bootstrap sample: n rows drawn with replacement from n rows. Same size, with repeats and gaps.
  • P(a given row is never picked) = (1 − 1/n)ⁿ → 1/e ≈ 0.368 as n grows (e ≈ 2.718, the base of the natural logarithm). So each bootstrap sample holds about 63.2% of the unique rows.

Analogy: 1,500 chits in a bowl; pick one, note it, put it back, 1,500 times.

Bagging from scratch

Bagging = bootstrap aggregating: (1) draw B bootstrap samples, (2) train one model on each, (3) aggregate: majority vote for classes, average for numbers.

import numpy as np
rng = np.random.default_rng(34); n = 2000                  # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]   # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9  # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)     # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
boot = np.random.default_rng(0)
i = boot.integers(0, 1500, 1500)                           # bootstrap: 1,500 picks WITH replacement
print("unique rows in one bootstrap sample:", len(np.unique(i)), f"({len(np.unique(i))/1500:.1%})")
print("most picks of one row:", np.bincount(i).max(), "  rows never picked:", 1500 - len(np.unique(i)))
def bag(rows, k):                                          # k full trees, each on a bootstrap of `rows`
    votes = []
    for _ in range(k):
        i = boot.choice(rows, len(rows))
        votes.append(DT(random_state=0).fit(X_tr[i], y_tr[i]).predict(X_te))
    return np.mean(votes, axis=0) > 0.5                    # majority vote
for k in [1, 10, 100]:
    print(f"bagging {k:>3} trees: test {np.mean(bag(np.arange(1500), k) == y_te):.3f}")
for k in [1, 100]:                                         # two shops, two different 80% samples
    a, b = [bag(np.random.default_rng(s).choice(1500, 1200, replace=False), k) for s in [1, 2]]
    print(f"{k:>3} tree(s) each: models disagree on {np.mean(a != b):.1%} of test customers")

Output:

unique rows in one bootstrap sample: 916 (61.1%)
most picks of one row: 6   rows never picked: 584
bagging   1 trees: test 0.914
bagging  10 trees: test 0.920
bagging 100 trees: test 0.934
  1 tree(s) each: models disagree on 5.6% of test customers
100 tree(s) each: models disagree on 0.2% of test customers
  • This sample has 61.1% unique rows (63.2% on average; one sample wobbles around it).
  • 100 bagged trees: 0.934, better than one full tree (0.908), the tuned depth-5 tree (0.928) and soft voting (0.930).
  • Stability: models trained on two different 80% samples disagree on 5.6% of customers as single trees, 0.2% as 100-tree bags. This is variance reduction.

One tree vs a pruned tree vs bagging

Why bagging works (and when it doesn't)

  • A deep tree has low bias (it can fit any shape) and high variance (it wobbles with the data). Part of each tree's wobble is random, so averaging many trees cancels it.
  • If B models each have variance σ² and their errors have pairwise correlation ρ, the variance of their average is ρσ² + (1 − ρ)σ²/B. More models shrink the second term; the first term stays. Bagged trees on the same data are still correlated (ρ > 0), which random forests attack next episode.
  • Bias is not reduced: averaging straight lines gives a straight line.
  • More models don't make bagging overfit more: the vote just settles down; extra models only cost time.

Out-of-bag (OOB) error

Each model never saw ~36.8% of the training rows. For every training row, let only the models that didn't see it vote, and score those votes: the out-of-bag score (accuracy) or OOB error (1 − accuracy). It's a validation estimate that needs no separate validation set.

import numpy as np
rng = np.random.default_rng(34); n = 2000                  # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]   # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9  # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)     # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import BaggingClassifier
r = np.random.default_rng(1)
left_out = [1 - len(np.unique(r.integers(0, 1500, 1500))) / 1500 for _ in range(1000)]
print(f"left out of a bootstrap (avg of 1,000): {np.mean(left_out):.1%}   1/e = {np.exp(-1):.1%}")
bagger = BaggingClassifier(DT(), n_estimators=100, oob_score=True, random_state=0)
bagger.fit(X_tr, y_tr)
print(f"OOB score {bagger.oob_score_:.3f}   test score {bagger.score(X_te, y_te):.3f}")
for name, m in [("tree", DT(random_state=0)), ("logreg", LogisticRegression(max_iter=1000))]:
    one = m.fit(X_tr, y_tr).score(X_te, y_te)
    many = BaggingClassifier(m, n_estimators=100, random_state=0).fit(X_tr, y_tr).score(X_te, y_te)
    print(f"{name:<6}: one model {one:.3f}   bagged x100 {many:.3f}")

Output:

left out of a bootstrap (avg of 1,000): 36.8%   1/e = 36.8%
OOB score 0.955   test score 0.942
tree  : one model 0.908   bagged x100 0.942
logreg: one model 0.870   bagged x100 0.870
  • 36.8% left out on average, matching 1/e.
  • OOB 0.955 vs test 0.942: close without touching the test set (any estimate from ~1,500 rows wobbles by a point or so). Keep a final test set anyway.
  • Bagging the unstable full tree: 0.908 → 0.942. Bagging stable, biased logistic regression: 0.870 → 0.870.

BaggingClassifier parameters

Parameter Meaning Default
estimator The base model that gets copied DecisionTreeClassifier()
n_estimators How many models 10
max_samples Rows per bag (int, or fraction of n) 1.0 (= n)
max_features Features per model (fraction or int) 1.0 (all)
bootstrap Sample rows with replacement True
bootstrap_features Sample features with replacement False
oob_score Compute the OOB score after fitting (oob_score_) False
n_jobs CPU cores to train on (−1 = all); bagging parallelises easily None (1)
random_state Seed for reproducible bags None

BaggingRegressor is the same for numbers (it averages). Pasting = bootstrap=False with max_samples < 1.0 (subsets without replacement).

The OOB mistake, reproduced:

import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
X = np.random.default_rng(0).normal(size=(100, 2)); y = (X[:, 0] > 0).astype(int)
BaggingClassifier(DT(), bootstrap=False, oob_score=True).fit(X, y)   # no bootstrap, but asks for OOB

Output:

Traceback (most recent call last):
  ... (library frames) ...
ValueError: Out of bag estimation only available if bootstrap=True

Common mistakes

Mistake What you see Why Fix
Voting with copies on the same data 0.908 → 0.906 Same mistakes Different models, or bootstrap samples
Hard voting with good probabilities 0.9, 0.4, 0.45 → "no" Ignores confidence voting="soft"
Voting with weak members (p < 0.5) Worse than one model Majority of wrong voters Only combine models better than guessing
Bagging a stable, simple model logreg 0.870 → 0.870 Bias, not variance Bag deep trees (or boost, Episode 35)
oob_score=True, bootstrap=False ValueError: Out of bag estimation only available if bootstrap=True No row is ever left out bootstrap=True
OOB with very few estimators UserWarning: Some inputs do not have OOB scores. Some rows were in every bag More estimators
Trusting OOB to 3 decimals OOB 0.955 vs test 0.942 Estimates wobble Keep a final test set

The few-estimators warning, reproduced:

import numpy as np, warnings
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
X = np.random.default_rng(0).normal(size=(200, 2)); y = (X[:, 0] > 0).astype(int)
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    b = BaggingClassifier(DT(), n_estimators=3, oob_score=True, random_state=0).fit(X, y)
    print(w[0].category.__name__ + ":", str(w[0].message).splitlines()[0])
print("oob_score_ with 3 trees:", round(b.oob_score_, 3))

Output:

UserWarning: Some inputs do not have OOB scores. This probably means too few estimators were used to compute any reliable oob estimates.
oob_score_ with 3 trees: 0.87

Mini project: bagging regression trees on real data

The bundled diabetes dataset (442 patients, 10 numeric features, target = a measure of disease progression one year after baseline), as in Episode 30's mini project:

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.metrics import mean_absolute_error as mae, r2_score
X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
tree = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
bag = BaggingRegressor(DecisionTreeRegressor(), n_estimators=200, oob_score=True, random_state=0).fit(X_tr, y_tr)
print(f"one tree   test MAE {mae(y_te, tree.predict(X_te)):.1f}")
print(f"200 trees  test MAE {mae(y_te, bag.predict(X_te)):.1f}   OOB R^2 {bag.oob_score_:.3f}   test R^2 {r2_score(y_te, bag.predict(X_te)):.3f}")
from sklearn.model_selection import cross_val_score
cv = cross_val_score(BaggingRegressor(DecisionTreeRegressor(), n_estimators=200, random_state=0), X_tr, y_tr, cv=5, scoring="r2")
print(f"200 trees  5-fold CV R^2 on the training part {cv.mean():.3f} (folds {cv.min():.2f} to {cv.max():.2f})")

Output:

one tree   test MAE 60.4
200 trees  test MAE 48.4   OOB R^2 0.471   test R^2 0.246
200 trees  5-fold CV R^2 on the training part 0.482 (folds 0.43 to 0.53)
  • Bagging 200 trees cuts the single tree's mean absolute error from 60.4 to 48.4.
  • The OOB R² (0.471) agrees with 5-fold cross-validation on the same training rows (0.482), but this particular 111-patient test split scores lower (0.246). With so few rows, one split is noisy and somewhat harder than average; that is why OOB and CV, which use many rows, are useful, and why a single test number should be read with care. (R² was introduced in Episode 19; for regressors oob_score_ is R².)
  • Episode 30 found that plain linear regression beats a single tree on this smooth target; ensembles narrow that gap, and the next episodes push further.

Practice

1. Print the bagging test score for 1, 5, 25, 100 and 200 trees on Mohan's data.

Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
for k in [1, 5, 25, 100, 200]:
    b = BaggingClassifier(DT(), n_estimators=k, random_state=0).fit(X_tr, y_tr)
    print(f"{k:>3} trees: test {b.score(X_te, y_te):.3f}")

Output:

  1 trees: test 0.914
  5 trees: test 0.930
 25 trees: test 0.938
100 trees: test 0.942
200 trees: test 0.940

The score climbs quickly and then levels off; after that it only wobbles by a few thousandths. Adding trees does not make bagging overfit.

2. BaggingRegressor on Episode 30's chai data (300 days, temperature → cups). Compare with one full tree.

Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(30)
temp = rng.uniform(10, 45, 300).round(1).reshape(-1, 1)
cups = (260 - 4 * temp[:, 0] + rng.normal(0, 15, 300)).round()
for name, m in [("one tree", DecisionTreeRegressor(random_state=0)),
                ("bagged x100", BaggingRegressor(DecisionTreeRegressor(), n_estimators=100, random_state=0))]:
    mae = -cross_val_score(m, temp, cups, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name:<12} CV MAE {mae:.1f} cups")

Output:

one tree     CV MAE 16.0 cups
bagged x100  CV MAE 14.2 cups

Averaging 100 jagged staircases gives a much smoother curve and a lower error.

3. Soft-vote logistic regression + kNN + a calibrated SVC on Mohan's data.

Solution
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import VotingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
models = [("logreg", make_pipeline(StandardScaler(), LogisticRegression())),
          ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
          ("svc", make_pipeline(StandardScaler(), CalibratedClassifierCV(SVC(), ensemble=False)))]
for name, m in models: print(f"{name:<6} {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
print(f"soft vote {VotingClassifier(models, voting='soft').fit(X_tr, y_tr).score(X_te, y_te):.3f}")

Output:

logreg 0.870
knn    0.946
svc    0.936
soft vote 0.934

CalibratedClassifierCV gives the SVC a predict_proba, so it can join a soft vote (Episodes 31-32 explain why SVMs give scores, not probabilities). Notice the soft vote (0.934) is below the best member, kNN (0.946): the weaker logistic regression pulls the average down. Voting is a safe default when you don't know which model is best, not a guarantee of beating every member; drop members that are clearly worse, or give them smaller weights.

4. Bag depth-1 trees (stumps). Any gain?

Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
stump = DT(max_depth=1, random_state=0).fit(X_tr, y_tr)
bag = BaggingClassifier(DT(max_depth=1), n_estimators=100, random_state=0).fit(X_tr, y_tr)
print(f"one stump {stump.score(X_te, y_te):.3f}   100 bagged stumps {bag.score(X_te, y_te):.3f}")

Output:

one stump 0.724   100 bagged stumps 0.738

Little or no gain: a stump is high-bias (one question can't follow a curve) and fairly stable, so averaging stumps still gives roughly one question. Stumps need boosting (Episode 35), which reduces bias.

5. By hand: n = 4 rows. Probability a given row is missing from a bootstrap sample? For n = 1,000?

Solution

(1 − 1/4)⁴ = 0.75⁴ ≈ 0.316. (1 − 1/1000)¹⁰⁰⁰ ≈ 0.368 ≈ 1/e.

Quick quiz

  1. Three models give P(yes) = 0.30, 0.45, 0.95. Hard vote? Soft vote? Hard: no. Soft: average 0.567 → yes.
  2. Why don't three trees trained on the same data help? They make the same mistakes (not independent).
  3. About what fraction of unique rows does a bootstrap sample contain? 63.2%.
  4. Does bagging mainly reduce bias or variance? Variance.
  5. What does the OOB score use to judge each row? Only the models that never saw that row.

Interview questions

  1. What is ensemble learning and why does it work? Combining several models; if members are better than chance and their errors are not perfectly correlated, errors partly cancel.
  2. Hard vs soft voting? Majority of predicted labels vs average of predicted probabilities; soft uses confidence and usually does a little better if probabilities are reasonable.
  3. What is a bootstrap sample? n draws with replacement from n rows; ~63.2% unique rows.
  4. What is bagging? Bootstrap aggregating: train models on bootstrap samples and average/vote.
  5. Which models benefit from bagging? Unstable, low-bias, high-variance ones such as deep trees; not stable or high-bias ones like linear models or stumps.
  6. What is the OOB error? Error of each training row predicted only by models whose bootstrap sample excluded it; a free validation estimate.
  7. Can adding more bagged trees cause overfitting? No; the ensemble converges, and extra trees only cost compute.
  8. Variance of an average of correlated models? ρσ² + (1 − ρ)σ²/B; correlation limits the gain, which is why random forests decorrelate trees.
  9. Bagging vs boosting? Bagging trains independent models in parallel and reduces variance; boosting trains models in sequence, each fixing the previous ones' errors, and reduces bias (Episode 35).
  10. What is pasting? Like bagging but sampling without replacement (bootstrap=False, max_samples < 1).

Glossary

Term Meaning (plain words) Example from this episode
Ensemble A group of models whose predictions are combined 100 bagged trees
Ensemble learning Building and combining such groups voting, bagging
Base learner / member One model inside an ensemble a full decision tree
Wisdom of crowds Combined independent guesses beat a typical individual 55 vs 3.2 laddoos off
Independence One model's mistakes don't predict another's violated by same-data copies
Hard voting Majority of predicted classes 0.922
Soft voting Average of predicted probabilities 0.930
VotingClassifier scikit-learn's voting ensemble logreg + kNN + tree
Sampling with replacement Picked items go back and can be picked again chits back in the bowl
Bootstrap sample n rows drawn with replacement from n 916 unique of 1,500
1/e ≈ 0.368, the share of rows a bootstrap leaves out 36.8%
Bagging Bootstrap aggregating 0.908 → 0.934
Aggregating Combining by vote (classes) or average (numbers) majority of 100 trees
Variance reduction Averaging cancels random wobble between models disagreement 5.6% → 0.2%
Out-of-bag (OOB) rows Rows a given model never saw ~37% per tree
OOB score / OOB error Accuracy / error judged only by models that didn't see each row 0.955
BaggingClassifier / BaggingRegressor scikit-learn's bagging ensembles n_estimators=100, oob_score=True
Pasting Subsets without replacement bootstrap=False, max_samples=0.8
n_jobs Number of CPU cores used in parallel −1 = all

Summary

  • Independent, better-than-chance models combined beat a typical member: the wisdom of crowds.
  • Copies on the same data share mistakes and don't help; members must be accurate and diverse.
  • Hard voting counts labels; soft voting averages probabilities.
  • Bootstrap samples (with replacement, ~63.2% unique) make many datasets from one.
  • Bagging trains a deep tree on each and votes/averages: variance reduction, more stable predictions.
  • The OOB score is a free validation estimate; bagging helps unstable models, not stable or simple ones.
  • Next episode: random forests, which make bagged trees even more different.