Machine Learning Full Course · Episode 33. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The mela guesses and Mohan's customers are made up for teaching; the mini project uses the diabetes dataset bundled with scikit-learn.
The big idea: one model can be unstable or wrong in its own way. If several models are each better than guessing and their mistakes are independent, their combined answer (an ensemble) is better than a typical member: the wisdom of crowds. Copies of one model trained on the same data make the same mistakes, so they don't help. Different models can vote (hard voting counts labels, soft voting averages probabilities). To get many different versions of one model from one dataset, train each on a bootstrap sample (n rows drawn with replacement): that is bagging. Averaging cancels the random wobble of deep trees (variance reduction), and the ~36.8% of rows each model never saw give a free validation score, the out-of-bag (OOB) score. Bagging helps unstable, low-bias models; it does nothing for stable, simple ones.
Words you need first
- Classifier, predict_proba: a model that predicts a class; the predicted probability of each class (Episodes 22-23).
- Decision tree, depth, pruning, instability: Episodes 29-30.
- Bias and variance: error from a model that is too simple; error from a model that changes a lot with the training data (Episode 20).
- Binomial distribution: the number of successes in n independent yes/no trials (Episode 10).
- Logistic regression, k-nearest neighbours (kNN): Episodes 22-23, 27.
- Validation set, cross-validation: data held back to score a model; scoring on several splits (Episodes 15, 26).
Wisdom of crowds
At a village mela, 500 made-up visitors guess the number of laddoos in a jar (412). Each guess = truth + random error.
import numpy as np
from math import comb
rng = np.random.default_rng(33) # mela game: guess the laddoos in a jar
true = 412
guesses = true + rng.normal(0, 80, 500).round() # 500 visitors, each off by about 80
print("typical single guess is off by:", np.median(np.abs(guesses - true)))
print("the crowd's average:", guesses.mean().round(1), " off by:", abs(guesses.mean() - true).round(1))
def majority(n, p): # n independent voters, each right with prob p
return sum(comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n // 2 + 1, n + 1))
for n in [1, 5, 25, 101]:
print(f"{n:>3} independent voters, each 70% right -> majority right {majority(n, 0.7):.3f}")
Output:
typical single guess is off by: 55.0
the crowd's average: 415.2 off by: 3.2
1 independent voters, each 70% right -> majority right 0.700
5 independent voters, each 70% right -> majority right 0.837
25 independent voters, each 70% right -> majority right 0.983
101 independent voters, each 70% right -> majority right 1.000
- Random errors point in different directions, so averaging cancels them: one guess is off by 55, the average by 3.2.
- For voting: the number of correct votes among n independent voters is binomial, so the chance that more than half are right is Σ over k > n/2 of C(n, k) pᵏ (1 − p)ⁿ⁻ᵏ. If p > 0.5, it rises towards 1 as n grows. If p < 0.5, it falls towards 0: a crowd of bad voters is worse than one.
- Ensemble learning: combining the predictions of several models. The models are the ensemble's members (or base learners).

Why copies don't help
Mohan's made-up khata: 2,000 customers with income (thousand rupees, 10-80) and years known (0-10); the hidden rule is curved (the more you earn, the fewer years of trust you need), with 3% of labels flipped as noise. 1,500 rows to train, 500 to test.
import numpy as np
rng = np.random.default_rng(34); n = 2000 # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)] # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9 # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int) # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import VotingClassifier
trees = [DT(random_state=s).fit(X_tr, y_tr) for s in range(3)] # 3 helpers, the SAME data
votes = np.mean([t.predict(X_te) for t in trees], axis=0) > 0.5
print(f"one tree {trees[0].score(X_te, y_te):.3f} vote of 3 same-data trees {np.mean(votes == y_te):.3f}")
models = [("logreg", LogisticRegression(max_iter=1000)), ("knn", KNeighborsClassifier(15)),
("tree", DT(max_depth=5, random_state=0))]
for name, m in models: print(f"{name:<6} test {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
for vote in ["hard", "soft"]:
v = VotingClassifier(models, voting=vote).fit(X_tr, y_tr)
print(f"{vote} voting: test {v.score(X_te, y_te):.3f}")
Output:
one tree 0.908 vote of 3 same-data trees 0.906
logreg test 0.870
knn test 0.922
tree test 0.928
hard voting: test 0.922
soft voting: test 0.930
- Three full trees on the same data (only
random_statediffers, which only breaks ties) are nearly identical: 0.908 alone, 0.906 voting. Same data, same mistakes, no independence, no gain. - Requirement for an ensemble to help: members that are accurate (better than guessing) and diverse (make different mistakes).
Hard and soft voting
- Hard voting: each model predicts a class; the majority wins.
- Soft voting: average the models'
predict_proba; pick the class with the highest average. Uses each model's confidence; needs every member to havepredict_proba. - Example: P(repaid) = 0.9, 0.4, 0.45. Hard: 1 of 3 says yes → no. Soft: average 0.583 → yes.
VotingClassifier([("name", model), ...], voting="hard" | "soft", weights=[...]);VotingRegressoraverages numbers.- Result above: logistic regression 0.870 (a straight boundary on a curved rule), kNN 0.922, depth-5 tree 0.928; hard voting 0.922, soft voting 0.930. A small gain: three voters is a small crowd and they share many mistakes. Voting is a safe way to combine models when you don't know in advance which is best.
Bootstrap sampling
- Sampling with replacement: pick a row, record it, put it back, repeat.
- Bootstrap sample: n rows drawn with replacement from n rows. Same size, with repeats and gaps.
- P(a given row is never picked) = (1 − 1/n)ⁿ → 1/e ≈ 0.368 as n grows (e ≈ 2.718, the base of the natural logarithm). So each bootstrap sample holds about 63.2% of the unique rows.
Analogy: 1,500 chits in a bowl; pick one, note it, put it back, 1,500 times.
Bagging from scratch
Bagging = bootstrap aggregating: (1) draw B bootstrap samples, (2) train one model on each, (3) aggregate: majority vote for classes, average for numbers.
import numpy as np
rng = np.random.default_rng(34); n = 2000 # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)] # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9 # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int) # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
boot = np.random.default_rng(0)
i = boot.integers(0, 1500, 1500) # bootstrap: 1,500 picks WITH replacement
print("unique rows in one bootstrap sample:", len(np.unique(i)), f"({len(np.unique(i))/1500:.1%})")
print("most picks of one row:", np.bincount(i).max(), " rows never picked:", 1500 - len(np.unique(i)))
def bag(rows, k): # k full trees, each on a bootstrap of `rows`
votes = []
for _ in range(k):
i = boot.choice(rows, len(rows))
votes.append(DT(random_state=0).fit(X_tr[i], y_tr[i]).predict(X_te))
return np.mean(votes, axis=0) > 0.5 # majority vote
for k in [1, 10, 100]:
print(f"bagging {k:>3} trees: test {np.mean(bag(np.arange(1500), k) == y_te):.3f}")
for k in [1, 100]: # two shops, two different 80% samples
a, b = [bag(np.random.default_rng(s).choice(1500, 1200, replace=False), k) for s in [1, 2]]
print(f"{k:>3} tree(s) each: models disagree on {np.mean(a != b):.1%} of test customers")
Output:
unique rows in one bootstrap sample: 916 (61.1%)
most picks of one row: 6 rows never picked: 584
bagging 1 trees: test 0.914
bagging 10 trees: test 0.920
bagging 100 trees: test 0.934
1 tree(s) each: models disagree on 5.6% of test customers
100 tree(s) each: models disagree on 0.2% of test customers
- This sample has 61.1% unique rows (63.2% on average; one sample wobbles around it).
- 100 bagged trees: 0.934, better than one full tree (0.908), the tuned depth-5 tree (0.928) and soft voting (0.930).
- Stability: models trained on two different 80% samples disagree on 5.6% of customers as single trees, 0.2% as 100-tree bags. This is variance reduction.

Why bagging works (and when it doesn't)
- A deep tree has low bias (it can fit any shape) and high variance (it wobbles with the data). Part of each tree's wobble is random, so averaging many trees cancels it.
- If B models each have variance σ² and their errors have pairwise correlation ρ, the variance of their average is ρσ² + (1 − ρ)σ²/B. More models shrink the second term; the first term stays. Bagged trees on the same data are still correlated (ρ > 0), which random forests attack next episode.
- Bias is not reduced: averaging straight lines gives a straight line.
- More models don't make bagging overfit more: the vote just settles down; extra models only cost time.
Out-of-bag (OOB) error
Each model never saw ~36.8% of the training rows. For every training row, let only the models that didn't see it vote, and score those votes: the out-of-bag score (accuracy) or OOB error (1 − accuracy). It's a validation estimate that needs no separate validation set.
import numpy as np
rng = np.random.default_rng(34); n = 2000 # cousin Mohan's khata: 2,000 made-up customers
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)] # income (thousand Rs), years known
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9 # a curved made-up rule
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int) # 1 = repaid (3% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import BaggingClassifier
r = np.random.default_rng(1)
left_out = [1 - len(np.unique(r.integers(0, 1500, 1500))) / 1500 for _ in range(1000)]
print(f"left out of a bootstrap (avg of 1,000): {np.mean(left_out):.1%} 1/e = {np.exp(-1):.1%}")
bagger = BaggingClassifier(DT(), n_estimators=100, oob_score=True, random_state=0)
bagger.fit(X_tr, y_tr)
print(f"OOB score {bagger.oob_score_:.3f} test score {bagger.score(X_te, y_te):.3f}")
for name, m in [("tree", DT(random_state=0)), ("logreg", LogisticRegression(max_iter=1000))]:
one = m.fit(X_tr, y_tr).score(X_te, y_te)
many = BaggingClassifier(m, n_estimators=100, random_state=0).fit(X_tr, y_tr).score(X_te, y_te)
print(f"{name:<6}: one model {one:.3f} bagged x100 {many:.3f}")
Output:
left out of a bootstrap (avg of 1,000): 36.8% 1/e = 36.8%
OOB score 0.955 test score 0.942
tree : one model 0.908 bagged x100 0.942
logreg: one model 0.870 bagged x100 0.870
- 36.8% left out on average, matching 1/e.
- OOB 0.955 vs test 0.942: close without touching the test set (any estimate from ~1,500 rows wobbles by a point or so). Keep a final test set anyway.
- Bagging the unstable full tree: 0.908 → 0.942. Bagging stable, biased logistic regression: 0.870 → 0.870.
BaggingClassifier parameters
| Parameter | Meaning | Default |
|---|---|---|
estimator |
The base model that gets copied | DecisionTreeClassifier() |
n_estimators |
How many models | 10 |
max_samples |
Rows per bag (int, or fraction of n) | 1.0 (= n) |
max_features |
Features per model (fraction or int) | 1.0 (all) |
bootstrap |
Sample rows with replacement | True |
bootstrap_features |
Sample features with replacement | False |
oob_score |
Compute the OOB score after fitting (oob_score_) |
False |
n_jobs |
CPU cores to train on (−1 = all); bagging parallelises easily | None (1) |
random_state |
Seed for reproducible bags | None |
BaggingRegressor is the same for numbers (it averages). Pasting = bootstrap=False with max_samples < 1.0 (subsets without replacement).
The OOB mistake, reproduced:
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
X = np.random.default_rng(0).normal(size=(100, 2)); y = (X[:, 0] > 0).astype(int)
BaggingClassifier(DT(), bootstrap=False, oob_score=True).fit(X, y) # no bootstrap, but asks for OOB
Output:
Traceback (most recent call last):
... (library frames) ...
ValueError: Out of bag estimation only available if bootstrap=True
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Voting with copies on the same data | 0.908 → 0.906 | Same mistakes | Different models, or bootstrap samples |
| Hard voting with good probabilities | 0.9, 0.4, 0.45 → "no" | Ignores confidence | voting="soft" |
| Voting with weak members (p < 0.5) | Worse than one model | Majority of wrong voters | Only combine models better than guessing |
| Bagging a stable, simple model | logreg 0.870 → 0.870 | Bias, not variance | Bag deep trees (or boost, Episode 35) |
oob_score=True, bootstrap=False |
ValueError: Out of bag estimation only available if bootstrap=True |
No row is ever left out | bootstrap=True |
| OOB with very few estimators | UserWarning: Some inputs do not have OOB scores. |
Some rows were in every bag | More estimators |
| Trusting OOB to 3 decimals | OOB 0.955 vs test 0.942 | Estimates wobble | Keep a final test set |
The few-estimators warning, reproduced:
import numpy as np, warnings
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
X = np.random.default_rng(0).normal(size=(200, 2)); y = (X[:, 0] > 0).astype(int)
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always")
b = BaggingClassifier(DT(), n_estimators=3, oob_score=True, random_state=0).fit(X, y)
print(w[0].category.__name__ + ":", str(w[0].message).splitlines()[0])
print("oob_score_ with 3 trees:", round(b.oob_score_, 3))
Output:
UserWarning: Some inputs do not have OOB scores. This probably means too few estimators were used to compute any reliable oob estimates.
oob_score_ with 3 trees: 0.87
Mini project: bagging regression trees on real data
The bundled diabetes dataset (442 patients, 10 numeric features, target = a measure of disease progression one year after baseline), as in Episode 30's mini project:
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.metrics import mean_absolute_error as mae, r2_score
X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
tree = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
bag = BaggingRegressor(DecisionTreeRegressor(), n_estimators=200, oob_score=True, random_state=0).fit(X_tr, y_tr)
print(f"one tree test MAE {mae(y_te, tree.predict(X_te)):.1f}")
print(f"200 trees test MAE {mae(y_te, bag.predict(X_te)):.1f} OOB R^2 {bag.oob_score_:.3f} test R^2 {r2_score(y_te, bag.predict(X_te)):.3f}")
from sklearn.model_selection import cross_val_score
cv = cross_val_score(BaggingRegressor(DecisionTreeRegressor(), n_estimators=200, random_state=0), X_tr, y_tr, cv=5, scoring="r2")
print(f"200 trees 5-fold CV R^2 on the training part {cv.mean():.3f} (folds {cv.min():.2f} to {cv.max():.2f})")
Output:
one tree test MAE 60.4
200 trees test MAE 48.4 OOB R^2 0.471 test R^2 0.246
200 trees 5-fold CV R^2 on the training part 0.482 (folds 0.43 to 0.53)
- Bagging 200 trees cuts the single tree's mean absolute error from 60.4 to 48.4.
- The OOB R² (0.471) agrees with 5-fold cross-validation on the same training rows (0.482), but this particular 111-patient test split scores lower (0.246). With so few rows, one split is noisy and somewhat harder than average; that is why OOB and CV, which use many rows, are useful, and why a single test number should be read with care. (R² was introduced in Episode 19; for regressors
oob_score_is R².) - Episode 30 found that plain linear regression beats a single tree on this smooth target; ensembles narrow that gap, and the next episodes push further.
Practice
1. Print the bagging test score for 1, 5, 25, 100 and 200 trees on Mohan's data.
Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
for k in [1, 5, 25, 100, 200]:
b = BaggingClassifier(DT(), n_estimators=k, random_state=0).fit(X_tr, y_tr)
print(f"{k:>3} trees: test {b.score(X_te, y_te):.3f}")
Output:
1 trees: test 0.914
5 trees: test 0.930
25 trees: test 0.938
100 trees: test 0.942
200 trees: test 0.940
The score climbs quickly and then levels off; after that it only wobbles by a few thousandths. Adding trees does not make bagging overfit.
2. BaggingRegressor on Episode 30's chai data (300 days, temperature → cups). Compare with one full tree.
Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(30)
temp = rng.uniform(10, 45, 300).round(1).reshape(-1, 1)
cups = (260 - 4 * temp[:, 0] + rng.normal(0, 15, 300)).round()
for name, m in [("one tree", DecisionTreeRegressor(random_state=0)),
("bagged x100", BaggingRegressor(DecisionTreeRegressor(), n_estimators=100, random_state=0))]:
mae = -cross_val_score(m, temp, cups, cv=5, scoring="neg_mean_absolute_error").mean()
print(f"{name:<12} CV MAE {mae:.1f} cups")
Output:
one tree CV MAE 16.0 cups
bagged x100 CV MAE 14.2 cups
Averaging 100 jagged staircases gives a much smoother curve and a lower error.
3. Soft-vote logistic regression + kNN + a calibrated SVC on Mohan's data.
Solution
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import VotingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
models = [("logreg", make_pipeline(StandardScaler(), LogisticRegression())),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
("svc", make_pipeline(StandardScaler(), CalibratedClassifierCV(SVC(), ensemble=False)))]
for name, m in models: print(f"{name:<6} {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
print(f"soft vote {VotingClassifier(models, voting='soft').fit(X_tr, y_tr).score(X_te, y_te):.3f}")
Output:
logreg 0.870
knn 0.946
svc 0.936
soft vote 0.934
CalibratedClassifierCV gives the SVC a predict_proba, so it can join a soft vote (Episodes 31-32 explain why SVMs give scores, not probabilities). Notice the soft vote (0.934) is below the best member, kNN (0.946): the weaker logistic regression pulls the average down. Voting is a safe default when you don't know which model is best, not a guarantee of beating every member; drop members that are clearly worse, or give them smaller weights.
4. Bag depth-1 trees (stumps). Any gain?
Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
from sklearn.ensemble import BaggingClassifier
rng = np.random.default_rng(34); n = 2000
X = np.c_[rng.uniform(10, 80, n), rng.uniform(0, 10, n)]
ok = X[:, 0] / 10 + X[:, 1] + 1.5 * np.sin(X[:, 0] / 8) > 9
y = np.where(rng.random(n) < 0.03, ~ok, ok).astype(int)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
stump = DT(max_depth=1, random_state=0).fit(X_tr, y_tr)
bag = BaggingClassifier(DT(max_depth=1), n_estimators=100, random_state=0).fit(X_tr, y_tr)
print(f"one stump {stump.score(X_te, y_te):.3f} 100 bagged stumps {bag.score(X_te, y_te):.3f}")
Output:
one stump 0.724 100 bagged stumps 0.738
Little or no gain: a stump is high-bias (one question can't follow a curve) and fairly stable, so averaging stumps still gives roughly one question. Stumps need boosting (Episode 35), which reduces bias.
5. By hand: n = 4 rows. Probability a given row is missing from a bootstrap sample? For n = 1,000?
Solution
(1 − 1/4)⁴ = 0.75⁴ ≈ 0.316. (1 − 1/1000)¹⁰⁰⁰ ≈ 0.368 ≈ 1/e.
Quick quiz
- Three models give P(yes) = 0.30, 0.45, 0.95. Hard vote? Soft vote? Hard: no. Soft: average 0.567 → yes.
- Why don't three trees trained on the same data help? They make the same mistakes (not independent).
- About what fraction of unique rows does a bootstrap sample contain? 63.2%.
- Does bagging mainly reduce bias or variance? Variance.
- What does the OOB score use to judge each row? Only the models that never saw that row.
Interview questions
- What is ensemble learning and why does it work? Combining several models; if members are better than chance and their errors are not perfectly correlated, errors partly cancel.
- Hard vs soft voting? Majority of predicted labels vs average of predicted probabilities; soft uses confidence and usually does a little better if probabilities are reasonable.
- What is a bootstrap sample? n draws with replacement from n rows; ~63.2% unique rows.
- What is bagging? Bootstrap aggregating: train models on bootstrap samples and average/vote.
- Which models benefit from bagging? Unstable, low-bias, high-variance ones such as deep trees; not stable or high-bias ones like linear models or stumps.
- What is the OOB error? Error of each training row predicted only by models whose bootstrap sample excluded it; a free validation estimate.
- Can adding more bagged trees cause overfitting? No; the ensemble converges, and extra trees only cost compute.
- Variance of an average of correlated models? ρσ² + (1 − ρ)σ²/B; correlation limits the gain, which is why random forests decorrelate trees.
- Bagging vs boosting? Bagging trains independent models in parallel and reduces variance; boosting trains models in sequence, each fixing the previous ones' errors, and reduces bias (Episode 35).
- What is pasting? Like bagging but sampling without replacement (
bootstrap=False,max_samples < 1).
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Ensemble | A group of models whose predictions are combined | 100 bagged trees |
| Ensemble learning | Building and combining such groups | voting, bagging |
| Base learner / member | One model inside an ensemble | a full decision tree |
| Wisdom of crowds | Combined independent guesses beat a typical individual | 55 vs 3.2 laddoos off |
| Independence | One model's mistakes don't predict another's | violated by same-data copies |
| Hard voting | Majority of predicted classes | 0.922 |
| Soft voting | Average of predicted probabilities | 0.930 |
VotingClassifier |
scikit-learn's voting ensemble | logreg + kNN + tree |
| Sampling with replacement | Picked items go back and can be picked again | chits back in the bowl |
| Bootstrap sample | n rows drawn with replacement from n | 916 unique of 1,500 |
| 1/e | ≈ 0.368, the share of rows a bootstrap leaves out | 36.8% |
| Bagging | Bootstrap aggregating | 0.908 → 0.934 |
| Aggregating | Combining by vote (classes) or average (numbers) | majority of 100 trees |
| Variance reduction | Averaging cancels random wobble between models | disagreement 5.6% → 0.2% |
| Out-of-bag (OOB) rows | Rows a given model never saw | ~37% per tree |
| OOB score / OOB error | Accuracy / error judged only by models that didn't see each row | 0.955 |
BaggingClassifier / BaggingRegressor |
scikit-learn's bagging ensembles | n_estimators=100, oob_score=True |
| Pasting | Subsets without replacement | bootstrap=False, max_samples=0.8 |
n_jobs |
Number of CPU cores used in parallel | −1 = all |
Summary
- Independent, better-than-chance models combined beat a typical member: the wisdom of crowds.
- Copies on the same data share mistakes and don't help; members must be accurate and diverse.
- Hard voting counts labels; soft voting averages probabilities.
- Bootstrap samples (with replacement, ~63.2% unique) make many datasets from one.
- Bagging trains a deep tree on each and votes/averages: variance reduction, more stable predictions.
- The OOB score is a free validation estimate; bagging helps unstable models, not stable or simple ones.
- Next episode: random forests, which make bagged trees even more different.