Ensemble Learning

A single decision tree overfits, a single decision stump underfits, and a single logistic regression rarely wins Kaggle. Yet hundreds of the same mediocre models, combined with a small amount of care, routinely beat a hand-tuned neural network. This lesson is the theory and practice of that trick: why a committee of weak learners can dominate any single member of it, the three families of combination (bagging, boosting, stacking), what each one does to variance and bias, and the real costs (latency, interpretability, compute) that decide when not to use them.

Learning Objectives

1. Explain why combining independent models reduces error, using the variance of a mean of $M$ models and the role of correlation between them.
2. Derive bagging as variance reduction, describe bootstrap sampling with replacement, and identify the bias–variance trade-off it operates on.
3. Describe how a random forest adds feature subsampling at every split to bagging, and explain how that de-correlates trees in a way bagging alone cannot.
4. Walk through AdaBoost's reweighting of misclassified examples and explain the cost-sensitive interpretation, then walk through gradient boosting's residual-fitting view and its connection to gradient descent in function space.
5. Build a stacking ensemble with a meta-learner, and explain why the out-of-fold prediction requirement prevents target leakage.
6. Identify the conditions under which an ensemble hurts: tiny datasets, latency budgets, interpretability requirements, and computational constraints.

1. Why Ensembles Beat Single Models

Suppose you have $M$ models, each making a prediction $\hat{f}_m(x)$ on the same example $x$. The simplest ensemble is the average:

The error of any single model on $x$ can be written as

where $f(x)$ is the true function and $\epsilon_m$ is the noise specific to model $m$. If the noises are independent and identically distributed with variance $\sigma^2$, the variance of the ensemble's prediction is

That is the headline: averaging $M$ independent models cuts variance by a factor of $M$. The bias of the ensemble is the bias of any single member (an unweighted mean does not change bias), so the net error — bias² plus variance — falls.

1.1 The correlation problem

The $\sigma^2/M$ figure is the best case. In practice the models are not independent; they are trained on the same data, with the same features, and often the same algorithm. Their errors correlate. Let $\rho$ be the pairwise correlation between model errors. The variance of the mean becomes

Two limits make this concrete. If the models are perfectly correlated ($\rho = 1$), the ensemble has the same variance as a single model — averaging buys nothing. If they are independent ($\rho = 0$), the variance collapses by $1/M$. Real ensembles live somewhere on this line; the entire practical craft of ensemble design is finding ways to push $\rho$ down without raising the bias of the individual models too much.

This is the difference between "averaging" and "ensembling". Averaging five identical decision trees trained on the same data is just one model run five times. Ensembling five decision trees trained on different bootstrap samples of the data is a committee whose members disagree because they have seen slightly different worlds. Random forest, AdaBoost, and stacking are three different answers to the same question: how do we manufacture disagreement.

1.2 Why committees generalise

There is also a second, statistical reason. The bias–variance decomposition of expected prediction error is

where $\sigma^2$ is irreducible noise. Bagging reduces variance; boosting reduces bias; stacking tries to do both at once by learning a smart combination rule. Each family attacks a different term in the decomposition, and the choice of family is dictated by which term dominates the single model's error. The first question to ask of any single-model project is: "is this model high variance, high bias, or both?".

2. Bagging: Variance Reduction by Averaging

Bagging — bootstrap aggregating — is the variance-reduction playbook. The algorithm:

1. Draw $M$ bootstrap samples $D_1, \ldots, D_M$ from the training set, each by sampling $n$ examples with replacement.
2. Train model $m$ on $D_m$ independently.
3. Average (regression) or majority-vote (classification) the predictions.

Because each bootstrap sample omits about 37% of the original examples on average (the ones not selected in $n$ draws with replacement), every model sees a slightly different dataset. The disagreement this introduces is exactly what bagging is after.

2.1 The variance derivation

For regression with squared error, the squared error of an estimator $\hat{f}$ decomposes into bias² and variance. For an unweighted mean of $M$ i.i.d. estimators with pairwise correlation $\rho$,

Bagging cannot move $\sigma^2$ (the noise is in the data, not the model). What it can do is keep $\rho$ low by training on different bootstrap samples. A useful rule of thumb: bagging helps most when the base learner is unstable — small changes in the training data produce large changes in the fitted model. Decision trees are unstable; linear regression is stable. Bagging a linear model buys you almost nothing, because the fitted coefficients barely move between bootstrap samples; bagging a deep decision tree buys you a lot, because every bootstrap sample grows a different tree.

2.2 The out-of-bag estimate

A side benefit of bootstrap sampling is the out-of-bag (OOB) estimate. For training example $i$, about 37% of the bootstrap samples do not contain $i$. The OOB prediction for $i$ is the average (or majority vote) of the predictions from those models that did not see $i$. Aggregating the OOB predictions across all $n$ examples gives an unbiased estimate of test error — without holding out a separate validation set. This is genuinely useful: it means bagging ensembles come with a built-in cross-validation.

2.3 Implementation

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.utils import resample

def bagging_fit_predict(X_train, y_train, X_test,
                        n_estimators=100, max_depth=None,
                        random_state=0):
    rng = np.random.default_rng(random_state)
    preds = np.zeros((n_estimators, X_test.shape[0]))
    for m in range(n_estimators):
        #Bootstrap sample with replacement.
        idx = rng.integers(0, X_train.shape[0], X_train.shape[0])
        Xb, yb = X_train[idx], y_train[idx]
        tree = DecisionTreeRegressor(max_depth=max_depth,
                                     random_state=int(rng.integers(1 << 31)))
        tree.fit(Xb, yb)
        preds[m] = tree.predict(X_test)
    return preds.mean(axis=0)         # average for regression

#Majority-vote variant for classification replaces .mean(axis=0)
#with a per-column mode.

from sklearn.ensemble import BaggingClassifier, BaggingRegressor
from sklearn.tree import DecisionTreeClassifier

#scikit-learn's bagging wraps the algorithm and exposes OOB scoring.
clf = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=200, oob_score=True,
    n_jobs=-1, random_state=0,
).fit(X_train, y_train)

print(f"OOB accuracy = {clf.oob_score_:.3f}")
#OOB accuracy = 0.953

2.4 When bagging helps, and when it does not

Bagging reduces variance but does nothing for bias. If the base learner is underfitting (a decision stump, a linear model on a clearly non-linear problem), averaging 500 of them does not help — they all share the same bias. Bagging shines when the base learner overfits: deep decision trees, fully grown unpruned trees, large neural networks without strong regularisation. For underfitting models, look at boosting (Section 4) or feature engineering first.

3. Random Forest: Bagging Plus Decorrelation

A random forest is bagging of decision trees with one extra ingredient: at every split, only a random subset of features is considered. Let $p$ be the total number of features; the standard random forest samples $\sqrt{p}$ features per split for classification and $p/3$ for regression.

3.1 Why feature subsampling matters

Bagging already decorrelates trees by giving each one a different bootstrap sample. But if the data has one very strong feature, every tree will split on it first, and the trees will end up looking similar at the top. Their errors correlate. The $\sigma^2/M$ benefit is wasted because $\rho$ is too high.

The fix is to withhold the strong feature from some trees. By considering only a random subset of features at each split, the algorithm forces some trees to use weaker features. The resulting trees are more diverse — they disagree more — and the ensemble's variance falls further than bagging alone can deliver.

This is the core insight of random forest: the variance reduction from bagging is bounded by the correlation between trees, and feature subsampling is the lever that lowers that correlation.

3.2 The two randomisations, side by side

Both ingredients are necessary. Bagging alone (rows only) is dominated by the strongest feature; feature subsampling alone (a single tree, but with random splits) is still one tree and cannot reduce variance at all. The random forest combines both, and the combination is what makes it competitive across so many problems.

3.3 Implementation

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=300,
    max_features="sqrt",       # classification default
    oob_score=True,
    n_jobs=-1,
    random_state=0,
).fit(X_train, y_train)

print(f"OOB accuracy = {rf.oob_score_:.3f}")
#Feature importances: aggregated impurity decrease across all trees.
importances = rf.feature_importances_

A note on hyperparameters:

• n_estimators: more trees is monotonically better up to a plateau. Random forests do not overfit as you add trees; they just get more expensive.
• max_features: the smaller it is, the more decorrelated the trees, but the weaker each one. Cross-validate.
• max_depth: leaving it None (full-depth trees) gives the most variance to reduce and the most to gain from averaging.

4. Boosting: Bias Reduction by Sequential Fitting

Bagging trains models in parallel and reduces variance. Boosting trains models sequentially, with each new model focusing on the mistakes of the ensemble built so far, and reduces bias.

4.1 AdaBoost: reweighting misclassified examples

AdaBoost (Freund & Schapire, 1997) maintains a weight $w_i$ for every training example. Initially all weights are $1/n$. At each round $m$:

1. Fit a weak learner $h_m$ on the weighted training data.
2. Compute the weighted error $\epsilon_m = \sum_{i : h_m(x_i) \neq y_i} w_i$.
3. Compute the learner's "trust" $\alpha_m = \tfrac{1}{2}\ln\!\left(\frac{1 - \epsilon_m}{\epsilon_m}\right)$.
4. Update the weights: $w_i \leftarrow w_i \exp(-\alpha_m y_i h_m(x_i))$.
5. Renormalise so the weights sum to 1.

The final prediction is a weighted vote: $\hat{f}(x) = \text{sign}\!\left(\sum_{m=1}^M \alpha_m h_m(x)\right)$.

The intuition: examples that the current ensemble misclassifies get their weight multiplied by $e^{\alpha_m}$, so the next weak learner is forced to focus on them. Examples that are already correct get their weight reduced. After $M$ rounds, the ensemble is a committee of specialists, each one correcting a slice of the residual errors.

4.2 The cost-sensitive view

The weight update is equivalent to minimising exponential loss

where $f(x) = \sum_m \alpha_m h_m(x)$ is the ensemble's signed margin. The gradient of exponential loss with respect to the margin is $-e^{-y f(x)}$, so misclassified examples (small or negative margin) contribute a much larger gradient than correctly classified ones. Each round of boosting is therefore a step of gradient descent on exponential loss in the space of linear combinations of weak learners. AdaBoost is not magical: it is a particular optimiser for a particular loss.

The cost-sensitive framing also explains why AdaBoost can be brittle to label noise: mislabelled examples keep getting up-weighted forever, and the ensemble chases noise. Robust variants (LogitBoost, BrownBoost, RUSBoost) cap the weight or use a loss that down-weights outliers.

4.3 Gradient boosting: residual fitting

Gradient boosting generalises the AdaBoost idea to any differentiable loss. Instead of reweighting, it fits the next model to the negative gradient of the loss with respect to the current ensemble's predictions. For squared error loss, the negative gradient at example $i$ is exactly the residual:

So at each round, gradient boosting fits a weak learner to the current ensemble's residuals, then adds it (with a step size, the learning rate $\eta$) to the running prediction:

For squared error, this is literally gradient descent in function space: each new tree is a step along the steepest descent of the loss surface.

from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor

#Regression on squared error; n_estimators=200 with learning_rate=0.05
#and max_depth=3 (shallow trees, since each one only needs to fit
#the residuals well).
reg = GradientBoostingRegressor(
    n_estimators=200, learning_rate=0.05, max_depth=3,
    random_state=0,
).fit(X_train, y_train)

#Classification variant exposes loss='log_loss' (deviance) or
#loss='exponential' (which makes it close to AdaBoost).

4.4 The bias–variance picture for boosting

The two families are complementary: bagging fixes a model that overfits; boosting fixes a model that underfits.

5. Stacking: A Meta-Learner on Top

Stacking (Wolpert, 1992) generalises the ensemble idea. Instead of averaging or weighted-voting the base models' predictions, train a meta-learner to combine them.

5.1 The architecture

Given base learners $h_1, \ldots, h_M$:

1. Split the training data into $K$ folds.
2. For each fold $k$, train each $h_m$ on the other $K-1$ folds and predict on fold $k$. Stack these predictions to form the meta-features $z_i = (h_1(x_i), \ldots, h_M(x_i))$ for every training example $i$.
3. Train the meta-learner $g$ on $(z_i, y_i)$ pairs.
4. Refit each base learner on the full training set. To predict on a new example, run it through every base learner to form $z$, then through $g$.

5.2 The out-of-fold requirement

The critical detail is step 2: the meta-features must be made by out-of-fold prediction. If you naïvely train each base learner on the full training set and predict on the same training set, the meta-learner sees predictions from models that have already memorised those examples. It learns to trust the overconfident base learners that happen to memorise their training data, and the stacking ensemble generalises worse than any of its parts. This is target leakage through the meta-features, and it is the single most common stacking bug.

The $K$-fold structure guarantees that every $z_i$ is generated by a model that did not see $x_i$ during training, so the meta-learner learns how to combine predictions the way it will have to combine them at test time. This is also why stacking is computationally expensive: each base learner is trained $K$ times, not once.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import StackingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

estimators = [
    ("tree", DecisionTreeClassifier(max_depth=5, random_state=0)),
    ("svc", SVC(probability=True, random_state=0)),
    ("knn", KNeighborsClassifier(n_neighbors=15)),
]
stack = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(),   # the meta-learner
    cv=5,                                  # out-of-fold meta-features
    n_jobs=-1,
).fit(X_train, y_train)

print(f"stack accuracy = {stack.score(X_test, y_test):.3f}")

5.3 Choosing the meta-learner

A simple linear or logistic meta-learner is almost always enough. The meta-learner is given $M$ features (one per base model), so heavy non-linear models are unnecessary and overfit easily. Two practical choices:

• Logistic regression / linear regression with optional non-negative coefficient constraints, which forces the meta-learner toward an averaging-like combination.
• A second-level GBM if the base learners' errors have complex structure, but only with strong regularisation.

A useful diagnostic: train the stacking ensemble, then inspect the meta-learner's coefficients. If one base learner dominates, you have discovered that it is the only one worth keeping. If the coefficients are all similar in magnitude, the ensemble is genuinely combining diverse signals.

6. When Ensembles Hurt

Ensembles are not free. Five situations where they are the wrong call:

6.1 Latency-sensitive deployment

A random forest with 500 trees has to run every tree on every request. On a CPU, a deep tree takes milliseconds; 500 of them take seconds. If the product needs a 50 ms response (search ranking, ad bidding, real-time fraud), a single well-tuned model is the right choice. When the latency budget allows it, ensembles shine; when it does not, they cost more than they save.

6.2 Interpretability requirements

A single decision tree or logistic regression can be read by a domain expert. A random forest of 500 trees cannot. If the application is regulated (credit, healthcare, criminal justice), the right deliverable may be a model a human can interrogate, even at the cost of a few points of accuracy. TreeSHAP and other post-hoc explanations help but do not restore the legibility of a single tree.

6.3 Tiny training sets

Ensembles reduce variance by averaging, but variance reduction requires that the individual models are not all the same. On a dataset of 200 examples, every bootstrap sample looks essentially the same, every tree is essentially the same, and the ensemble has very little to gain. Empirically, ensembles start to clearly beat single models once the training set has at least a few thousand examples. Below that, the single-model baseline is competitive and the ensemble overhead is wasted.

6.4 No interpretation requirement, but the problem is solved by one good model

A second common reason to reach for an ensemble is "the single model isn't good enough". Before reaching for a 500-tree forest, check whether the single-model ceiling is actually being hit. Sometimes the bottleneck is data quality (label noise, missing values, leakage in the features), not model capacity. Cleaning the data usually beats ensembling dirty data.

6.5 Compute budget

A single XGBoost run on 10 million examples is expensive; running 500 of them is a different budget entirely. Training-time ensembles (random forest, gradient boosting with many trees) and inference-time ensembles (stacking with cross-validated meta-features) both multiply the cost of a single model by an order of magnitude or more. If the compute budget is fixed, the trade is: spend it on a bigger single model (more features, deeper tree, more training data) or on an ensemble. The bigger single model wins more often than intuition suggests.

7. Comparison and Practitioner Checklist

7.1 Bagging vs boosting vs stacking

7.2 When to reach for what

A short, opinionated playbook:

1. Start with a single model. A well-tuned gradient-boosted tree or random forest is the right baseline for most tabular problems. Spend the first day of any project on it.
2. If the single model overfits, bag it. Switch to a random forest with the same hyperparameters. Watch the OOB score to confirm the gain.
3. If the single model underfits, boost it. Switch to gradient boosting with shallow trees. Tune learning_rate and n_estimators together (smaller learning rate, more trees).
4. If both underfit and overfit are worries, blend them. A 50/50 average of a random forest and a gradient boosting model is a strong, low-effort ensemble.
5. If accuracy is paramount and compute is no object, stack. Use out-of-fold meta-features, a simple linear meta-learner, and only when every base learner has been individually tuned.

7.3 The honest summary

Ensembles buy accuracy at the price of latency, interpretability, and compute. The decision to use one is a product decision as much as a modelling decision. The decision to use a particular family — bagging, boosting, or stacking — is determined by which term in the bias–variance decomposition is the bottleneck for the single model. That diagnostic — bias-dominated, variance-dominated, or both — is the most useful thing this lesson teaches.

Key Takeaways

• Ensembles beat single models because averaging $M$ independent models reduces variance by a factor of $M$ (under independence); in practice the reduction is $\sigma^2 (\rho + (1 - \rho)/M)$ and the entire craft of ensemble design is reducing the pairwise correlation $\rho$ between members.
• Bagging is variance reduction: train $M$ models on $M$ bootstrap samples (each omits ~37% of the data) and average them; it works best with unstable base learners like deep decision trees, leaves bias unchanged, and provides a free OOB cross-validation estimate.
• A random forest adds feature subsampling (e.g. $\sqrt{p}$ features per split for classification) to bagging. The bagging-level decorrelation is bounded by the strongest feature in the data; withholding features from some splits pushes correlation down further and is the second lever that makes random forests work.
• Boosting is bias reduction: AdaBoost reweights misclassified examples and minimises exponential loss, while gradient boosting fits the next weak learner to the negative gradient of an arbitrary loss (residuals for squared error), implementing gradient descent in function space.
• Stacking trains a meta-learner on out-of-fold predictions of the base learners; the out-of-fold requirement is essential to prevent target leakage through the meta-features, and a simple linear meta-learner is usually sufficient.
• Ensembles hurt when latency is tight, interpretability matters, the dataset is tiny (a few hundred examples), the compute budget is fixed, or the bottleneck is data quality rather than model capacity — in those cases, fix the data or grow the single model rather than averaging many copies of it.