Bagging

EnsemblesIntermediate~7 min

Bagging — Average many models trained on bootstrap samples.

One deep decision tree overfits. But train many trees, each on a different random resample of the data, and average their votes — the noise cancels out. That’s bagging: bootstrap aggregating.

Tree 1’s bootstrap sample

Majority vote of 0 trees

Class 0
Class 1
Out-of-bag (hollow)

Out-of-bag error % vs iteration

One point per tree added. Every point is scored only by the trees that never saw it.

Bagging controls

Data

Dataset

Points per class20

Spread1.0×

Add points as

Model

Number of trees16

Max depth per tree7

Features per split

Bootstrap seed11

Playback

Step 0 / 31

Speed

Resample
Grow + vote
↻

Step 0 of 31 — tree 1 — resampled 40 points with replacement, 14 left out-of-bag (35%)

Hover any point on the left plot to see how many times tree 1 drew it. Drag a point on the right plot, or click empty space to drop a new one, and every tree retrains.

Break it

The idea in plain words

One deep decision tree overfits — its boundary is jagged and unstable. Bagging (bootstrap aggregating) trains many trees, each on a different random resample of the data drawn with replacement, then averages their votes. The noise cancels out.

Drag the number of trees up and the averaged boundary resolves from noisy to smooth. It’s pure variance reduction: averaging many high-variance, low-bias models keeps the low bias while shrinking the variance.

Now, the math

The ensemble prediction averages B trees, each fit on a bootstrap sample:

\hat{f}(x) = \frac{1}{B}\sum_{b=1}^{B} T_b(x)

$B$: the number of trees (bootstrap replicates).
$T_b$: the b-th tree, fit on a resample drawn with replacement.

▸ Show the derivation

If each tree has variance σ² and the trees were independent, averaging B of them would cut the variance to σ²/B. Real trees are correlated (they share the same data distribution), so the gain is smaller — which is exactly what random forests improve by decorrelating the trees.

Trace it by hand

Six points on the plane: P1 (1,2), P2 (2,1), P3 (4,3) are class 0 and P4 (8,7), P5 (9,2), P6 (6,8) are class 1. We bag B = 3 depth-1 trees (stumps), each grown on its own bootstrap resample of the 6 points, using the seeded generator (mulberry32, seed 7). Every number below is exact except 0.33, which is 1/3 rounded to 2 decimals.

Step 1 — draw bootstrap sample 1 with replacement
$\mathcal{D}_1 = \{P_1, P_1, P_3, P_4, P_5, P_6\} \;\Rightarrow\; \text{OOB}_1 = \{P_2\}$
Drawing 6 times with replacement picked P1 twice and P2 never — so P2 is out-of-bag (OOB) for tree 1: a point this tree can be honestly tested on later.
Step 2 — the other two resamples repeat and drop points too
$\mathcal{D}_2 = \{P_1, P_2, P_2, P_3, P_4, P_5\}, \qquad \mathcal{D}_3 = \{P_2, P_2, P_3, P_4, P_4, P_5\}$
A bootstrap sample leaves out about 37 percent of points on average; here the three samples of 6 left out 1, 1 and 2 points.
Step 3 — fit one stump per resample
$T_1(x) = \mathbf{1}[x_1 > 5], \qquad T_2(x) = \mathbf{1}[x_1 > 6], \qquad T_3(x) = \mathbf{1}[x_1 > 6]$
Same algorithm, different resamples, different boundaries. Samples 2 and 3 never drew P6 (the class-1 point at x1 = 6), so their split slid right — this is the tree-to-tree variance that bagging exists to average away.
Step 4 — majority vote at the test point (5.5, 4)
$\hat{f}(5.5,\,4) = \frac{1}{B}\sum_{b=1}^{3} T_b = \frac{1 + 0 + 0}{3} = 0.33 \;\Rightarrow\; \text{class } 0$
The trees genuinely disagree here — T1 votes 1, T2 and T3 vote 0 — and the average smooths three jumpy stumps into one steadier answer.
Step 5 — out-of-bag error, a free validation score
$\text{OOB error} = \frac{1}{3} \approx 0.33$
Each point is scored only by trees that never trained on it. P1, P2 and P6 were OOB at least once; P6 is misvoted by trees 2 and 3 (their split at x1 = 6 puts it on the class-0 side), so 1 of the 3 scored points is wrong.

What just happened: Three stumps on three resamples produced thresholds 5, 6 and 6 — real tree-to-tree variance — then the majority vote combined them, and the same bootstrap draws handed us an OOB error of 1/3 without touching a separate validation set.

Now Break It

Try this: With too few trees the ensemble is still noisy; bagging identical models adds nothing.

Control: Number of estimators slider (set to 1)

What happens: No ensemble benefit! With a single estimator you get all the variance of one overfit tree.

Where bagging is used

Bagging, short for bootstrap aggregating, shows up wherever a single model is too jittery to trust. It underpins random forests used in credit scoring, medical risk models, and remote-sensing land classification, where averaging many trees smooths out the noise any one tree would chase. Practitioners also apply it to bagged neural networks or bagged regression stumps in tabular competitions to shave variance off an already-tuned base learner. Because each model trains on an independent bootstrap sample, bagging parallelizes cleanly across cores or machines, which makes it attractive for large batch-scoring pipelines. A useful side benefit is the out-of-bag estimate: roughly a third of rows are left out of each bootstrap sample, giving a nearly free validation score without a separate holdout set.

The biggest misconception is that bagging reduces bias. It mainly reduces variance, so bagging a high-bias model like a shallow linear fit changes little; you want base learners that are individually low-bias but unstable, which is why deep, unpruned trees are the classic choice. A second pitfall is assuming the bootstrap resampling alone decorrelates the models. On its own it often does not, because strong features dominate every sample and the trees end up highly correlated, capping the variance reduction. That is exactly the gap random forests close by also sampling features. Finally, do not confuse bagging with boosting: bagging trains models independently and in parallel, while boosting trains them sequentially so each one corrects its predecessor.

Frequently asked questions

What is bagging in machine learning?

Bagging, or bootstrap aggregating, trains many copies of the same model on different bootstrap samples of the data and averages their predictions, or takes a majority vote for classification. Each bootstrap sample is drawn with replacement, so it is the same size as the original but contains duplicates and omits some rows. The averaging cancels out much of the random variation in individual models.

Why does bagging reduce variance but not bias?

Averaging many predictions pulls the ensemble toward the true expected prediction, shrinking the scatter caused by which particular training rows each model saw. That scatter is variance, so it drops. Bias, the systematic error shared by every model, is not removed by averaging because all the models make the same kind of mistake, so the average keeps it.

What is the difference between bagging and boosting?

Bagging trains its models independently and in parallel on resampled data, then combines them by averaging or voting, mainly to cut variance. Boosting trains models one after another, with each new model focusing on the examples the previous ones got wrong, mainly to cut bias. Bagging is robust to overfitting; boosting is more powerful but more prone to it.

What are out-of-bag samples?

For each bootstrap sample, the rows left out, about a third on average, are called out-of-bag samples for that model. You can score each model on its own out-of-bag rows and aggregate, producing a validation-quality error estimate without holding out a separate test set. This makes tuning cheaper for bagged ensembles like random forests.

When should I use bagging?

Reach for bagging when your base model is accurate on average but unstable, meaning small changes in the data cause large swings in its predictions, as with deep decision trees. It also helps when you have plenty of compute to train models in parallel. If your model is already stable or high-bias, bagging offers little and you may prefer boosting.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.