Bagging vs Boosting

In daily life we make decisions by working through possibilities one condition at a time — which is essentially how a decision tree works. Organizations use decision trees widely in supervised machine learning to analyze data and support better decisions.

But a single decision tree often isn't good enough on its own. Ensemble learning addresses this by combining multiple models into one stronger predictor, on the principle that a group of imperfect models, combined correctly, outperforms any one of them alone.

Bagging and Boosting are the two dominant ensemble strategies. They both build many decision trees and combine them, which makes them look similar — but they are solving two different problems, and knowing which problem you have is what tells you which method to reach for.

Why a Single Decision Tree Isn't Enough

A model's prediction error comes from two main sources, and they pull in opposite directions.

Variance is how much the model changes when the training data changes. A deep decision tree grown until every leaf is pure will memorize the training set — including its noise. Hand it a slightly different sample of data and it produces a noticeably different tree. It fits the training data extremely well and generalizes poorly. This is overfitting.

Bias is error from a model being too simple to capture the real pattern. A decision "stump" — a tree with a single split — is stable across different training samples, but it is too crude to represent anything complicated. This is underfitting.

This is the key to the whole lesson:

• Bagging reduces variance. It takes low-bias, high-variance models (deep trees) and averages away their instability.
• Boosting reduces bias. It takes high-bias, low-variance models (shallow trees) and combines them into something progressively more expressive.

Both produce a strong model from a collection of individually inadequate ones. They just start from opposite kinds of inadequacy.

Bagging (Bootstrap Aggregating)

Bagging's goal is to reduce variance and improve prediction accuracy.

The mechanism is bootstrap sampling: draw a random sample from the training data with replacement, the same size as the original dataset. This is not the same as splitting the data into separate pieces. Because each draw is independent and replacement is allowed, a single bootstrap sample will contain some records more than once and miss others entirely.

This small example makes it concrete:

import random
random.seed(7)

data = ["A", "B", "C", "D", "E"]

for i in range(3):
    sample = [random.choice(data) for _ in data]
    print(f"Sample {i+1}: {sample}  | left out: {sorted(set(data) - set(sample))}")

Expected output:

Sample 1: ['C', 'B', 'D', 'A', 'A']  | left out: ['E']
Sample 2: ['E', 'A', 'C', 'E', 'A']  | left out: ['B', 'D']
Sample 3: ['E', 'B', 'A', 'A', 'D']  | left out: ['C']

Every sample has five items, the same as the original — but "A" appears twice in sample 1, "E" twice in sample 2, and each sample misses something. That variety is the entire point: each tree sees a different version of reality and therefore makes different mistakes.

There's a reliable mathematical result here. As the dataset grows, each bootstrap sample contains about 63.2% of the original records, leaving roughly 36.8% unused. Simulating it on 1,000 records confirms the theory:

Average unique rows per bootstrap sample: 63.4%
Theoretical limit 1 - 1/e            : 63.2%

Each bootstrap sample trains its own decision tree. Once all trees are trained, their predictions are combined — majority voting for classification, averaging for regression.

Why does this beat a single tree? Each tree overfits, but they overfit differently, because each saw different data. Their errors are partly independent, so averaging cancels much of the random error while the genuine signal — which every tree picks up — survives. The trees are also trained independently and in parallel, which matters both for speed and for the contrast with boosting.

Random Forest

Random Forest is bagging with one additional source of randomness.

Alongside bootstrapping the rows, Random Forest also selects a random subset of features at each split when building a tree. Without this, one dominant predictor would be chosen as the top split in nearly every tree, producing trees that look alike and make correlated mistakes — and averaging correlated mistakes doesn't cancel them. Restricting the available features forces trees to explore different structures, which is what makes their errors independent enough for averaging to work well.

Steps in Random Forest

1. Assume the training dataset contains N observations and M features.
2. Draw a bootstrap sample of N records from the dataset, with replacement.
3. Build a decision tree on that sample, considering only a random subset of the M features at each split.
4. Repeat steps 2–3 many times to grow a forest of trees.
5. Combine all tree predictions — by majority vote for classification, or averaging for regression.

Advantages

• Works well on large, high-dimensional datasets.
• Strong accuracy with little tuning — the defaults are usually reasonable, unlike boosting.
• Resistant to overfitting: adding more trees does not cause it to overfit, it just stabilizes the result.
• Handles missing values in modern implementations (verified in scikit-learn 1.8; note this is version-dependent, as older versions required imputing missing values first).
• Provides a free validation estimate. The ~36.8% of records left out of each bootstrap sample — the out-of-bag records — were never seen by that tree, so they can be used to score it without a separate validation set.

Disadvantages

• Loss of interpretability. A single decision tree can be read and explained to a non-technical stakeholder. A forest of hundreds cannot, which matters in regulated settings where decisions must be justified.
• Higher memory and prediction cost, since every tree must be stored and consulted for each prediction.
• Cannot extrapolate in regression. Because predictions average the values held in leaf nodes, the output can never fall outside the range of the training targets. If house prices in training run from ₹20–80 lakh, the model cannot predict ₹95 lakh no matter what the input says.

Boosting

Boosting also combines many trees, but builds them sequentially rather than independently. Each new tree is trained specifically to correct the errors left behind by the trees before it.

The classic mechanism, used by AdaBoost (Adaptive Boosting), works through weighting. Every training record starts with equal weight. After a tree is trained, the records it got wrong have their weights increased, so the next tree pays more attention to exactly those cases. Repeat this many times and the ensemble progressively covers the cases that earlier models kept missing.

The base models here are deliberately weak learners — often trees just one or two levels deep. This is the reverse of bagging, and it follows from the bias-variance logic above: boosting adds expressive power step by step, so it must start from something simple. Starting boosting with deep, fully-grown trees is a common and costly mistake.

Gradient Boosting

Gradient Boosting generalizes the idea, replacing weight adjustment with optimization:

Gradient Boosting = Gradient Descent + Boosting

In this method:

• Trees are built one after another.
• Each new tree is fitted to the errors remaining from the combined model so far, not to the original target.
• The model's total error is measured by a loss function, and each tree is fitted to the negative gradient of that loss — which is why the name mentions gradient descent. Ordinary gradient descent adjusts numeric parameters step by step; gradient boosting adds a whole tree at each step instead.

One clarification worth making precisely, because it's easy to state loosely: the difference between the actual and predicted value is the residual. The loss is a function of that residual — squared error, absolute error, log loss, and so on. For squared error loss the negative gradient works out to be exactly the residual, which is why the two are often described interchangeably. For other loss functions they differ, and the gradient is what the algorithm actually uses.

A third critical ingredient is the learning rate (shrinkage). Each tree's contribution is scaled down by a factor — typically 0.01 to 0.1 — before being added. Smaller steps require more trees but generalize considerably better than fewer, larger steps.

Advantages

• Supports many different loss functions, so it adapts to classification, regression, and ranking problems.
• Excellent at capturing complex relationships and feature interactions.
• Typically delivers top-tier accuracy on structured, tabular data.

Disadvantages

• Requires careful hyperparameter tuning — principally the number of trees (n_estimators), the learning_rate, and tree depth (max_depth). These interact: lowering the learning rate means raising the number of trees.
• Slower to train. Because trees are sequential, training cannot be parallelized across trees the way bagging can.
• Can overfit if too many boosting rounds are run, since the model keeps fitting ever-finer details, eventually modelling noise. This is a genuine difference from bagging, where extra trees are harmless.

Modern implementations — XGBoost, LightGBM, and CatBoost — are optimized versions of gradient boosting that add regularization and much faster training, and they remain among the strongest performers on tabular data.

Bagging vs Boosting: Side by Side

The row about noisy data deserves emphasis, because it flows directly from the mechanism. Boosting increases the weight on whatever it keeps getting wrong. If a record is wrong because it was mislabeled, boosting will chase that error harder and harder. Bagging just averages it away.

Seeing the Difference in Code

This compares a single decision tree against both ensemble methods on the same data:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

models = {
    "Single decision tree": DecisionTreeClassifier(random_state=42),
    "Random Forest (bagging)": RandomForestClassifier(n_estimators=100, random_state=42),
    "Gradient Boosting":      GradientBoostingClassifier(n_estimators=100, random_state=42),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    print(f"{name:<25} accuracy: {model.score(X_test, y_test):.3f}")

Expected output:

Single decision tree      accuracy: 0.942
Random Forest (bagging)   accuracy: 0.971
Gradient Boosting         accuracy: 0.959

Both ensembles beat the single tree, which is the expected and central result — one tree's mistakes get corrected by combining many.

Note also that gradient boosting did not beat random forest here, despite being the more sophisticated method. That is worth taking seriously rather than treating as a fluke: gradient boosting's advantage depends on tuning, and these are untuned defaults. A random forest with default settings is often the stronger baseline, and only a properly tuned boosting model reliably overtakes it.

Which Should You Use?

Start with Random Forest when you want a strong result quickly, your data is noisy, you have limited time for tuning, or you can parallelize training across cores.

Move to Gradient Boosting when you need the highest achievable accuracy on tabular data, you have clean labels, and you have the time and validation setup to tune the learning rate, depth, and number of rounds properly.

Keep a single decision tree when interpretability outweighs accuracy — when someone has to read the model and explain why a specific decision was made.

Common Mistakes

• Using deep trees in boosting. Boosting is built on weak learners. Deep trees leave no errors for subsequent trees to correct, and overfit fast.
• Adding boosting rounds until training error hits zero. Unlike bagging, more rounds eventually hurts. Use a validation set to stop early.
• Assuming boosting always wins. As the output above shows, an untuned boosting model can lose to a default random forest.
• Treating bootstrap sampling as splitting the data. Each sample is the full dataset size, drawn with replacement — not a disjoint slice.

Related Concepts

Both methods build on the decision tree and on classification and regression, covered in this series' lesson on data mining techniques. The overfitting and model evaluation ideas here connect to the Evaluation phase of CRISP-DM in the implementation process lesson — the train/test split in the code above is exactly the "Generate Test Design" task described there.