esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.30

Gradient boosting: fixing mistakes

A forest asks many trees and averages them. Boosting builds trees one at a time, each one fixing what the last ones got wrong.

13 min readFree

Trees that fix each other's mistakes

The last lesson ended on a promise: gradient boosting, trees that fix each other's mistakes. Here it is.

Start with a picture. A group of friends tries to guess how many sweets are in a jar. In a forest, everyone guesses on their own, at the same time, and you average the guesses. That is the random forest from the last lesson: many trees, each trained on its own random sample of the data, and their predictions are averaged. Boosting works differently. The first friend guesses. The second friend hears how far off that guess was, and guesses only the gap. The third friend guesses what is still missing. Each one cleans up after the one before.

That is boosting, and gradient boosting is its best-known form. Google's Decision Forests course describes it like this: at each step, a new weak model is trained to predict the error of the current strong model. This lesson builds one from scratch, in plain Python, and watches it learn.

Key words

Ensemble
A group of small models whose answers are combined into one answer.
Bagging
Training many models on their own at the same time, then averaging them. A random forest does this.
Boosting
Training models one after another, so each new one fixes the mistakes of the ones before it.
Weak model
A small, simple model that is only a little better than a guess. In boosting, it is usually a short decision tree.
Residual
The gap between the real answer and the current prediction. Real minus predicted.
Learning rate
A small number that shrinks each new tree's contribution, so the model improves in small steps. Also called shrinkage.
Stump
A decision tree with one question only, so two possible answers.
Overfitting
Learning the training data so closely, noise included, that new data goes badly.

If decision trees still feel new, the last lesson covers them. For the words train and test data, see the lesson on train and test data.

Boosting versus bagging

A quick recap of the forest. Scikit-learn says each tree in a random forest is built from a bootstrap sample (a sample drawn with replacement) of the training set, and a random subset of the features is considered while each tree is built. The forest's answer is the averaged prediction of its trees.

Google's course puts bagging and boosting side by side: like bagging and boosting, gradient boosting is a methodology applied on top of another machine learning algorithm.

  • Bagging (random forest)

    Many trees, trained independently. Averaging cancels out their individual quirks. The trees never talk to each other.

  • Boosting (gradient boosting)

    Many small trees, trained in a line. Each one looks at what the team still gets wrong. The trees depend on each other.

  • In practice

    Wikipedia says that when a decision tree is the weak learner, the result usually outperforms random forest. Usually, not always. Test both on your own data.

Fit the mistakes, not the answers

Here is the whole trick. We want to predict a number, for example a house price. Boosting does not build one tree for the price. It builds a team, step by step.

  1. 1

    1. Start with a plain guess

    Predict the average price for every house. It is a poor model, but it is a start.

  2. 2

    2. Measure the mistakes

    For every house, work out the residual: the real price minus the current prediction. Some are positive (we guessed too low) and some negative (too high).

  3. 3

    3. Train a small tree on the mistakes

    The new tree's job is not to predict the price. Its job is to predict the residual. If it learns that big houses were underpriced, it will say "add something here".

  4. 4

    4. Add its answer, a little at a time

    Add the tree's prediction to the current model, shrunk by the learning rate.

  5. 5

    5. Repeat

    Work out the new, smaller mistakes and train the next tree on those. Stop after a set number of rounds, or when the model stops improving.

Wikipedia's informal introduction says the same in one line: gradient boosting will fit the new model to the residual, and each new model attempts to correct the errors of its predecessor. Google's course notes that the loop repeats until a stopping criterion is met, such as a maximum number of iterations, or if the model begins to overfit as measured on a separate validation dataset.

The final prediction is the starting guess plus the sum of every small tree's (shrunk) answer. No single tree is clever. The team is.

Quick check

A house sold for 300, and the model currently predicts 250. What does the next tree try to predict for that house?

Why is it called gradient?

So far, nothing sounded like a slope. Where does "gradient" come from? From the lesson on linear regression. There, gradient descent improved a guess in many small steps, each one downhill on the loss (one number saying how wrong the model is).

Boosting does the same thing, but instead of moving a number, it moves the whole model. Wikipedia gives the key fact for the usual squared-error loss: the residuals are proportional to the negative gradients of the mean squared error. In plain words, the residual points in exactly the direction that lowers the error fastest. Fitting a tree to the residuals is taking a step downhill.

−∂L∂F(xi)=2n (yi−F(xi))-\frac{\partial L}{\partial F(x_i)} = \frac{2}{n}\,\bigl(y_i - F(x_i)\bigr)

Say it in words: the downhill direction for example i is the real answer minus the model's prediction, times two over n (n is the number of examples). The constant does not matter, because the learning rate scales the step anyway. Google's course makes the same point: the gradient is the signed error times 2.

Tip

Why bother with the word? Because the gradient works for other kinds of loss too, not just squared error. For those, the "mistake" the next tree fits is no longer a plain subtraction. Google's course says that this equivalence holds only for regression with squared error. The recipe is the same. Only the loss changes.

The settings that matter

Gradient boosting has a handful of dials. Scikit-learn says the two most important are the number of trees and the learning rate. Another dial is the size of each tree. All of these are hyperparameters, so the lesson on tuning applies.

Learning rate: small steps

Imagine walking down a hill in fog. Long strides are quick but you may overshoot and stumble. Small steps are slow but safe. The learning rate is your stride length.

Google's course calls it shrinkage and says it is analogous to the learning rate in neural networks. It controls how fast the strong model is learning, which helps limit overfitting. A value closer to 0.0 reduces overfitting more than a value closer to 1.0. Its own code uses 0.1 as a common value.

Scikit-learn adds the catch: smaller values of the learning rate require larger numbers of weak learners to keep the same training error. Wikipedia agrees: small rates give dramatic improvements in generalisation, but at the price of more training time, because a lower rate needs more iterations.

Tip

A common recipe from scikit-learn: set the learning rate to a small constant such as 0.1 or less, then choose enough trees that early stopping applies. Small steps, as many as it takes.

Number of trees and early stopping

Every round adds a tree. Wikipedia explains the risk: adding more trees reduces the error on the training set, but raises the risk of overfitting. The best number of trees is often found by watching the error on a separate validation set.

Early stopping turns that into a rule. After each round, check the error on the validation set. If it has not improved for a set number of rounds, stop and keep the model from the best moment. Scikit-learn's histogram-based version switches this on by default when there are more than 10,000 samples.

Quick check

You lower the learning rate from 0.1 to 0.01. What usually happens to the number of trees you need?

Tree depth: keep trees short

Boosting wants weak models. A short tree asks few questions, so it cannot memorise much, and the team has to build up slowly. Wikipedia says that the higher the depth, the more likely the model is to overfit the training data.

Scikit-learn says a tree of depth h can capture interactions of order h. An interaction means two facts that only matter together, such as "small house AND city centre". Depth 1, a stump, looks at one fact at a time. Wikipedia quotes Hastie and colleagues: values between 4 and 8 for the number of leaves typically work well, results are fairly insensitive to the choice in that range, and a stump is insufficient for many applications.

Our example below uses stumps, to keep the code short. It works, but real projects usually use trees a little taller.

Worked example: boosting stumps

Let us build it. The data is made up: a smooth wave plus noise, 40 points to train on and 200 to test on. A seed fixes the random numbers, so the run repeats exactly. The model is the starting average, plus 100 stumps. Each stump finds the one cut along x that splits the current residuals best, and predicts the average residual on each side. The learning rate is 0.1. The error is the mean squared error (MSE): the average of the squared misses. Lower is better.

Read the loop at the bottom. It is the five steps from above: work out the residuals, fit a stump to them, add it to the model.

boost.py
import math
import random

random.seed(3)

ROUNDS = 100  # how many small trees to add
EVERY = 10    # print a line every this many rounds
RATE = 0.1    # the learning rate

# Made-up data: a smooth curve plus noise.
def make(n):
    rows = []
    for _ in range(n):
        x = random.uniform(0, 10)
        rows.append((x, math.sin(x) * 3 + random.gauss(0, 1)))
    return rows

train, test = make(40), make(200)

def mse(model, rows):
    return sum((predict(model, x) - y) ** 2 for x, y in rows) / len(rows)

def best_stump(rows):
    # Try a split between each pair of neighbouring x values; keep the lowest squared error.
    rows = sorted(rows)
    best = None
    for i in range(1, len(rows)):
        cut = (rows[i - 1][0] + rows[i][0]) / 2
        left = [r for _, r in rows[:i]]
        right = [r for _, r in rows[i:]]
        lm, rm = sum(left) / len(left), sum(right) / len(right)
        err = sum((r - lm) ** 2 for r in left) + sum((r - rm) ** 2 for r in right)
        if best is None or err < best[0]:
            best = (err, cut, lm, rm)
    return best[1:]

def predict(model, x):
    base, rate, stumps = model
    total = base
    for cut, lm, rm in stumps:
        total += rate * (lm if x < cut else rm)
    return total

base = sum(y for _, y in train) / len(train)
model = (base, RATE, [])
print("round  train MSE  test MSE")
print(f"{0:5d}  {mse(model, train):9.3f}  {mse(model, test):8.3f}")
for rnd in range(1, ROUNDS + 1):
    # The mistakes so far: real answer minus what the model says now.
    residuals = [(x, y - predict(model, x)) for x, y in train]
    model[2].append(best_stump(residuals))
    if rnd % EVERY == 0:
        print(f"{rnd:5d}  {mse(model, train):9.3f}  {mse(model, test):8.3f}")

Output

round  train MSE  test MSE
    0      3.553     4.819
   10      2.916     4.147
   20      2.463     3.632
   30      2.117     3.179
   40      1.851     2.883
   50      1.640     2.675
   60      1.470     2.476
   70      1.332     2.354
   80      1.220     2.248
   90      1.126     2.155
  100      1.046     2.064

Look at the two columns. At round 0 the model is just the average: train MSE 3.553, test MSE 4.819. After 10 stumps the train MSE falls to 2.916. After 100 it is 1.046, and the test MSE has dropped from 4.819 to 2.064. Each stump added a little. None of them did much alone.

Notice the test error is higher than the train error. That is normal: the model has seen the training points, and not the test ones. The noise in this data has a spread of 1, so no model can do much better than a test MSE near 1.0. We are not there yet. Should we keep going?

Run it too long and it overfits

Let us find out. Same code, but 1,000 rounds, printing every 100. The command below changes the two numbers at the top and runs the file again.

Terminal
sed 's/ROUNDS = 100/ROUNDS = 1000/; s/EVERY = 10 /EVERY = 100/' boost.py | python3 -I -

Output

round  train MSE  test MSE
    0      3.553     4.819
  100      1.046     2.064
  200      0.629     1.766
  300      0.463     1.772
  400      0.382     1.811
  500      0.334     1.847
  600      0.299     1.883
  700      0.273     1.921
  800      0.254     1.958
  900      0.239     1.968
 1000      0.226     1.974

The train error keeps falling, all the way to 0.226. The test error tells a different story. It falls to 1.766 at round 200, then climbs back up, to 1.974 at round 1000. The model has started to learn the noise in the 40 training points, and that does not help on new data. This is overfitting, in numbers.

This is why early stopping exists. Here you would stop around round 200, where the test error was lowest. In real projects, you do not use the test set for this. You use a validation set, as in the tuning lesson, and keep the test set for one last check.

Tip

Watch the gap between the train error and the test error, not the train error alone. When the gap keeps growing while the test error rises, the model is memorising.

Tools you will actually use

You will rarely write this loop yourself. Scikit-learn offers two versions of gradient-boosted trees. Its guide says the histogram-based one, HistGradientBoostingRegressor (and a matching Classifier), can be orders of magnitude faster when there are more than tens of thousands of samples, and supports missing values natively. It is the one to try first. The older GradientBoostingRegressor may be preferred for small sample sizes.

Scikit-learn's guide calls these models an excellent choice for both regression and classification, in particular for tabular data. Wikipedia's page also names two other well-known libraries, XGBoost and LightGBM.

One honest cost. Wikipedia notes that boosting sacrifices interpretability: following one decision tree is easy, but following hundreds or thousands of trees is much harder. Scikit-learn says the same: a boosted model has hundreds of trees, so you cannot read it by looking at the trees, and it points to techniques that summarise what the model uses.

  • Strengths

    Strong on tables of numbers and categories. Trees can pick up interactions between facts, as long as they are deep enough.

  • Costs

    Hundreds of trees are hard to read. Wikipedia notes a higher computational demand. The dials need tuning.

  • Good first move

    Try HistGradientBoosting with a learning rate near 0.1, and let early stopping pick the number of trees.

What to remember

  1. 1

    1. Boosting builds a line, bagging builds a crowd

    Boosting trains small models one after another, each fixing the last one's mistakes. A forest trains many at once and averages them.

  2. 2

    2. Each tree learns the residuals

    Real minus predicted. Add its answer to the model, scaled by the learning rate.

  3. 3

    3. Gradient

    For squared error, the residual is the downhill direction. Boosting is gradient descent on the whole model.

  4. 4

    4. Small steps, short trees

    A learning rate near 0.1 or lower, trees a few levels deep, and enough rounds.

  5. 5

    5. Stop in time

    Watch the validation error. Stop when it stops improving, or the model will start memorising.

Next: clustering, finding groups when nobody gives you the answers.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.