esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.21

Back propagation: who is to blame?

How a network finds out which of its weights caused a mistake, and fixes each one a little. Built by hand, with the result checked.

13 min readFree

From a score to a fix

The last lesson, Loss Functions, ended on a promise: backpropagation, how a network learns from its mistakes. A loss function gives a network one number that says how wrong it is. That is useful, but it is only a grade. It does not say which part to change.

Picture a kitchen team that sends out a soup, and the customer says it is far too salty. The grade is "too salty". Who fixes it? The cook who added the salt, the helper who salted the stock earlier, or the person who ordered a salty brand? A good head chef traces the mistake back through the line, step by step, and works out how much each person added to it. Then each person adjusts a little.

Backpropagation does exactly that inside a network. It starts at the grade and walks backwards, handing out a share of the blame to every weight. Then each weight is nudged a little to take less blame next time. Repeat that thousands of times and the network gets good.

Key words

Weight
A number inside the network that the model learns. A network can have thousands or millions of them.
Loss
One number that says how wrong the network is. Smaller is better.
Gradient
For one weight, how much the loss changes if you nudge that weight up a tiny bit. It is the weight's share of the blame.
Forward pass
Pushing an example through the network, layer by layer, to get a prediction.
Backward pass
Walking from the loss back towards the input, working out the gradient of every weight.
Chain rule
A rule from calculus for how fast the end of a chain of steps changes. The engine behind everything here.
Learning rate
The size of the step each weight takes. Small and careful, or big and risky.
Epoch
One full trip through all the training examples.

Tip

New to networks? The Autoencoders lesson explains a tiny network in plain words: weights, layers and a bend called an activation function. This lesson uses only that much.

What backpropagation is

Google's Machine Learning Crash Course calls backpropagation the most common training algorithm for neural networks, and says it makes gradient descent feasible for multi-layer neural networks. Gradient descent, from the Linear regression and Logistic regression lessons, is the walk downhill: find which way lowers the loss, then step that way. Backpropagation is the part that finds the way, for every weight at once.

Wikipedia gives the precise job. Backpropagation efficiently computes the gradient of the loss with respect to the network weights for a single input-output example. It does this by propagating derivatives backward, one layer at a time, from the output layer to the input layer, thereby avoiding redundant chain-rule calculations.

Wikipedia adds a careful point. Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used. People often use the word loosely for the whole learning recipe.

The chain rule, passed backwards

You met the chain rule in its own lesson. Two lines to refresh it. When one step feeds the next, like pedal to gear to wheel, the speed at the end is the speed of each link multiplied together. Nudge the start a tiny bit, and each link passes on its own share of that nudge.

A network is a chain of that kind. A weight sits early on. It changes a number, which goes through a bend, which feeds the next layer, and so on until the loss. To find how much that one weight matters, multiply the slopes of every link between it and the loss.

∂L∂w=∂L∂out⋅∂out∂z⋅∂z∂w\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \text{out}} \cdot \frac{\partial \text{out}}{\partial z} \cdot \frac{\partial z}{\partial w}

In words: how much the loss changes when the weight changes, equals how much the loss changes with the output, times how much the output changes with the unit's total input z, times how much z changes with the weight. Three small slopes, multiplied. The curly d is just the maths way of writing a slope that looks at one thing while the others stay still.

Now the clever part. Weights in the first layer and the last layer share most of that chain. Why work it out again for each one? Stanford's CS231n notes show the way to think about it. Every step in the chain is a small gate that knows only its own slope. In their words, the gate should take the gradient arriving from later on and multiply it into every gradient it normally computes for all of its inputs.

Think of a whisper passed back along the kitchen line. Each person hears the complaint, adds their own slice of responsibility, and passes it on. Nobody needs to see the whole kitchen. That is the backward pass.

The backward pass, step by step

Take a small network: two inputs, two hidden units, one output unit. Ours uses the sigmoid bend, which squeezes any number into a value between 0 and 1, and squared error as the loss.

  1. 1

    1. Forward pass

    Push the example through. Remember what every unit produced on the way.

  2. 2

    2. Score it

    Compare the prediction with the right answer and work out the loss.

  3. 3

    3. Blame the output

    Find the gradient at the output unit: how the loss changes as the output unit's total input changes.

  4. 4

    4. Blame its weights

    Each weight into the output unit gets the output blame multiplied by what flowed along that weight.

  5. 5

    5. Pass it back

    Send the blame to each hidden unit, scaled by the weight that connects them and by the hidden unit's own slope.

  6. 6

    6. Blame the first weights

    Repeat step 4 one layer earlier, using the blame each hidden unit just received.

  7. 7

    7. Take a step

    Move every weight a little against its gradient, using the learning rate.

Why the forward pass is not wasted

The backward pass needs numbers from the forward pass: what each unit gave out, and how steep its bend was there. Wikipedia says that for backpropagation, those activations as well as the derivatives must be cached for use during the backwards pass. Cached means saved so you do not work them out again. CS231n agrees that recomputing them is possible but wasteful.

This is the saving that makes the method fast. Wikipedia says backpropagation avoids duplication, because when it computes the gradient at one layer it does not recompute all the derivatives on later layers each time. Work out each layer's blame once, from back to front, and reuse it.

For the sigmoid there is a neat shortcut too. CS231n shows that its slope is the unit's own output times one minus the output. So once you have the output from the forward pass, the slope costs almost nothing.

Build it by hand

Time to see it work. The task is called XOR, short for exclusive or. The answer is 1 when exactly one of two inputs is 1, and 0 otherwise. It is a classic small test. We use the 2-2-1 network above and plain Python, with no libraries.

That nudge test is a way to catch mistakes. Move one weight a tiny bit up (by 0.00001), then a tiny bit down, and see how the loss changed. The change divided by the size of the nudge estimates the slope without any calculus. If backpropagation is right, the two answers match.

Python
import math
import random

random.seed(4)

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

data = [([0, 0], 0), ([0, 1], 1), ([1, 0], 1), ([1, 1], 0)]

# 2 inputs -> 2 hidden -> 1 output. Each unit has weights plus a bias.
w1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(2)]
b1 = [0.0, 0.0]
w2 = [random.uniform(-1, 1) for _ in range(2)]
b2 = 0.0

def forward(x):
    h = [sigmoid(w1[j][0] * x[0] + w1[j][1] * x[1] + b1[j]) for j in range(2)]
    out = sigmoid(w2[0] * h[0] + w2[1] * h[1] + b2)
    return h, out

def loss():
    return sum((forward(x)[1] - y) ** 2 for x, y in data) / 4

def gradients():
    g_w1 = [[0.0, 0.0], [0.0, 0.0]]
    g_b1 = [0.0, 0.0]
    g_w2 = [0.0, 0.0]
    g_b2 = 0.0
    for x, y in data:
        h, out = forward(x)                  # forward pass, kept for later
        d_out = 2 * (out - y) * out * (1 - out) / 4   # blame at the output
        g_b2 += d_out
        for j in range(2):
            g_w2[j] += d_out * h[j]
            d_h = d_out * w2[j] * h[j] * (1 - h[j])   # blame passed back
            g_b1[j] += d_h
            for i in range(2):
                g_w1[j][i] += d_h * x[i]
    return g_w1, g_b1, g_w2, g_b2

# Check one gradient (w1[0][0]) against a numerical estimate, before any training.
mine = gradients()[0][0][0]
nudge = 1e-5
w1[0][0] += nudge
up = loss()
w1[0][0] -= 2 * nudge
down = loss()
w1[0][0] += nudge
print()
print(f"backprop gradient:  {mine:.8f}")
print(f"numerical estimate: {(up - down) / (2 * nudge):.8f}")

print()
print(f"start: loss {loss():.4f}")
rate = 2.0
for step in range(1, 5001):
    g_w1, g_b1, g_w2, g_b2 = gradients()
    for j in range(2):
        for i in range(2):
            w1[j][i] -= rate * g_w1[j][i]
        b1[j] -= rate * g_b1[j]
        w2[j] -= rate * g_w2[j]
    b2 -= rate * g_b2
    if step % 1000 == 0:
        print(f"step {step}: loss {loss():.4f}")

print()
for x, y in data:
    print(f"input {x}  want {y}  got {forward(x)[1]:.3f}")

Output


backprop gradient:  0.00675888
numerical estimate: 0.00675888

start: loss 0.2581
step 1000: loss 0.0026
step 2000: loss 0.0009
step 3000: loss 0.0005
step 4000: loss 0.0004
step 5000: loss 0.0003

input [0, 0]  want 0  got 0.016
input [0, 1]  want 1  got 0.984
input [1, 0]  want 1  got 0.984
input [1, 1]  want 0  got 0.020

Read the output from the top. The first two lines are the check. The gradient from backpropagation and the nudge estimate both say 0.00675888, so the hand-written backward pass is correct for that weight. The loss starts at 0.2581, close to what you get by guessing 0.5 for everything. After 1,000 steps it is 0.0026, and by step 5,000 it is 0.0003.

The predictions agree. For inputs [0, 1] and [1, 0] the answer should be 1, and the network says 0.984. For [0, 0] and [1, 1] it should be 0, and the network says 0.016 and 0.020. Nobody told the hidden units what to do. Blame passed backwards shaped them.

Using the gradient: the update

Backpropagation only produces the gradients. The step that follows is plain gradient descent. Google's Machine Learning Crash Course says the learning rate determines the magnitude of the changes to make to the weights during each step, and that the model multiplies the gradient by the learning rate to get the next values.

wnew=wold−η⋅∂L∂ww_{\text{new}} = w_{\text{old}} - \eta \cdot \frac{\partial L}{\partial w}

In words: the new weight is the old weight minus the learning rate (the Greek letter eta, written like a small n) times that weight's gradient. If the gradient is positive, raising the weight raises the loss, so you lower the weight. If it is negative, you raise it. In the code above, the learning rate is 2.0, and the line w1[j][i] -= rate * g_w1[j][i] is this formula.

Epochs and mini-batches

Our four examples fit in one go, so every step used all of them. Real data sets have millions. Google's hyperparameters page describes the choices. Batch size is the number of training examples processed before updating the model, and epochs is how many times the entire training set is used.

Using every example for every step is slow. One example at a time is noisy: the loss can go up on some steps. The usual middle road is mini-batch stochastic gradient descent. In its words, the batch can be any size above 1 and below N, the examples are chosen at random, their gradients are averaged, and the weights update once per step.

When blame fades or blows up

Blame passes back by multiplying. That is fine in a small network. In a deep one, with many layers, it can go wrong in two ways.

  • Vanishing gradients

    Many small numbers multiplied shrink to almost nothing. Early layers hear only a faint whisper and learn very slowly, or not at all.

  • Exploding gradients

    Many large numbers multiplied grow huge. The steps become wild and training bounces around instead of settling.

  • The same cause

    Both come from a long product of terms, so deeper networks suffer more.

Google's crash course agrees: layers with vanishing gradients train very slowly, or not at all, and with very large weights the gradients get too large to converge.

Here is a run of numbers. The steepest the sigmoid's slope ever gets is 0.25. So each sigmoid layer the blame passes through can keep at most a quarter of it. The code multiplies that slope again and again.

Python
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# The sigmoid's slope is s * (1 - s). Its steepest point is at z = 0.
slope = sigmoid(0) * (1 - sigmoid(0))
print("steepest slope of the sigmoid:", slope)

# Passing blame back through n sigmoid layers multiplies by a slope per layer.
for n in (1, 3, 5, 10):
    print(n, "layers: blame left =", slope ** n)

Output

steepest slope of the sigmoid: 0.25
1 layers: blame left = 0.25
3 layers: blame left = 0.015625
5 layers: blame left = 0.0009765625
10 layers: blame left = 9.5367431640625e-07

After 10 layers, at best about one millionth of the blame is left. Wikipedia's article on the vanishing gradient problem gives the same picture: the gradients of earlier weights will be exponentially smaller than the gradients of later weights, which can slow training or halt it entirely.

Each problem has fixes that are named in our sources:

  • ReLU

    Google says the ReLU bend can help prevent vanishing gradients. Wikipedia says rectifiers like ReLU suffer less because they only saturate in one direction, meaning they go flat on one side only.

  • Careful starting weights

    Wikipedia lists weight initialization as another approach proposed to reduce the problem, choosing the random start to suit the bend that is used.

  • Gradient clipping

    For exploding gradients, Wikipedia says Pascanu and colleagues recommended clipping: if the gradient gets longer than a chosen limit, scale it back down to that limit.

Do I write this myself?

Almost never. Google says many libraries, such as Keras, handle backpropagation automatically, so you do not need to perform any of the underlying calculations yourself. Wikipedia explains why that works: automatic differentiation lets you implement backpropagation in a neural network without a manually computed derivative.

Quick check

In a long chain of layers, why does the blame for early weights sometimes fade to almost nothing?

Quick check

Why does the backward pass need what the forward pass produced?

Next: deep learning architectures, the main shapes networks come in.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.