Machine Learning Foundations / Advanced / 03.21
Back propagation: who is to blame?
How a network finds out which of its weights caused a mistake, and fixes each one a little. Built by hand, with the result checked.
13 min readFree
From a score to a fix
The last lesson, Loss Functions, ended on a promise: backpropagation, how a network learns from its mistakes. A loss function gives a network one number that says how wrong it is. That is useful, but it is only a grade. It does not say which part to change.
Picture a kitchen team that sends out a soup, and the customer says it is far too salty. The grade is "too salty". Who fixes it? The cook who added the salt, the helper who salted the stock earlier, or the person who ordered a salty brand? A good head chef traces the mistake back through the line, step by step, and works out how much each person added to it. Then each person adjusts a little.
Backpropagation does exactly that inside a network. It starts at the grade and walks backwards, handing out a share of the blame to every weight. Then each weight is nudged a little to take less blame next time. Repeat that thousands of times and the network gets good.
Key words
- Weight
- A number inside the network that the model learns. A network can have thousands or millions of them.
- Loss
- One number that says how wrong the network is. Smaller is better.
- Gradient
- For one weight, how much the loss changes if you nudge that weight up a tiny bit. It is the weight's share of the blame.
- Forward pass
- Pushing an example through the network, layer by layer, to get a prediction.
- Backward pass
- Walking from the loss back towards the input, working out the gradient of every weight.
- Chain rule
- A rule from calculus for how fast the end of a chain of steps changes. The engine behind everything here.
- Learning rate
- The size of the step each weight takes. Small and careful, or big and risky.
- Epoch
- One full trip through all the training examples.
Tip
New to networks? The Autoencoders lesson explains a tiny network in plain words: weights, layers and a bend called an activation function. This lesson uses only that much.
What backpropagation is
Google's Machine Learning Crash Course calls backpropagation the most common training algorithm for neural networks, and says it makes gradient descent feasible for multi-layer neural networks. Gradient descent, from the Linear regression and Logistic regression lessons, is the walk downhill: find which way lowers the loss, then step that way. Backpropagation is the part that finds the way, for every weight at once.
Wikipedia gives the precise job. Backpropagation efficiently computes the gradient of the loss with respect to the network weights for a single input-output example. It does this by propagating derivatives backward, one layer at a time, from the output layer to the input layer, thereby avoiding redundant chain-rule calculations.
Wikipedia adds a careful point. Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used. People often use the word loosely for the whole learning recipe.
The chain rule, passed backwards
You met the chain rule in its own lesson. Two lines to refresh it. When one step feeds the next, like pedal to gear to wheel, the speed at the end is the speed of each link multiplied together. Nudge the start a tiny bit, and each link passes on its own share of that nudge.
A network is a chain of that kind. A weight sits early on. It changes a number, which goes through a bend, which feeds the next layer, and so on until the loss. To find how much that one weight matters, multiply the slopes of every link between it and the loss.
In words: how much the loss changes when the weight changes, equals how much the loss changes with the output, times how much the output changes with the unit's total input z, times how much z changes with the weight. Three small slopes, multiplied. The curly d is just the maths way of writing a slope that looks at one thing while the others stay still.
Now the clever part. Weights in the first layer and the last layer share most of that chain. Why work it out again for each one? Stanford's CS231n notes show the way to think about it. Every step in the chain is a small gate that knows only its own slope. In their words, the gate should take the gradient arriving from later on and multiply it into every gradient it normally computes for all of its inputs.
Think of a whisper passed back along the kitchen line. Each person hears the complaint, adds their own slice of responsibility, and passes it on. Nobody needs to see the whole kitchen. That is the backward pass.
The backward pass, step by step
Take a small network: two inputs, two hidden units, one output unit. Ours uses the sigmoid bend, which squeezes any number into a value between 0 and 1, and squared error as the loss.
- 1
1. Forward pass
Push the example through. Remember what every unit produced on the way.
- 2
2. Score it
Compare the prediction with the right answer and work out the loss.
- 3
3. Blame the output
Find the gradient at the output unit: how the loss changes as the output unit's total input changes.
- 4
4. Blame its weights
Each weight into the output unit gets the output blame multiplied by what flowed along that weight.
- 5
5. Pass it back
Send the blame to each hidden unit, scaled by the weight that connects them and by the hidden unit's own slope.
- 6
6. Blame the first weights
Repeat step 4 one layer earlier, using the blame each hidden unit just received.
- 7
7. Take a step
Move every weight a little against its gradient, using the learning rate.
Why the forward pass is not wasted
The backward pass needs numbers from the forward pass: what each unit gave out, and how steep its bend was there. Wikipedia says that for backpropagation, those activations as well as the derivatives must be cached for use during the backwards pass. Cached means saved so you do not work them out again. CS231n agrees that recomputing them is possible but wasteful.
This is the saving that makes the method fast. Wikipedia says backpropagation avoids duplication, because when it computes the gradient at one layer it does not recompute all the derivatives on later layers each time. Work out each layer's blame once, from back to front, and reuse it.
For the sigmoid there is a neat shortcut too. CS231n shows that its slope is the unit's own output times one minus the output. So once you have the output from the forward pass, the slope costs almost nothing.
Build it by hand
Time to see it work. The task is called XOR, short for exclusive or. The answer is 1 when exactly one of two inputs is 1, and 0 otherwise. It is a classic small test. We use the 2-2-1 network above and plain Python, with no libraries.
That nudge test is a way to catch mistakes. Move one weight a tiny bit up (by 0.00001), then a tiny bit down, and see how the loss changed. The change divided by the size of the nudge estimates the slope without any calculus. If backpropagation is right, the two answers match.
import math
import random
random.seed(4)
def sigmoid(z):
return 1 / (1 + math.exp(-z))
data = [([0, 0], 0), ([0, 1], 1), ([1, 0], 1), ([1, 1], 0)]
# 2 inputs -> 2 hidden -> 1 output. Each unit has weights plus a bias.
w1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(2)]
b1 = [0.0, 0.0]
w2 = [random.uniform(-1, 1) for _ in range(2)]
b2 = 0.0
def forward(x):
h = [sigmoid(w1[j][0] * x[0] + w1[j][1] * x[1] + b1[j]) for j in range(2)]
out = sigmoid(w2[0] * h[0] + w2[1] * h[1] + b2)
return h, out
def loss():
return sum((forward(x)[1] - y) ** 2 for x, y in data) / 4
def gradients():
g_w1 = [[0.0, 0.0], [0.0, 0.0]]
g_b1 = [0.0, 0.0]
g_w2 = [0.0, 0.0]
g_b2 = 0.0
for x, y in data:
h, out = forward(x) # forward pass, kept for later
d_out = 2 * (out - y) * out * (1 - out) / 4 # blame at the output
g_b2 += d_out
for j in range(2):
g_w2[j] += d_out * h[j]
d_h = d_out * w2[j] * h[j] * (1 - h[j]) # blame passed back
g_b1[j] += d_h
for i in range(2):
g_w1[j][i] += d_h * x[i]
return g_w1, g_b1, g_w2, g_b2
# Check one gradient (w1[0][0]) against a numerical estimate, before any training.
mine = gradients()[0][0][0]
nudge = 1e-5
w1[0][0] += nudge
up = loss()
w1[0][0] -= 2 * nudge
down = loss()
w1[0][0] += nudge
print()
print(f"backprop gradient: {mine:.8f}")
print(f"numerical estimate: {(up - down) / (2 * nudge):.8f}")
print()
print(f"start: loss {loss():.4f}")
rate = 2.0
for step in range(1, 5001):
g_w1, g_b1, g_w2, g_b2 = gradients()
for j in range(2):
for i in range(2):
w1[j][i] -= rate * g_w1[j][i]
b1[j] -= rate * g_b1[j]
w2[j] -= rate * g_w2[j]
b2 -= rate * g_b2
if step % 1000 == 0:
print(f"step {step}: loss {loss():.4f}")
print()
for x, y in data:
print(f"input {x} want {y} got {forward(x)[1]:.3f}")Output
backprop gradient: 0.00675888 numerical estimate: 0.00675888 start: loss 0.2581 step 1000: loss 0.0026 step 2000: loss 0.0009 step 3000: loss 0.0005 step 4000: loss 0.0004 step 5000: loss 0.0003 input [0, 0] want 0 got 0.016 input [0, 1] want 1 got 0.984 input [1, 0] want 1 got 0.984 input [1, 1] want 0 got 0.020
Read the output from the top. The first two lines are the check. The gradient from backpropagation and the nudge estimate both say 0.00675888, so the hand-written backward pass is correct for that weight. The loss starts at 0.2581, close to what you get by guessing 0.5 for everything. After 1,000 steps it is 0.0026, and by step 5,000 it is 0.0003.
The predictions agree. For inputs [0, 1] and [1, 0] the answer should be 1, and the network says 0.984. For [0, 0] and [1, 1] it should be 0, and the network says 0.016 and 0.020. Nobody told the hidden units what to do. Blame passed backwards shaped them.
Using the gradient: the update
Backpropagation only produces the gradients. The step that follows is plain gradient descent. Google's Machine Learning Crash Course says the learning rate determines the magnitude of the changes to make to the weights during each step, and that the model multiplies the gradient by the learning rate to get the next values.
In words: the new weight is the old weight minus the learning rate (the Greek letter eta, written like a small n) times that weight's gradient. If the gradient is positive, raising the weight raises the loss, so you lower the weight. If it is negative, you raise it. In the code above, the learning rate is 2.0, and the line w1[j][i] -= rate * g_w1[j][i] is this formula.
Epochs and mini-batches
Our four examples fit in one go, so every step used all of them. Real data sets have millions. Google's hyperparameters page describes the choices. Batch size is the number of training examples processed before updating the model, and epochs is how many times the entire training set is used.
Using every example for every step is slow. One example at a time is noisy: the loss can go up on some steps. The usual middle road is mini-batch stochastic gradient descent. In its words, the batch can be any size above 1 and below N, the examples are chosen at random, their gradients are averaged, and the weights update once per step.
When blame fades or blows up
Blame passes back by multiplying. That is fine in a small network. In a deep one, with many layers, it can go wrong in two ways.
Vanishing gradients
Many small numbers multiplied shrink to almost nothing. Early layers hear only a faint whisper and learn very slowly, or not at all.
Exploding gradients
Many large numbers multiplied grow huge. The steps become wild and training bounces around instead of settling.
The same cause
Both come from a long product of terms, so deeper networks suffer more.
Google's crash course agrees: layers with vanishing gradients train very slowly, or not at all, and with very large weights the gradients get too large to converge.
Here is a run of numbers. The steepest the sigmoid's slope ever gets is 0.25. So each sigmoid layer the blame passes through can keep at most a quarter of it. The code multiplies that slope again and again.
import math
def sigmoid(z):
return 1 / (1 + math.exp(-z))
# The sigmoid's slope is s * (1 - s). Its steepest point is at z = 0.
slope = sigmoid(0) * (1 - sigmoid(0))
print("steepest slope of the sigmoid:", slope)
# Passing blame back through n sigmoid layers multiplies by a slope per layer.
for n in (1, 3, 5, 10):
print(n, "layers: blame left =", slope ** n)Output
steepest slope of the sigmoid: 0.25 1 layers: blame left = 0.25 3 layers: blame left = 0.015625 5 layers: blame left = 0.0009765625 10 layers: blame left = 9.5367431640625e-07
After 10 layers, at best about one millionth of the blame is left. Wikipedia's article on the vanishing gradient problem gives the same picture: the gradients of earlier weights will be exponentially smaller than the gradients of later weights, which can slow training or halt it entirely.
Each problem has fixes that are named in our sources:
ReLU
Google says the ReLU bend can help prevent vanishing gradients. Wikipedia says rectifiers like ReLU suffer less because they only saturate in one direction, meaning they go flat on one side only.
Careful starting weights
Wikipedia lists weight initialization as another approach proposed to reduce the problem, choosing the random start to suit the bend that is used.
Gradient clipping
For exploding gradients, Wikipedia says Pascanu and colleagues recommended clipping: if the gradient gets longer than a chosen limit, scale it back down to that limit.
Do I write this myself?
Almost never. Google says many libraries, such as Keras, handle backpropagation automatically, so you do not need to perform any of the underlying calculations yourself. Wikipedia explains why that works: automatic differentiation lets you implement backpropagation in a neural network without a manually computed derivative.
Quick check
In a long chain of layers, why does the blame for early weights sometimes fade to almost nothing?
Quick check
Why does the backward pass need what the forward pass produced?
Next: deep learning architectures, the main shapes networks come in.
Sources
- Wikipedia: Backpropagation
- Google Machine Learning Crash Course: Neural networks, training using backpropagation
- Google Machine Learning Crash Course: Linear regression, hyperparameters
- Stanford CS231n: Backpropagation, intuitions
- Wikipedia: Vanishing gradient problem
- Wikipedia: Automatic differentiation
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.