Machine Learning Foundations / Advanced / 03.27
Convolution
How a network looks at a picture: a small grid of numbers slides across the image, multiplies and adds, and finds edges and patterns.
13 min readFree
How a network looks at pictures
The last lesson, PyTorch, ended with a promise: convolution, how a network looks at pictures. This is that lesson. It is the idea behind the picture-reading networks you met in Deep Learning Architectures, and it is simpler than the name sounds.
Think of reading a page with a small magnifying glass. You cannot see the whole page at once. You slide the glass along, one spot at a time, and at each spot you notice something: a dark line, a curve, a gap. After the whole page you have a list of what you noticed and where. Convolution is that, done with numbers.
By the end you will be able to say what a filter is, how it slides, what it hands back, why a few made-up numbers can find an edge, and why a network can use the same few weights over a whole picture.
Key words
- Pixel
- One tiny dot of a picture. In a grey picture it is a single number, from dark to bright.
- Filter (kernel)
- A small grid of numbers, such as 3 by 3, that slides over a picture looking for one pattern. Filter and kernel mean the same thing.
- Convolution
- Sliding a filter over a picture and, at every spot, multiplying and adding. The word also names the whole operation.
- Feature map
- The grid of results you get after sliding one filter over the whole picture. Also called an activation map.
- Channel
- One layer of a picture. A colour picture has three: red, green and blue.
- Weight
- A number the network learns. The numbers inside a filter are weights.
- Edge
- A place where the picture changes fast, such as dark next to bright.
A picture is a grid of numbers
A computer does not see a cat. It sees numbers. A grey picture is a grid, and each cell holds one number for how bright that pixel is. A small number is dark. A big number is bright. A colour picture is three such grids stacked, one each for red, green and blue. We will come back to that.
To keep things small, we will use a 6 by 6 picture with a very simple scene: the left half is dark (0) and the right half is bright (9). Between them sits a straight, vertical edge. That is the one thing we want our filter to find.
A small filter slides over the picture
A filter is a tiny grid of numbers, here 3 by 3. Google's glossary describes a convolutional filter as a matrix with the same rank as the input matrix, but a smaller shape. Rank just means the number of directions: both are flat grids. Wikipedia describes a kernel as a small matrix used for blurring, sharpening, embossing, edge detection and more.
- 1
1. Lay it down
Put the filter on the top left corner of the picture, so it covers a 3 by 3 patch.
- 2
2. Multiply
Multiply each filter number by the pixel directly under it. Nine pairs, nine products.
- 3
3. Add
Add the nine products into one number. That number is the result for this spot.
- 4
4. Slide
Move the filter one pixel to the right and repeat. At the end of a row, drop down one row and start again.
- 5
5. Collect
Write each result into a new grid, in the position where the filter sat. That new grid is the feature map.
Here is the same thing as a formula. Say the output cell at row r and column c is what we want. Then we add up, over every row i and column j inside the filter, the filter number K at (i, j) times the picture pixel I at (r plus i, c plus j).
In words: line the filter up on a patch, multiply it cell by cell with the pixels under it, and add everything up. One patch gives one number.
Tip
Strict textbook convolution flips the filter before sliding it. Wikipedia writes "convolution (or cross-correlation)" about CNN layers, and the plain slide used here is the cross-correlation kind. Our code does the plain slide.
How big is the result?
A 3 by 3 filter cannot hang over the edge of the picture, at least not in this simple version. On a 6 by 6 picture it fits in 4 spots across and 4 spots down. So the feature map is 4 by 4, a little smaller than the picture. The next lesson, padding, shows what to do about that.
Filters made by hand: edges and blur
Before networks learned their filters, people made them by hand, picking numbers that did a job. Two classic jobs are finding edges and blurring.
Vertical edge filter
Each row reads [-1, 0, 1]: minus the left pixel, plus the right pixel. Where left and right match, the answer is 0. Where the right side is brighter, the answer is large. It lights up on vertical edges.
Horizontal edge filter
The same idea turned on its side: minus the pixels above, plus the pixels below. It lights up on horizontal edges, and stays quiet on vertical ones.
Blur filter
All nine numbers are 1/9. Each spot becomes the plain average of its patch, so sharp jumps get smoothed. Wikipedia lists this as a normalized box blur.
Let us do it with a plain Python function. It has two loops to walk across the picture, and two more to multiply and add inside each patch. No special library, so every step stays visible. We print the picture, then the feature map from each filter.
image = [
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
]
vertical_edge = [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]]
blur = [[1/9] * 3 for _ in range(3)]
def convolve(img, k):
n, m = len(img), len(k)
out = []
for r in range(n - m + 1):
row = []
for c in range(n - m + 1):
total = 0
for i in range(m):
for j in range(m):
total += img[r + i][c + j] * k[i][j]
row.append(round(total, 1))
out.append(row)
return out
def show(name, grid):
print(name)
for row in grid:
print(" ".join(f"{v:5.1f}" for v in row))
print()
show("input", image)
show("vertical edge filter", convolve(image, vertical_edge))
show("blur filter", convolve(image, blur))Output
input 0.0 0.0 0.0 9.0 9.0 9.0 0.0 0.0 0.0 9.0 9.0 9.0 0.0 0.0 0.0 9.0 9.0 9.0 0.0 0.0 0.0 9.0 9.0 9.0 0.0 0.0 0.0 9.0 9.0 9.0 0.0 0.0 0.0 9.0 9.0 9.0 vertical edge filter 0.0 27.0 27.0 0.0 0.0 27.0 27.0 0.0 0.0 27.0 27.0 0.0 0.0 27.0 27.0 0.0 blur filter 0.0 3.0 6.0 9.0 0.0 3.0 6.0 9.0 0.0 3.0 6.0 9.0 0.0 3.0 6.0 9.0
Look at the vertical edge filter's grid. Its first and last columns are 0: there the filter sat over flat areas, all dark or all bright, and found nothing. The two middle columns are 27. There the patch holds a dark column and a bright one, so each of the three rows gives 0 + 0 + 9 = 9, and three rows give 27. The edge is found, and its place shows in the map.
The blur grid tells a different story. The sharp jump from 0 to 9 has been spread over four steps: 0, 3, 6, 9. A patch on the edge holds some dark and some bright pixels, so its average sits in between. That is a blur: the hard line became a ramp.
Now turn the other way. A horizontal edge filter on the same picture finds nothing, because the picture has no horizontal edge.
image = [[0, 0, 0, 9, 9, 9] for _ in range(6)]
horizontal_edge = [[-1, -1, -1], [0, 0, 0], [1, 1, 1]]
out = [[sum(image[r + i][c + j] * horizontal_edge[i][j] for i in range(3) for j in range(3))
for c in range(4)] for r in range(4)]
for row in out:
print(row)Output
[0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0]
All zeros. Every patch has the same brightness top to bottom, so the minus row and the plus row cancel. A filter answers only the pattern it was made for. That is the whole trick: one filter, one kind of pattern.
Tip
You can build your own. Change the numbers in the filter and run the code again. In this lesson's code, the edge filter's numbers add up to 0 and the blur filter's add up to 1.
In a network, filters are learned
Hand-made filters are limited by our imagination. We can think of edges and blur. But what should a filter look for to tell a cat from a dog? Nobody knows the right numbers up front.
So a network does not use hand-made filters. Google's glossary says that in machine learning, convolutional filters are typically seeded with random numbers and then the network trains the ideal values. That is the same learning you saw in Back Propagation: guess, check the mistake, nudge the weights. Here the weights being nudged are the numbers in the filters.
Wikipedia says the same in one line: a convolutional neural network is a type of feedforward neural network that learns features via filter (or kernel) optimization. Nobody tells it to look for edges. Training pushes the filters toward whatever helps it get the answer right.
Colour: filters that span all the channels
A colour picture has three channels: red, green and blue. At each pixel you have three numbers, not one. So what does a filter do? It gets a layer of its own for each channel.
Stanford's CS231n notes put it like this: the extent of the connectivity along the depth axis is always equal to the depth of the input volume. The connections are local in 2D space (along width and height) but always full along the entire depth of the input. In plain words: a filter is small across the picture, but it always takes in every channel at once. Their example is a 5 by 5 filter on a picture with 3 colour channels: 5 times 5 times 3 is 75 weights, plus 1 bias.
At each spot the filter multiplies and adds across all its layers and gives back one single number. So a colour picture still produces a flat feature map. Here is one spot, with a patch where every pixel is 5 in all three channels, and a filter whose every weight is 1.
# one spot of a colour picture: a 3x3 patch in each of red, green, blue
patch = [[[5] * 3 for _ in range(3)] for _ in range(3)]
filt = [[[1] * 3 for _ in range(3)] for _ in range(3)] # one 3x3 set of weights per channel
total = sum(patch[ch][i][j] * filt[ch][i][j]
for ch in range(3) for i in range(3) for j in range(3))
print("one spot, 3 channels, one number:", total)
print("weights in this filter:", 3 * 3 * 3)
print("weights for 8 filters:", 8 * 3 * 3 * 3)Output
one spot, 3 channels, one number: 135 weights in this filter: 27 weights for 8 filters: 216
Nine pixels in each of three channels is 27 pairs. Each product is 5, and 27 times 5 is 135. One spot, one number. And the filter has 3 times 3 times 3, which is 27 weights.
Many filters, many feature maps
One filter finds one kind of pattern. A network wants many: a vertical edge, a horizontal edge, a corner, a patch of green. So a convolutional layer uses many filters side by side, each with its own weights.
CS231n says the depth of the output corresponds to the number of filters we would like to use, each learning to look for something different in the input. If you use, say, 12 filters, each produces a separate 2D activation map, and these maps are stacked together into the output.
The stack of feature maps then becomes the input of the next layer. The next layer's filters span all of those maps as channels, exactly like the colour case above. Wikipedia adds that higher-layer features are extracted from wider context windows than lower-layer features. Early filters find small things such as edges. Later ones combine those into larger shapes.
From the code above: 8 filters that each span 3 colour channels hold 8 times 27, which is 216 weights in total. That is the whole layer.
Check yourself
Quick check
A vertical edge filter slides over a picture that is all one flat colour. What does the feature map show?
Quick check
A filter is 3 by 3 and the picture has 3 colour channels. How many weights does the filter have?
Quick check
Why does a convolutional layer need so few weights?
Recap
- 1
1. Grid
A picture is a grid of numbers, one per pixel, three grids for colour.
- 2
2. Slide
A small filter slides over the grid. At each spot: multiply with the pixels under it and add.
- 3
3. Map
The results make a feature map. A vertical edge filter lights up on vertical edges.
- 4
4. Learn
In a network the filter numbers are weights, learned in training. Many filters make many maps.
- 5
5. Share
One filter used at every spot means few weights, however big the picture.
Next: padding, keeping the edges of a picture.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.