esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.31

Convolutional Neural Network

Put the filter, padding, stride and pooling together and you get the network that reads pictures. Here is the stack, layer by layer.

15 min readFree

Putting the pieces together

The last lesson, Pooling, ended with a promise: the convolutional neural network, putting the pieces together. This is that lesson. You already know every part. A filter slides over a picture. Padding keeps the edges. A stride sets how far the filter jumps. Pooling shrinks the result. A convolutional neural network is those parts lined up in a row, each one handing its output to the next.

Think of a factory line that sorts parcels. One worker looks for labels. The next worker groups the labelled parcels into piles. The last worker looks at the piles and decides which van each one goes on. No single worker sees everything, but the line as a whole gets the job done. A convolutional network works the same way, one layer after another.

Key words

Convolutional neural network (CNN)
A neural network built from layers that slide small filters over a picture. Also called a ConvNet.
Layer
One step in the line. It takes numbers in, does one job, and passes numbers on.
Activation
A small rule applied to each number after a layer. ReLU is the most common one: it keeps positive numbers and turns negative numbers into zero.
Flatten
Unrolling a stack of small grids into one long list of numbers.
Dense layer
A layer where every input number is connected to every output number. Also called a fully connected layer.
Softmax
A last step that turns a list of scores into chances that add up to 1.
Weight
A number the network learns. Filters and dense layers are made of weights.

The usual stack

Most CNNs follow one pattern. The Stanford course notes (CS231n) say the most common form stacks a few CONV-RELU layers, follows them with POOL layers, and repeats this pattern until the image has been merged spatially to a small size. After that it is common to switch to fully connected layers, and the last one holds the output, such as the class scores. CONV means a convolution layer, RELU is the activation, POOL is pooling and FC is a dense layer.

Text
picture
  -> conv -> ReLU -> pool
  -> conv -> ReLU -> pool     (repeat as needed)
  -> flatten
  -> dense
  -> softmax
  -> chances for each class

CS231n also lists some shapes the pattern can take. One has a single conv layer between every pool layer. Another has two conv layers before each pool, which it calls generally a good idea for larger and deeper networks. The network later in this lesson is the smallest version: one conv, one ReLU, one pool, then the end of the line.

  • Convolution

    Slides filters over the picture and writes down how strongly each patch matches each filter.

  • Pooling

    Shrinks each grid by keeping the biggest number in every small patch, so the next layer has less to handle.

  • Dense and softmax

    Takes the long list of numbers, scores each class, and turns the scores into chances.

Tip

Why not use only dense layers? CS231n gives the reason with real numbers. A picture of 200x200x3 (200 wide, 200 high, 3 colour channels) gives one dense neuron 200*200*3 = 120,000 weights. Filters are small and reused over the whole picture, so a CNN needs far fewer.

What each layer learns

Nobody tells the filters what to look for. Training sets them. Still, a pattern shows up. CS231n says the network will learn filters that activate on an edge of some orientation or a blotch of some color on the first layer, and eventually entire honeycomb or wheel-like patterns on higher layers of the network.

Wikipedia explains why. Each neuron looks at only a small region of the layer before it. Stacking many layers makes the filters respond to larger and larger regions, so the network first creates representations of small parts of the input, then from them assembles representations of larger areas.

Picture reading a word. Your eye first sees short strokes, then letters, then the whole word. Early layers are the strokes. Later layers are the letters and words.

Tracking the shape

The best way to understand a CNN is to follow the size of the data as it moves through. We write a shape as channels x height x width. A grey picture of 8 by 8 is 1 x 8 x 8. Each filter in a conv layer makes one new grid, so two filters make two grids stacked together.

  1. 1

    Input: 1 x 8 x 8

    One grey picture, 8 rows and 8 columns.

  2. 2

    Conv, 2 filters of 3x3: 2 x 6 x 6

    With no padding and stride 1, the size rule from the Padding and Strides lessons gives (8 - 3)/1 + 1 = 6. Two filters, so two grids.

  3. 3

    ReLU: 2 x 6 x 6

    ReLU changes the numbers but not the shape.

  4. 4

    Max pool 2x2: 2 x 3 x 3

    Each 2x2 patch becomes its biggest number, so 6 becomes 3.

  5. 5

    Flatten: 18

    2 x 3 x 3 = 18 numbers in one list.

  6. 6

    Dense: 2

    18 numbers go in, 2 scores come out, one per class.

  7. 7

    Softmax: 2

    The 2 scores become 2 chances that add up to 1.

CS231n adds a few habits for sizing real networks. The input size should be divisible by 2 many times, with common numbers like 32, 64, 96 and 224. Conv layers use small filters (3x3 or at most 5x5) with a stride of 1 and enough zero padding that the layer does not change the size. Pool layers do the shrinking: the most common setting is max pooling with 2x2 patches and a stride of 2, which discards exactly 75% of the activations. An activation is just one number a layer produces.

Run it: a tiny CNN forward pass

Here is the line above as plain Python, with no libraries. A forward pass means sending one picture through the network from start to end. Image 1 is 8 by 8: a bright vertical bar on a dark background. Image 2 is a bright horizontal bar. There are two filters. The first likes vertical edges and the second likes horizontal edges. The dense layer is set by hand: the first filter's numbers vote for class 1 (vertical bar) and the second filter's numbers vote for class 2 (horizontal bar). A real network would learn these numbers.

Python
import math

filters = [
    [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]],
    [[-1, -1, -1], [0, 0, 0], [1, 1, 1]],
]
conv_bias = [0, 0]
w_vertical = [0.01] * 9 + [-0.01] * 9
w_horizontal = [-0.01] * 9 + [0.01] * 9
dense_bias = [0.0, 0.0]

def conv(img, flt, bias):
    n, f = len(img), len(flt)
    return [[bias + sum(img[t + i][l + j] * flt[i][j]
                        for i in range(f) for j in range(f))
             for l in range(n - f + 1)] for t in range(n - f + 1)]

def relu(grid):
    return [[max(0, v) for v in row] for row in grid]

def pool(grid):
    return [[max(grid[r][c], grid[r][c + 1], grid[r + 1][c], grid[r + 1][c + 1])
             for c in range(0, len(grid[0]), 2)] for r in range(0, len(grid), 2)]

def shape(maps):
    return f"{len(maps)} x {len(maps[0])} x {len(maps[0][0])}"

def forward(image, show):
    if show:
        print("input        : 1 x", len(image), "x", len(image[0]))
    maps = [conv(image, f, b) for f, b in zip(filters, conv_bias)]
    if show:
        print("conv 3x3 x2  :", shape(maps))
    maps = [relu(m) for m in maps]
    if show:
        print("ReLU         :", shape(maps))
    maps = [pool(m) for m in maps]
    if show:
        print("max pool 2x2 :", shape(maps))
    flat = [v for m in maps for row in m for v in row]
    if show:
        print("flatten      :", len(flat))
    scores = [sum(w * x for w, x in zip(w_vertical, flat)) + dense_bias[0],
              sum(w * x for w, x in zip(w_horizontal, flat)) + dense_bias[1]]
    if show:
        print("dense        :", len(scores))
    top = max(scores)
    exps = [math.exp(s - top) for s in scores]
    chances = [e / sum(exps) for e in exps]
    if show:
        print("softmax      :", len(chances))
    return chances

vertical_bar = [[0, 0, 0, 9, 9, 0, 0, 0] for _ in range(8)]
horizontal_bar = [[9] * 8 if r in (3, 4) else [0] * 8 for r in range(8)]
names = ["vertical bar", "horizontal bar"]

chances = forward(vertical_bar, True)
print()
for name, c in zip(names, chances):
    print(f"image 1 {name:14s}: {c:.3f}")
chances = forward(horizontal_bar, False)
for name, c in zip(names, chances):
    print(f"image 2 {name:14s}: {c:.3f}")

conv_w = sum(len(f) * len(f[0]) for f in filters) + len(conv_bias)
dense_w = len(w_vertical) + len(w_horizontal) + len(dense_bias)
print()
print("conv weights :", conv_w)
print("dense weights:", dense_w)
print("total        :", conv_w + dense_w)

Output

input        : 1 x 8 x 8
conv 3x3 x2  : 2 x 6 x 6
ReLU         : 2 x 6 x 6
max pool 2x2 : 2 x 3 x 3
flatten      : 18
dense        : 2
softmax      : 2

image 1 vertical bar  : 0.962
image 1 horizontal bar: 0.038
image 2 vertical bar  : 0.038
image 2 horizontal bar: 0.962

conv weights : 20
dense weights: 38
total        : 58

The shapes match the steps above, one line for each layer: 1 x 8 x 8, then 2 x 6 x 6, 2 x 6 x 6, 2 x 3 x 3, then 18, 2 and 2. The vertical bar gets a 0.962 chance of being a vertical bar and 0.038 of being a horizontal bar. The horizontal bar gets the opposite. The two chances in each pair add up to 1, as softmax promises.

Why does it work? The vertical-edge filter fires strongly on the two sides of the vertical bar and gives nothing on a horizontal one, and the horizontal-edge filter does the reverse. ReLU drops the negative responses. Pooling keeps the strongest response in each patch. The dense layer then only has to ask which filter fired more.

Counting the weights

Every weight is something the network has to learn, so counting them tells you how big the job is. In the code above, the conv layer has 2 filters of 3x3, which is 18 weights, plus one bias each, a number added to the result. That makes 20. The dense layer has 18 inputs times 2 outputs, 36 weights, plus 2 biases, which makes 38. ReLU and pooling have nothing to learn. The total is 58.

A conv layer's count does not depend on the size of the picture, because the same filter is reused everywhere. The count is the number of filters, times the filter's height, width and incoming channels, plus one bias per filter.

weights=K×(F×F×C)+K\text{weights} = K \times (F \times F \times C) + K

In words: K filters, each F by F and reaching through all C incoming channels, plus one bias for each filter. CS231n uses the same counting for the first layer of a famous network: 96 filters of 11x11 on a 3-channel picture, 96*11*11*3 = 34,848 unique weights, or 34,944 parameters with the 96 biases.

Now a slightly bigger network, still made up. A 32x32 colour picture goes through a conv layer with 8 filters of 3x3, a pool, a conv layer with 16 filters of 3x3, a pool, then a dense layer to 10 classes. The conv layers are padded so they keep the size, the way CS231n suggests. The code counts the weights in each layer.

Python
# Count the weights in a small, made-up CNN, layer by layer (3 colour channels in, 10 classes out).
def conv_params(filters, size, channels_in):
    return filters * (size * size * channels_in) + filters

def dense_params(n_in, n_out):
    return n_in * n_out + n_out

side, channels = 32, 3
layers = []

p = conv_params(8, 3, channels)
layers.append(("conv 3x3, 8 filters", f"8 x {side} x {side}", p))
channels = 8
side //= 2
layers.append(("max pool 2x2", f"8 x {side} x {side}", 0))
p = conv_params(16, 3, channels)
layers.append(("conv 3x3, 16 filters", f"16 x {side} x {side}", p))
channels = 16
side //= 2
layers.append(("max pool 2x2", f"16 x {side} x {side}", 0))
flat = channels * side * side
layers.append(("flatten", f"{flat}", 0))
layers.append(("dense to 10", "10", dense_params(flat, 10)))

total = 0
for name, shape, p in layers:
    total += p
    print(f"{name:22s} {shape:12s} {p:6d}")
print(f"{'total':22s} {'':12s} {total:6d}")

print()
print("one dense neuron on a 32x32x3 picture:", 32 * 32 * 3, "weights")

Output

conv 3x3, 8 filters    8 x 32 x 32     224
max pool 2x2           8 x 16 x 16       0
conv 3x3, 16 filters   16 x 16 x 16   1168
max pool 2x2           16 x 8 x 8        0
flatten                1024              0
dense to 10            10            10250
total                                11642

one dense neuron on a 32x32x3 picture: 3072

Look at where the weights live. The two conv layers hold 224 and 1,168 weights, together 1,392. The one dense layer holds 10,250, most of the total of 11,642. Dense layers are expensive because every input connects to every output. That is also why conv layers come first and shrink the picture before the dense layer sees it. The last line matches CS231n's example for a 32x32x3 picture: a single dense neuron there has 3072 weights.

How it learns

In the code above we typed the weights in by hand. Real networks learn them. The good news is that nothing new is needed. CS231n says a ConvNet still expresses a single differentiable score function, still has a loss function on the last layer, and that all the tips and tricks developed for learning regular neural networks still apply.

So the training loop is the one you met in the Backpropagation lesson. Show the network a labelled picture. Run the forward pass. Measure how wrong the answer was. Work backwards through the layers to see how much each weight added to the mistake. Nudge every weight a little in the better direction. Repeat for thousands of pictures.

One detail is special to CNNs. A filter's weights are shared across the whole picture, so each shared weight gets a nudge from every spot it was used. CS231n says the gradients are added up across each depth slice and only update a single set of weights per slice.

Tip

CS231n gives practical advice for real projects: don't be a hero. Instead of designing your own architecture, look at whatever architecture currently works best, download a pretrained model and finetune it on your data. You should rarely have to train one from scratch.

Two famous networks

Two networks mark the story. LeNet came first. Wikipedia says it is a series of CNN architectures created at AT&T Bell Laboratories between 1988 and 1998, centered around Yann LeCun, designed for reading small grayscale images of handwritten digits and letters. They were used in ATMs for reading cheques. The best-known version is LeNet-5, and Wikipedia lists its layers: 2 convolutions, 2 subsamplings and 3 fully connected layers, with around 60000 trainable parameters. Its first conv layer turns a 1x32x32 input into 6x28x28 with 5x5 kernels and 156 trainable parameters. That fits our counting rule: 6 filters of 5x5 is 150 weights, plus 6 biases is 156.

AlexNet came later. Wikipedia says it won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error rate of 15.3%, more than 10.8% points better than the runner-up. Top-5 error means the right answer was not among the network's five best guesses. It has eight layers: the first five are convolutional layers, some followed by max-pooling layers, and the last three are fully connected layers. CS231n calls it the first work that popularized Convolutional Networks in Computer Vision, with an architecture very similar to LeNet but deeper, bigger, and with conv layers stacked on top of each other.

  • LeNet

    Built at AT&T Bell Laboratories, 1988 to 1998. Reads handwritten digits. Used in ATMs for reading cheques.

  • AlexNet

    Won the 2012 ImageNet challenge with a top-5 error of 15.3%. Five conv layers and three dense layers.

  • The pattern

    Both are the same stack you just ran, only bigger. Conv and pool layers first, dense layers last.

More pictures from the same pictures

Networks learn better with more examples, and good labelled pictures are hard to come by. Data augmentation is a cheap fix. Wikipedia says input images can be cropped, rotated, or rescaled to create new examples with the same labels as the original training set. A rotated cat is still a cat.

Quick check

Quick check

A 10x10 grey picture goes through a 3x3 conv layer (no padding, stride 1) and then a 2x2 max pool with stride 2. What is the width of the result?

Quick check

Which layer in a typical CNN has the most weights to learn in our 11,642-weight example?

What to remember

  • The stack

    Conv, ReLU and pool, repeated, then flatten, dense and softmax.

  • The shape

    Conv layers keep or slightly shrink the grid, pooling halves it, flatten makes one list, dense and softmax give the class chances.

  • The learning

    Backpropagation, the same as any network. Conv weights are shared, so the count stays small.

A CNN is not magic. It is a line of simple steps, each one small enough to run by hand, as you just did. Next: what convolutional networks are used for in the real world.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.