Machine Learning Foundations / Advanced / 03.29
Strides
How far the filter jumps each step, and how a bigger jump makes the output smaller and the work lighter.
13 min readFree
How far does the filter jump?
The last lesson ended with a question: how far does the filter jump? Before this, the filter always slid one cell at a time. That is only one choice. The jump size has a name, the stride, and changing it changes how big the result is and how much work the computer does.
Think of walking along a row of paving stones. You can step on every stone, or you can step on every second stone. Both get you to the end of the road. The second way takes fewer steps and you leave some stones untouched. A filter sliding over a picture works the same way.
Key words
- Filter
- A small grid of weights that slides over a picture and measures how strongly each patch matches one pattern. Also called a kernel.
- Stride
- How many cells the filter moves between one step and the next. Stride 1 moves one cell. Stride 2 moves two.
- Padding
- A border of extra cells, usually zeros, added around the picture before the filter slides. Padding 1 means one ring of zeros.
- Output
- The grid of results the filter produces, one number per stop. Also called a feature map.
- Floor
- Rounding a number down to the whole number below it. The floor of 4.5 is 4.
- Spot
- One place the filter sits. The filter makes one output number for each spot.
What a stride is
Here is how CS231n, the Stanford course notes on convolutional networks, puts it. It says you must specify the stride with which we slide the filter. Then the two common cases: when the stride is 1 then we move the filters one pixel at a time, and when the stride is 2 the filters jump 2 pixels at a time as we slide them around.
Wikipedia says the same in other words: the stride is the number of pixels that the analysis window moves on each iteration, and a stride of 2 means that each kernel is offset by 2 pixels from its predecessor.
Wikipedia also says what a bigger stride does. A greater stride means smaller overlap of receptive fields and smaller spatial dimensions of the output volume. A receptive field is just the patch of the picture one output number looks at. With stride 1 those patches overlap a lot, because each patch is shifted by one cell. With stride 2 they overlap less, and there are fewer of them.
Slide it by hand
Take one row of 7 cells and a filter 3 cells wide. No padding yet. Count the spots where the filter fits without hanging off the edge.
- 1
Stride 1
The filter starts at cell 0. Then it moves to cell 1, 2, 3 and 4. After that it would hang off the end. That is 5 spots.
- 2
Stride 2
The filter starts at cell 0. Then it jumps to cell 2, then cell 4. A jump to cell 6 would hang off the end. That is 3 spots.
- 3
Compare
Same row, same filter. Stride 1 gives 5 results, stride 2 gives 3. The cells the filter skipped over are still read, because the filter is 3 wide. The filter just starts fewer times.
Tip
A stride bigger than the filter's width would skip cells completely, so no output number would ever see them. In practice strides stay small. Wikipedia says a stride of 3 or more is rare in practice, and CS231n agrees.
The output-size formula
You do not have to count by hand. CS231n gives a formula. It uses four numbers: W is the width of the input, F is the width of the filter, S is the stride, and P is the amount of zero padding on each side. CS231n says the correct formula for calculating how many neurons fit is (W - F + 2P)/S + 1.
In words: take the input width, subtract the filter width, and add the padding twice, once for each side. Divide by the stride. Round down to a whole number. Add 1.
Bigger stride
The output gets smaller. Going from stride 1 to stride 2 cuts the width roughly in half.
More padding
The output gets bigger, because the picture is wider before the filter starts.
Wider filter
The output gets smaller, because the filter hangs off the edge sooner.
CS231n also gives a handy case. It says that setting zero padding to be P = (F - 1)/2 when the stride is S = 1 ensures that the input volume and output volume will have the same size spatially. For a 3x3 filter that means padding 1. Keras has the same idea under another name: its padding setting "same" with a stride of 1 gives an output the same size as the input, and "valid" means no padding.
Run it: a 7x7 grid and a 3x3 filter
Time to check the formula against real sliding. The code below builds a 7x7 grid of small numbers and a 3x3 filter. The filter has 1 in its left column, 0 in the middle and -1 in the right column, so it reacts where the numbers change from left to right. The function slides the filter with a given stride and padding, and for each spot multiplies the nine numbers under the filter by the nine weights and adds them up. It prints each output grid, its size, and what the formula predicts.
grid = [[(r * 3 + c * 2) % 5 for c in range(7)] for r in range(7)]
filt = [[1, 0, -1],
[1, 0, -1],
[1, 0, -1]]
def convolve(grid, filt, stride, pad):
n, f = len(grid), len(filt)
size = n + 2 * pad
padded = [[0] * size for _ in range(size)]
for r in range(n):
for c in range(n):
padded[r + pad][c + pad] = grid[r][c]
out = []
for top in range(0, size - f + 1, stride):
row = []
for left in range(0, size - f + 1, stride):
total = 0
for i in range(f):
for j in range(f):
total += padded[top + i][left + j] * filt[i][j]
row.append(total)
out.append(row)
return out
def formula(w, f, stride, pad):
return (w - f + 2 * pad) // stride + 1
print("input grid:")
for row in grid:
print(" ".join(f"{v:2d}" for v in row))
for stride, pad in [(1, 0), (2, 0), (1, 1), (2, 1)]:
out = convolve(grid, filt, stride, pad)
size = len(out)
want = formula(7, 3, stride, pad)
print()
print(f"stride {stride}, padding {pad}: output {size}x{size}, formula says {want}, match: {size == want}")
for row in out:
print(" ".join(f"{v:3d}" for v in row))Output
input grid: 0 2 4 1 3 0 2 3 0 2 4 1 3 0 1 3 0 2 4 1 3 4 1 3 0 2 4 1 2 4 1 3 0 2 4 0 2 4 1 3 0 2 3 0 2 4 1 3 0 stride 1, padding 0: output 5x5, formula says 5, match: True -2 -2 -2 3 3 3 -2 -2 -2 3 3 3 -2 -2 -2 -2 3 3 -2 -2 -2 -2 3 3 -2 stride 2, padding 0: output 3x3, formula says 3, match: True -2 -2 3 3 -2 -2 -2 3 -2 stride 1, padding 1: output 7x7, formula says 7, match: True -2 -3 -3 2 2 2 3 -5 -2 -2 -2 3 3 4 -4 3 -2 -2 -2 3 8 -8 3 3 -2 -2 -2 7 -7 -2 3 3 -2 -2 6 -6 -2 -2 3 3 -2 5 -2 -3 -3 2 2 2 3 stride 2, padding 1: output 4x4, formula says 4, match: True -2 -3 2 3 -4 -2 -2 8 -7 3 -2 6 -2 -3 2 3
All four sizes match the formula: 5, 3, 7 and 4. Look at the stride 2 results. The stride 2, padding 0 grid has 3 rows of 3. It is the same shape as CS231n's example. The first row of that grid, -2 -2 3, is every second number from the first row of the stride 1 grid, -2 -2 -2 3 3. That makes sense, because stride 2 simply visits every second spot of the stride 1 walk.
Padding changes the edge numbers. With padding 1 the filter sees some zeros at the border, so the first and last columns of the output are new values (-3, -5, -4 and so on) that the unpadded runs never produce. The middle of the padded grid matches the unpadded grid, because there the filter sees the same cells as before.
When the numbers do not fit evenly
Sometimes the division does not come out as a whole number. CS231n gives an example: an input with W = 10, no padding, a filter of F = 3, and a stride of S = 2. Then (10 - 3 + 0)/2 + 1 = 4.5. You cannot have half a spot. CS231n says that means the neurons do not fit neatly and symmetrically across the input, and calls this setting invalid.
It also says what a library might do about it: throw an exception, zero pad the rest to make it fit, or crop the input to make it fit. So it depends on the tool. The floor in our formula stands for the crop option: drop the leftover edge. The code below shows what that means for this example.
# When the numbers do not fit: width 10, filter 3, stride 2, no padding.
w, f, s, p = 10, 3, 2, 0
print("formula before rounding:", (w - f + 2 * p) / s + 1)
print("formula rounded down:", (w - f + 2 * p) // s + 1)
starts = list(range(0, w - f + 1, s))
print("filter starts at:", starts)
print("last start + filter width:", starts[-1] + f, "of", w)
print("left-over cells:", w - (starts[-1] + f))Output
formula before rounding: 4.5 formula rounded down: 4 filter starts at: [0, 2, 4, 6] last start + filter width: 9 of 10 left-over cells: 1
The filter starts at cells 0, 2, 4 and 6. The last start covers cells 6, 7 and 8. It reaches cell 9 only if there were one more spot, but the next jump would put the filter at cell 8, hanging off the end. So cell 9, the last one, is never read. The floor turns 4.5 into 4, and one cell is lost at the edge.
There is a plain fix on your side: pick sizes that fit. A bit of padding, a different filter or a different input size can make the division exact.
Tip
Before you build a network, run the formula for each layer. If the division has a remainder, you are about to lose an edge. It costs ten seconds and saves a long search for a size mismatch.
A cheaper way to shrink
Shrinking a picture is useful. A smaller grid means fewer numbers for the next layer to handle. The next lesson, Pooling, is the classic way to do it: it squashes small patches down to one number each. CS231n describes the most common form as a pooling layer with filters of size 2x2 applied with a stride of 2, which discards 75% of the activations.
That gives a shortcut. Some researchers skipped the pooling step. CS231n says one paper, Striving for Simplicity: The All Convolutional Net, proposes to discard the pooling layer in favor of an architecture that only consists of repeated CONV layers, and to reduce the size of the representation they suggest using larger stride in CONV layer once in a while. In other words, let the convolution do the shrinking by jumping further.
Why might that be cheaper? Count the work. A convolution does 9 multiplications at every spot for a 3x3 filter. Fewer spots means fewer multiplications. The code below counts the spots and the multiplications for one filter, on the small 7x7 grid and on a 224x224 picture, a common picture size in image networks.
def spots(w, f, s, p):
n = (w - f + 2 * p) // s + 1
return n * n
for w in (7, 224):
for s in (1, 2):
n = spots(w, 3, s, 1)
print(f"{w}x{w}, stride {s}, padding 1: {n} spots, {n * 9} multiplications")
print("stride 2 does", spots(224, 3, 1, 1) // spots(224, 3, 2, 1), "times fewer multiplications on 224x224")Output
7x7, stride 1, padding 1: 49 spots, 441 multiplications 7x7, stride 2, padding 1: 16 spots, 144 multiplications 224x224, stride 1, padding 1: 50176 spots, 451584 multiplications 224x224, stride 2, padding 1: 12544 spots, 112896 multiplications stride 2 does 4 times fewer multiplications on 224x224
On the 7x7 grid, stride 2 does 144 multiplications where stride 1 does 441. On the 224x224 picture, it does 112,896 where stride 1 does 451,584, exactly 4 times fewer. The reason is simple: a stride of 2 halves the number of spots along the width and along the height, and 2 times 2 is 4.
Compare with doing it the other way round. Convolve with stride 1, and only then pool. The convolution has already done its full 451,584 multiplications before pooling throws most of the result away. Shrinking inside the convolution skips that work from the start. These counts are for one filter on one layer, not for a whole network, and they say nothing about accuracy.
On accuracy, be careful. CS231n's own advice for ordinary layers goes the other way: it asks why use a stride of 1 in a CONV layer, and answers that smaller strides work better in practice. A bigger stride is a trade. You gain speed and a smaller output. You give up some detail, because the filter looks at fewer spots.
Quick checks
Quick check
An input is 9 cells wide, the filter is 3 wide, the stride is 2 and there is no padding. How wide is the output?
Quick check
A 32-wide input, a 5-wide filter, padding 2 and stride 2. What is the output width?
Quick check
Why can a bigger stride make a network faster?
What to remember
Stride
How many cells the filter jumps each step. Stride 1 visits every spot, stride 2 every second one, and a bigger stride gives a smaller output.
The formula
Output width is the floor of (W - F + 2P)/S, plus 1. Check it before you build, so a leftover edge does not surprise you.
The trade
A bigger stride cuts the work and shrinks the output, as a cheaper alternative to pooling in some designs. The cost is less detail.
Next: pooling, shrinking a picture while keeping what matters.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.