Machine Learning Foundations / Advanced / 03.32
Applications of CNNs
Where convolutional networks work in the real world: photos, scans, faces, sound and video, and how to reuse one with transfer learning.
12 min readFree
What a convolutional network is used for
The last lesson, Convolutional Neural Network, ended with a promise: what convolutional networks are used for in the real world. Here is the tour.
Think of a convolutional network as a trained pair of eyes. Once it has learned to spot edges, corners and shapes, you can point those eyes at many jobs: sorting holiday photos, reading a check, looking at a scan. The eyes stay much the same. The job changes.
By the end you will know the main jobs, the two or three that need care, and the trick most people use to get a working network without training one from scratch. You will also run a tiny version of that trick yourself.
Key words
- Image classification
- Looking at a whole picture and naming what is in it, such as "cat" or "bridge".
- Object detection
- Finding each thing in a picture and drawing a box around it.
- Segmentation
- Sorting every pixel of a picture into a group, such as road, sky or person.
- Pretrained network
- A network that someone already trained, usually on a huge set of pictures, and shared.
- Transfer learning
- Re-using what a network learned on one task to do better on a related task.
- Freezing
- Locking a layer's numbers so training cannot change them.
Three jobs on a picture
Most picture work with convolutional networks comes down to three questions. Wikipedia's list of uses names image classification and image segmentation, and its section on image recognition names object detection. Each gets its own lesson later in this course.
What is in this picture?
Image classification. One answer for the whole picture. The next lesson covers it.
Where is each thing?
Object detection. A box and a name for every item, like labelling the people in a group photo.
Which pixels belong to what?
Segmentation. Colouring in the picture by group, like a map with a colour for every region.
Wikipedia says a big test called the ImageNet Large Scale Visual Recognition Challenge is a benchmark in object classification and detection, with millions of images and hundreds of object classes. In the 2014 run, almost every highly ranked team used a convolutional network. Its words: "That performance of convolutional neural networks on the ImageNet tests was close to that of humans."
Wikipedia adds a fair warning. The best algorithms still struggle with small or thin objects, such as a small ant on a stem of a flower. They also have trouble with images distorted by filters, which rarely trouble people. People, in turn, are not good at fine-grained sorting, such as the exact breed of a dog, and the networks handle that well.
Tip
A network is not better or worse than a person. It is good at different things. Remember that when you decide where to trust one.
Medical images
Wikipedia lists medical image analysis among the uses of convolutional networks, and its page on transfer learning names medical imaging too. A scan is a grid of numbers, just like a photo, so the same sliding filters apply.
The sources fetched for this lesson do not give figures for how well these systems do on real patients, so none are quoted here. If you read a claim about a medical tool, look for the study behind it.
Self-driving perception
A car with no driver has to see lanes, signs and people. Our sources say little about how that is built, and this lesson does not guess. Two facts do appear. Wikipedia says human-readable explanations are required for critical systems such as self-driving cars, and it describes methods that highlight the parts of a picture that most shaped the network's answer. The adversarial section below shows a case where a stop sign was misread.
Tip
When a mistake can hurt someone, "it works" is not enough. You also need to see why it answered the way it did.
Faces: useful and contested
Wikipedia reports that a many-layered convolutional network, in 2015, could spot faces from a wide range of angles, including upside down and partly hidden. It was trained on 200,000 images with faces and a further 20 million without.
Spotting a face is one thing. Naming whose face it is raises harder questions. Wikipedia's article on facial recognition says its use "has also raised controversy, with claims that the systems violate citizens' privacy, commonly make incorrect identifications, encourage gender norms and racial profiling, and do not protect important biometric data."
On bias, the same article says that facial recognition systems often show lower accuracy for people with non-Eurocentric facial features, and links this to groups that are overrepresented in the training data. It also reports that Meta shut down its Facebook face-recognition system in 2021, deleting the face-scan data of more than one billion users.
The lesson for a designer or builder: ask who is in the training pictures, who is left out, and whether the people shown on the screen agreed to be recognised.
Reading text in pictures (OCR)
Optical character recognition, or OCR, turns a picture of typed, handwritten or printed text into text a computer can edit and search. Wikipedia's OCR page says it is widely used for passports, invoices, bank statements, receipts and business cards.
Convolutional networks were reading text early. Wikipedia says a 1989 system recognised hand-written ZIP Code numbers, and that LeNet-5, published in 1998, was built into check-reading systems fielded in several American banks since June 1996, reading millions of checks per day.
Video, sound and other data
A video is pictures in a row, so time is one more direction to slide across. Wikipedia describes one approach that treats space and time as equal dimensions and convolves over both. It notes there is relatively little work on video compared with still images.
Convolution is not only for pictures. Wikipedia says the time-delay network, introduced in 1987 for phoneme recognition, was a 1-D convolutional network that slid along the time axis of speech, and that a 1990 variant worked on spectrograms. A spectrogram is a picture of sound, time across and pitch up the side. It also says convolutional networks have been tried on text, and that a network with 1-D convolutions was used to detect anomalies in a time series.
Wikipedia's list of uses also includes recommender systems, brain-computer interfaces and financial time series. The pattern: if nearby values belong together, a sliding filter has something to find.
Transfer learning: borrow a trained network
Training a big network from nothing needs a lot of pictures. Wikipedia says convolutional networks usually need a large amount of training data to avoid overfitting. Most people do not have that. So they borrow.
Think of hiring an experienced chef to cook your restaurant's menu. They already know knife skills and heat. You only teach them your recipes. Wikipedia defines transfer learning this way: knowledge learned from a task is re-used to boost performance on a related task, and its example is that knowledge gained learning to recognise cars could help recognise trucks.
The TensorFlow tutorial shows the usual recipe. It starts from MobileNet V2, a network pre-trained on ImageNet, "a large dataset consisting of 1.4M images and 1000 classes", and uses it to tell cats from dogs. It says the base network already holds features that are generically useful for classifying pictures, while its final classification part is specific to the original classes.
- 1
Pick a pretrained network
Choose one trained on a large set of pictures that resemble yours.
- 2
Cut off its last layer
That layer knows the old task's answers. Yours has different ones.
- 3
Freeze the rest
The tutorial says it is important to freeze the base before you train, so its weights are not updated.
- 4
Add a small new head
A new classifier on top, trained from scratch on your pictures.
- 5
Train only the head
Few pictures and little time are enough, because the hard seeing is already done.
- 6
Optionally fine-tune
The tutorial then unfreezes a few top layers of the base and trains them together with the head, to improve performance.
Wikipedia's page on convolutional networks says this lets them "successfully be applied to problems with tiny training sets."
Tip
Transfer is not guaranteed. Wikipedia notes a 2020 paper reporting that pre-training can hurt accuracy. Always measure on your own pictures.
A tiny transfer-learning demo
Here is the recipe in miniature, in plain Python with no libraries. The feature extractor is two hand-made edge filters (the kind you saw in Convolution) followed by 2 by 2 max pooling, which keeps the biggest number in each small block. It never changes. It turns each made-up 6 by 6 picture into 8 numbers. On top sits a tiny head, a logistic regression that turns the 8 numbers into a yes-or-no answer. It learns to say whether a picture shows a vertical bar or a horizontal bar. Each picture gets a little random noise.
import math, random
rng = random.Random(7)
def make_pattern(kind):
grid = [[0.0] * 6 for _ in range(6)]
pos = rng.randrange(1, 5)
for i in range(6):
if kind == 1:
grid[i][pos] = 1.0 # vertical bar
else:
grid[pos][i] = 1.0 # horizontal bar
return [[v + rng.uniform(-0.3, 0.3) for v in row] for row in grid]
def make_set(n):
data = []
for i in range(n):
kind = i % 2
data.append((make_pattern(kind), kind))
return data
# The frozen feature extractor: two hand-made edge filters, then 2x2 max pooling.
VERT = [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]]
HORZ = [[-1, -1, -1], [0, 0, 0], [1, 1, 1]]
def convolve(img, k):
out = []
for r in range(4):
row = []
for c in range(4):
s = sum(img[r + i][c + j] * k[i][j] for i in range(3) for j in range(3))
row.append(abs(s))
out.append(row)
return out
def pool(m):
return [max(m[r][c], m[r][c + 1], m[r + 1][c], m[r + 1][c + 1])
for r in (0, 2) for c in (0, 2)]
def features(img):
return pool(convolve(img, VERT)) + pool(convolve(img, HORZ))
# The small head: logistic regression on the 8 numbers.
def predict(w, b, x):
z = b + sum(wi * xi for wi, xi in zip(w, x))
return 1 / (1 + math.exp(-z))
def accuracy(w, b, rows):
hits = sum((predict(w, b, x) > 0.5) == (y == 1) for x, y in rows)
return hits / len(rows)
train = [(features(img), y) for img, y in make_set(40)]
test = [(features(img), y) for img, y in make_set(20)]
w = [rng.uniform(-0.5, 0.5) for _ in range(8)]
b = 0.0
saved = [row[:] for row in VERT + HORZ]
print("numbers the head can change:", len(w) + 1)
print("before training the head: train", accuracy(w, b, train), "test", accuracy(w, b, test))
for epoch in range(200):
for x, y in train:
err = predict(w, b, x) - y
w = [wi - 0.1 * err * xi for wi, xi in zip(w, x)]
b -= 0.1 * err
print("after training the head: train", accuracy(w, b, train), "test", accuracy(w, b, test))
print("filter numbers changed:", saved != [row[:] for row in VERT + HORZ])Output
numbers the head can change: 9 before training the head: train 0.575 test 0.5 after training the head: train 1.0 test 1.0 filter numbers changed: False
Before training, the head's random starting numbers do about as well as a coin flip: 0.575 on the training pictures and 0.5 on 20 new test pictures. After training only 9 numbers (8 weights and a bias), it gets 1.0 on both. The last line shows the filters did not change at all.
This is a toy. The bars are easy and the test set is small, so treat the 1.0 as a demonstration of the recipe, not as proof that it works on real data.
Limits to keep in mind
Hungry for data
Training from scratch needs a large amount of training data. Transfer learning eases this but does not remove the need to test.
Easy to fool
Wikipedia says that from 2014, researchers showed deep networks could be fooled by adversaries. In one case, researchers changed a stop sign's appearance so an autonomous vehicle read it as a merge or speed limit sign.
Fragile in the wild
The same article says such attacks are harder outside the lab: a small rotation or slight lighting change can destroy the effect.
An adversarial example is a picture changed in a small, careful way so that the network gives the wrong answer. The fix is not to avoid networks. It is to test them against tricks, odd lighting and the people they will meet.
Check yourself
Quick check
In transfer learning with a pretrained network, which part do you usually train on your own pictures?
Quick check
A spectrogram is a picture of sound. Which fact from this lesson does that connect to?
Quick check
Which is a concern Wikipedia raises about face recognition?
Recap
1. Many jobs
Classification, detection and segmentation, plus scans, faces, text, video, sound and time series.
2. Borrow, then adjust
Reuse a pretrained network, freeze it, train a small new head, and fine-tune if needed.
3. Stay careful
Check for bias and privacy, test against tricks, and measure on your own data.
Next: image classification, naming what is in a picture.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.