Machine Learning Foundations / Advanced / 03.33
Image Classification
Give a model a picture and it names what is in it. See the steps, the famous datasets, how to score it, and a tiny classifier you can run.
13 min readFree
Naming what is in a picture
The last lesson, Applications of CNNs, ended with a promise: image classification, naming what is in a picture. This is that lesson. It is the most basic job a picture-reading network does, and almost every bigger vision job grows out of it.
Think of a sorting table at a post office. Each letter lands in one box: local, national or abroad. Image classification is the same. Each picture lands in one box, and the box has a name such as cat, truck or the digit 7.
By the end you will know the steps from a pile of photos to a working model, three famous practice datasets, how to score the result, and what to do when the data is small. You will also run a tiny classifier of your own.
Key words
- Image classification
- Giving a picture one label from a fixed list of choices, such as cat, dog or hat.
- Label
- The right answer attached to a picture, such as "cat". People usually write labels by hand.
- Class
- One of the choices on the list. "Cat" is a class.
- Training set
- The labelled pictures a model learns from.
- Test set
- Labelled pictures kept hidden during learning, used afterwards to score the model on pictures it has never seen.
- Normalise
- Rescale the numbers in a picture to a small, tidy range, such as 0 to 1.
- Data augmentation
- Making extra training pictures by changing the ones you have: flipping, cropping, turning.
- Overfitting
- Learning the training pictures so closely that new pictures go badly.
One label for one picture
Stanford's CS231n notes define the task as assigning an input image one label from a fixed set of categories. They add that it is one of the core problems in computer vision, and that many other tasks, such as object detection and segmentation, can be reduced to it.
Their example is a picture of a cat that is 248 pixels wide and 400 pixels tall, with three colour channels. That is 248 times 400 times 3, which is 297,600 numbers, each from 0 (black) to 255 (white). The job is to turn those 297,600 numbers into one word: "cat". You met this idea in Convolution: to a computer, a picture is a grid of numbers.
Wikipedia's computer vision article lists the neighbours of this task. Object recognition, also called object classification, names objects. Detection scans the image for specific objects along with their locations. So classification answers "what is it?" and detection answers "what is it, and where?". The next lesson covers the second one.
It is hard because CS231n's list of troubles is long: the object can be seen from many angles, partly hidden, or lit very differently, and the background can blend in.
Wikipedia also covers the case where one picture can carry more than one label. In multi-label classification, several non-exclusive labels may be assigned to each item. It describes multi-class classification as the single-label problem: exactly one class out of several. A photo that shows a beach and a dog is a multi-label case. A photo of one handwritten digit is a multi-class case. This lesson stays with one label per picture.
The pipeline, step by step
CS231n describes the whole thing in three words: input, learning, evaluation. You start with N images, each labelled with one of K classes. In practice there are a few more small steps in between.
- 1
1. Collect and label
Gather pictures of every class and attach the right label to each one.
- 2
2. Resize and normalise
Make all pictures the same size and squeeze the numbers into a small range.
- 3
3. Split
Set some pictures aside for checking. The model never learns from them.
- 4
4. Train
Show the model the training pictures. It adjusts itself to get more of them right.
- 5
5. Evaluate
Score the model on the held-back pictures, using the numbers from the Accuracy and Confusion Matrix lessons.
- 6
6. Predict
Give the finished model a brand-new picture and read off its answer.
Collect and label
CS231n calls this the data-driven approach. Nobody can write rules for what a cat looks like, so, as with a child, you show the computer many examples of each class and let it learn what each class looks like.
Labels cost real effort. The TensorFlow tutorial, for example, works with about 3,700 photos of flowers in five folders: daisy, dandelion, roses, sunflowers and tulips. Each folder name is the label for every photo inside it. Wikipedia reports that ImageNet needed 49 thousand workers from 167 countries to filter and label over 160 million candidate images. Mistakes creep in: Wikipedia notes the original MNIST data contains at least 4 wrong labels.
Resize and normalise
A network expects every picture to have the same shape, so photos are resized to one size first. The TensorFlow tutorial then notes that the RGB channel values are in the 0 to 255 range, which is not ideal for a neural network: in general you should seek to make your input values small. It standardises them to the 0 to 1 range.
# Normalise: squeeze pixel values from 0-255 into 0-1.
row = [0, 64, 128, 255]
print([round(p / 255, 2) for p in row])
# Split: 80% to train, the rest to check (the same 3,670 photos the TensorFlow tutorial uses).
photos = 3670
train = int(photos * 0.8)
print("train:", train, "check:", photos - train)Output
[0.0, 0.25, 0.5, 1.0] train: 2936 check: 734
Every value is divided by 255. Black stays 0, white becomes 1, and everything else lands in between.
Split before you train
The TensorFlow tutorial uses 80% of the images for training and 20% for validation. The code above shows the sum for its 3,670 photos: 2,936 to learn from and 734 to check with. The check set must stay unseen: a student who has read the exam questions in advance proves nothing.
Three datasets everyone uses
Researchers compare models on shared collections, the way runners compare times on the same track. Three come up again and again.
MNIST: handwritten digits
Wikipedia says it holds 60,000 training images and 10,000 testing images of handwritten digits. Each digit was size-normalised to fit a 20 by 20 box, turned into grey tones and placed in a 28 by 28 picture. Ten classes, 0 to 9.
CIFAR-10: small colour photos
60,000 colour images, each 32 by 32 pixels, in 10 classes: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks, with 6,000 images of each. Wikipedia notes the low resolution lets researchers quickly try different methods.
ImageNet: the big one
More than 14 million images in a large visual database. Its yearly challenge used 1,000 classes. The 1,000-class version holds 1,281,167 training images, 50,000 validation images and 100,000 test images.
Wikipedia tells the ImageNet story in numbers. In 2012 a convolutional network called AlexNet reached a top-5 error of 15.3%, a result that was more than 10.8 percentage points lower than the runner-up. In the next couple of years top-5 accuracy grew to above 90%. The computer vision article says performance of convolutional networks on the ImageNet tests is now close to that of humans.
Scoring a classifier
Once a model has answers for the held-back pictures, you score it. Three tools matter most here.
Top-1 and top-5 accuracy
Plain accuracy, from the Accuracy lesson, is the share of pictures where the model's first choice is the right label. For image classification this is called top-1 accuracy.
With 1,000 classes, some are close neighbours, such as two breeds of dog. So the ImageNet challenge also reports top-5 accuracy: the picture counts as right if the true label is anywhere among the model's five best guesses. A top-5 error of 15.3% means the right label was missing from the five guesses in 15.3% of pictures.
The confusion matrix
One accuracy number hides which classes get mixed up. The Confusion Matrix lesson showed the fix: a table with the true class down the side and the predicted class across the top. The diagonal holds the right answers. Anything off the diagonal is a mix-up, and it tells you exactly which two classes the model confuses, for instance cats taken for dogs.
Class imbalance
Suppose 95 of every 100 pictures in a factory camera set show a good part and 5 show a crack. A model that always says "good" scores 95% accuracy and finds no cracks. The Accuracy lesson called this the accuracy paradox. For rare classes, read that class's row of the confusion matrix, plus precision and recall.
Data augmentation: more pictures from the ones you have
Real pictures are costly to collect, so a small set is common. Data augmentation stretches it. The TensorFlow tutorial describes it as generating additional training data from your existing examples by augmenting them using random transformations that yield believable-looking images.
Common changes are a flip (mirror the picture left to right), a crop (cut out a part), and a rotation (turn it a little). A flipped, shifted or tilted cat is still a cat. Showing the model these variations teaches it that the label stays the same.
Pick changes that keep the label true. Flipping a photo of a cat is safe. A flipped digit 2 is no longer a 2, and a 6 turned upside down is a 9.
Tip
Augment the training pictures only. Keep the test set exactly as it comes.
Overfitting on small datasets
With few pictures, a model can learn them by heart, like a student who memorises last year's answers. The TensorFlow tutorial states that overfitting generally occurs when there are a small number of training examples.
The signs are in the scores. In the tutorial's plots the training accuracy keeps rising while the validation accuracy stalls around 60%. It calls that noticeable gap between training and validation accuracy a sign of overfitting. So watch both numbers. A model that is great on the pictures it studied and weak on the held-back ones has memorised them.
The tutorial fights it with data augmentation, from the last section, and with dropout.
Worked example: three shapes on a 5 by 5 grid
Time to build one. We invent tiny 5 by 5 pictures of three shapes: a diagonal line, a cross and a hollow square. Each pixel is 1 for ink and 0 for paper. We add random noise to every pixel, like static on a bad photo, with a fixed seed so the run repeats.
The classifier is the simplest we can think of, nearest-centroid. For every class it averages its training pictures into one blurry average picture, the centroid. A new picture gets the label of the closest average, measured by summing squared pixel differences. There is no network, but the pipeline is all here.
We set up a trap on purpose. Training has only 4 pictures per class, and every training line leans one way, like a backslash. The test set has 30 pictures per class, and half of the lines lean the other way, like a slash. Real data is often lopsided like this. We then train twice, once as is and once with flip augmentation, which adds a mirrored copy of every training picture. Finally we count top-1 and top-2 accuracy.
import random
# 5x5 pictures: 1 = ink, 0 = paper. Three shapes.
SHAPES = {
"line": ["10000", "01000", "00100", "00010", "00001"],
"cross": ["00100", "00100", "11111", "00100", "00100"],
"square": ["11111", "10001", "10001", "10001", "11111"],
}
NAMES = list(SHAPES)
def to_pixels(rows):
return [int(ch) for row in rows for ch in row]
def flip(pixels):
# mirror left to right
return [pixels[r * 5 + (4 - c)] for r in range(5) for c in range(5)]
def noisy(pixels, rng):
return [min(1.0, max(0.0, p + rng.gauss(0, 0.8))) for p in pixels]
rng = random.Random(7)
train, test = [], []
for name in NAMES:
base = to_pixels(SHAPES[name])
for _ in range(4):
train.append((noisy(base, rng), name)) # training: lines lean one way only
for i in range(30):
shown = flip(base) if (name == "line" and i % 2) else base
test.append((noisy(shown, rng), name)) # test: lines lean both ways
def centroids(data):
out = {}
for name in NAMES:
rows = [x for x, y in data if y == name]
out[name] = [sum(col) / len(rows) for col in zip(*rows)]
return out
def distance(a, b):
return sum((p - q) ** 2 for p, q in zip(a, b))
def predict(model, x):
return min(NAMES, key=lambda n: distance(model[n], x))
def report(title, data):
model = centroids(data)
matrix = {t: {p: 0 for p in NAMES} for t in NAMES}
for x, y in test:
matrix[y][predict(model, x)] += 1
right = sum(matrix[n][n] for n in NAMES)
print(f"{title}: test accuracy {right}/{len(test)} = {right / len(test):.3f}")
print("true \\ predicted " + " ".join(f"{n:>6}" for n in NAMES))
for t in NAMES:
print(f"{t:<17} " + " ".join(f"{matrix[t][p]:>6}" for p in NAMES))
report("no augmentation", train)
print()
augmented = train + [(flip(x), y) for x, y in train]
report("with flip augmentation", augmented)
# top-1 and top-2: is the right label the first guess, or among the first two?
model = centroids(augmented)
top1 = top2 = 0
for x, y in test:
ranked = sorted(NAMES, key=lambda n: distance(model[n], x))
top1 += ranked[0] == y
top2 += y in ranked[:2]
print()
print(f"top-1 accuracy {top1 / len(test):.3f}, top-2 accuracy {top2 / len(test):.3f}")Output
no augmentation: test accuracy 78/90 = 0.867 true \ predicted line cross square line 27 3 0 cross 3 26 1 square 4 1 25 with flip augmentation: test accuracy 82/90 = 0.911 true \ predicted line cross square line 30 0 0 cross 4 25 1 square 2 1 27 top-1 accuracy 0.911, top-2 accuracy 0.978
Read the first matrix. Without augmentation the model got 78 of 90 right, which is 0.867. Row "line" shows 27 right and 3 labelled as a cross: the slash-shaped lines it had never seen.
With the flip, the score rose to 82 of 90, or 0.911. All 30 lines are now found. The gain is small and came from one fixed seed, so do not read it as a promise. Look at the cross row though: 4 crosses were still taken for lines, because noise this heavy blurs thin shapes together. Augmentation fixed the problem it was built for and left the rest alone.
The last line shows the top-k idea. The model's first guess was right for 0.911 of the pictures, and the right label was among its first two guesses for 0.978.
Tip
Change the 0.8 noise to 0.35 and the 4 pictures per class to 10, then run it again. Everything is right in both runs and the gap disappears, so augmentation helps most when data is scarce or noisy.
Check yourself
Quick check
A model scores 99% on its training pictures and 62% on the held-back ones. What is the likely problem?
Quick check
A photo shows both a bicycle and a dog, and you want both named. Which task is this?
Recap
- 1
1. One label
Image classification gives each picture one label from a fixed list.
- 2
2. Pipeline
Collect and label, resize and normalise, split, train, evaluate, predict.
- 3
3. Datasets
MNIST has digits, CIFAR-10 has small colour photos, ImageNet has millions of pictures in 1,000 classes.
- 4
4. Scores
Top-1 and top-5 accuracy, plus a confusion matrix to see which classes mix up. Watch for rare classes.
- 5
5. Small data
Augmentation (flip, crop, rotate) makes extra pictures that keep the label true. Watch the training-versus-validation gap for overfitting.
Next: image and video recognition, finding things and following them.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.