Machine Learning Foundations / Advanced / 03.35
Image Segmentation
Instead of one label for a whole picture, give every pixel its own label: how networks draw outlines, and how we score them.
14 min readFree
From one label to a label for every pixel
The last lesson, Image & Video Recognition, ended with a promise: image segmentation, labelling every pixel. Here it is. Recognition says what is in a picture. Segmentation says exactly where, down to the last dot.
Think of a colouring book. Recognition is the sentence on the cover: this book has a cat in it. Segmentation is you with a pencil, colouring the cat in and staying inside the lines. Every part of the page ends up with a colour, and the colour says what that part is.
By the end you will know the three kinds of segmentation, the shrink-then-grow shape most networks use, what a skip connection does, how to score a result without being fooled, and you will have run a small segmenter yourself.
Key words
- Pixel
- One tiny dot of a picture. A grey picture gives each pixel one number, from dark to bright.
- Segmentation
- Giving every pixel a label, so pixels with the same label belong together.
- Mask
- A grid the same size as the picture that holds one label per pixel. Think of the colouring book page with the colours kept and the picture gone.
- Class
- A kind of thing, such as road, person or background.
- Encoder
- The first half of the network. It shrinks the picture while it works out what is in it.
- Decoder
- The second half. It grows the result back to the size of the picture so it can label each pixel.
- Skip connection
- A shortcut that carries fine detail from the encoder straight across to the decoder.
- IoU
- Intersection over union: how much two shapes overlap, as a share of everything either one covers.
What segmentation means
Wikipedia puts it in one line: image segmentation is "the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics." The goal, it adds, is to turn the picture into something more meaningful and easier to analyze.
The TensorFlow tutorial draws the same line against classification. In classification the network gives one label to the whole image. If you also want the shape of the object, and which pixel belongs to which object, you need a class for each pixel. That job is segmentation, and the tutorial says it returns much more detailed information about the image.
The answer is a mask. If the picture is 8 pixels wide and 8 tall, the mask is 8 by 8 as well, and each cell holds a label. In the simplest case the labels are 0 for background and 1 for object.
Three kinds of segmentation
Wikipedia lists three groups. Picture a photo of a crowd on a street.
Semantic
Every pixel gets a class. All the pixels on people get the same label, person, and the pixels on the street get another. Two people standing close share one label.
Instance
Each separate object gets its own identity. Every person in the picture is outlined on their own, so you can count them.
Panoptic
Both at once. Every pixel has a class, and different objects of the same class are told apart.
Tip
A quick way to remember: semantic answers what, instance answers which one, and panoptic answers both.
Shrink to understand, grow back to draw
A network that labels pixels has an awkward job. To know that something is a cat, it must look at a big patch of the picture. To draw the cat's edge neatly, it must work at the finest detail. The usual answer is a network with two halves.
Wikipedia says that in 2015 convolutional networks reached state of the art in semantic segmentation, and describes the design as a classic autoencoder shape. If you read Autoencoders, you have met it. The TensorFlow tutorial names the halves in a way that is easy to remember: an encoder (downsampler) and a decoder (upsampler).
- 1
1. Shrink
The encoder passes the picture through layers of convolution and pooling. Each step makes the grid smaller and the understanding richer. Convolution and Pooling covered how.
- 2
2. Understand
At the smallest size, each cell stands for a big area of the original and carries what the network has learned about it: fur, road, bone.
- 3
3. Grow
The decoder blows the grid back up, step by step, until it is the size of the picture again.
- 4
4. Label
At full size, every pixel gets a score for each class, and the highest score wins.
In the TensorFlow tutorial, the decoder grows a 4 by 4 grid to 8 by 8, then 16 by 16, 32 by 32 and 64 by 64. Its pet photos use three classes: the pet, the border around the pet, and everything else.
U-Net and the skip connection
The best-known network of this shape is U-Net. Wikipedia says it was made by Olaf Ronneberger, Philipp Fischer and Thomas Brox in 2015, and reported in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation." It was first developed to find cell boundaries in biomedical images.
The name comes from the drawing. The shrinking half, which Wikipedia calls the contracting path, runs down the left side of the letter. The growing half, the expansive path, comes back up the right side. Wikipedia says the design was changed to work with fewer training images and to give more precise segmentation. It also notes that a 512 by 512 image takes less than a second on a modern (2015) GPU.
There is a problem. When the encoder shrinks the picture, it throws away fine detail, such as exactly where an edge sat. The decoder cannot invent it again.
U-Net's fix is the skip connection. Wikipedia describes skip connections as placed between layers of the same shape, to preserve details that would have been lost otherwise. Picture a student who read a long book and wrote a short summary. The summary says what the book is about. Before drawing, the student is also handed the original page with its small print. Summary for meaning, page for detail.
Tip
This is why a U-Net can draw crisp edges. The encoder says what it is, and the skip connection tells the decoder where the edge was.
Growing a grid back: upsampling
Upsampling means making a grid bigger. The plainest way is to copy: turn every cell into a 2 by 2 block of the same value. That is called nearest-neighbour upsampling, because each new cell takes its value from the nearest old one. Real networks often use a learned version instead, which Wikipedia calls transposed convolution. The copy version is the one to see with your own eyes, so here it is.
small = [
[0, 0, 1, 0],
[0, 1, 1, 1],
[0, 1, 1, 1],
[0, 0, 0, 0],
]
def upsample2(mask):
big = []
for row in mask:
wide = [v for v in row for _ in range(2)] # repeat each value across
big.append(wide)
big.append(list(wide)) # repeat the row down
return big
big = upsample2(small)
print(f"{len(small)}x{len(small[0])} -> {len(big)}x{len(big[0])}")
for row in big:
print(" ".join(str(v) for v in row))Output
4x4 -> 8x8 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
The 4 by 4 mask became 8 by 8. Each original 1 is now a block of four 1s, so the shape is bigger but blockier. Copying cannot add the detail that shrinking removed, which is why skip connections help.
Scoring a segmentation
How do you grade a mask? You compare it with the true mask, drawn by a person, pixel by pixel. There are three common scores.
Pixel accuracy
The share of pixels the network got right. Easy to understand, easy to fool.
IoU, per class
For one class, how much the predicted shape and the true shape overlap compared with everything either covers. Wikipedia defines the Jaccard index as the intersection size divided by the union size, also called intersection over union.
Dice
Another overlap score. It runs from 0 for no overlap to 1 for a perfect match.
Here is the first score as a formula, and then in words.
In words: count the pixels the network labelled correctly and divide by the number of pixels in the picture.
In words: P is the set of pixels the network says belong to the class, and T is the set that truly do. Count the pixels in both sets, and divide by the number of pixels in either set.
In words: twice the number of pixels in both sets, divided by the size of the first set plus the size of the second. Wikipedia gives this same formula for the Dice-Sørensen coefficient, which Lee Raymond Dice and Thorvald Sørensen developed independently and published in 1945 and 1948. It also says Dice is used in image segmentation to compare an algorithm's output against reference masks in medical applications.
Wikipedia notes one difference from the Jaccard index: Jaccard counts the shared pixels once in the top and bottom of the fraction, and Dice counts them twice.
Try it: a tiny segmenter
Time to run one. We make up an 8 by 8 grey picture. A bright square object sits in the middle of a dark background, with a little noise: two object pixels are dim, and two background pixels are bright. The segmenter is as simple as it gets, a threshold. Wikipedia calls thresholding the simplest method of image segmentation: pick a cut-off and turn a grey picture into a two-colour one. Here, any pixel at 5 or above is called object.
Then we print the predicted mask, the true mask, and the scores. The code uses only Python's standard library.
image = [
[1, 2, 1, 1, 2, 1, 2, 1],
[2, 1, 2, 1, 1, 2, 1, 1],
[1, 2, 7, 8, 8, 2, 1, 2],
[1, 1, 8, 9, 9, 8, 3, 1],
[2, 1, 8, 9, 4, 8, 1, 2],
[1, 2, 7, 8, 9, 8, 2, 1],
[2, 1, 1, 6, 1, 2, 5, 1],
[1, 2, 1, 1, 2, 1, 1, 2],
]
truth = [
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 1, 1, 1, 0, 0],
[0, 0, 1, 1, 1, 1, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0],
]
# the segmenter: label a pixel 1 (object) if it is bright, else 0 (background)
threshold = 5
pred = [[1 if v >= threshold else 0 for v in row] for row in image]
def show(name, mask):
print(name)
for row in mask:
print(" ".join(str(v) for v in row))
print()
show("predicted mask", pred)
show("true mask", truth)
pairs = [(p, t) for pr, tr in zip(pred, truth) for p, t in zip(pr, tr)]
total = len(pairs)
right = sum(p == t for p, t in pairs)
print(f"pixel accuracy: {right}/{total} = {right / total:.3f}")
def iou(label):
both = sum(p == label and t == label for p, t in pairs)
either = sum(p == label or t == label for p, t in pairs)
return both, either
for label, name in [(1, "object"), (0, "background")]:
both, either = iou(label)
print(f"IoU {name}: {both}/{either} = {both / either:.3f}")
both, either = iou(1)
p1 = sum(p == 1 for p, _ in pairs)
t1 = sum(t == 1 for _, t in pairs)
print(f"Dice object: 2*{both}/({p1}+{t1}) = {2 * both / (p1 + t1):.3f}")Output
predicted mask 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 0 0 1 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 true mask 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 pixel accuracy: 60/64 = 0.938 IoU object: 14/18 = 0.778 IoU background: 46/50 = 0.920 Dice object: 2*14/(16+16) = 0.875
Read the masks side by side. The segmenter found most of the object. It missed two true object pixels, one on the top row of the square and one near its middle, because both were dim. It also called two background pixels object, in the row under the square, because both were bright.
Now the scores. Pixel accuracy is 60 out of 64, or 0.938: only 4 pixels are wrong, so it sounds great. IoU for the object is 14 out of 18, or 0.778: the prediction and the truth share 14 pixels, and together they cover 18. Dice for the object is 2 times 14 over 16 plus 16, which is 0.875. The same four mistakes give three different numbers, and IoU is the strictest.
Quick check
In the run above, the two masks share 14 object pixels and together cover 18. What is the object's IoU?
Why accuracy alone can fool you
Most of this picture is background: 48 of 64 pixels. That means a lazy segmenter, one that calls every pixel background, still scores well on pixel accuracy. Run it and see.
# a lazy "segmenter" that calls every pixel background
truth_object_pixels = 16
total_pixels = 64
right = total_pixels - truth_object_pixels
print(f"pixel accuracy: {right}/{total_pixels} = {right / total_pixels:.3f}")
print(f"IoU object: 0/{truth_object_pixels} = {0 / truth_object_pixels:.3f}")Output
pixel accuracy: 48/64 = 0.750 IoU object: 0/16 = 0.000
It gets 0.750 on pixel accuracy and finds nothing at all: IoU for the object is 0.000. Imagine a scan where the thing you care about fills only a few pixels in a hundred. A network that never finds it would still score very high on accuracy. This is why people also report IoU per class, so every class gets its own grade, and Dice, which looks only at the object and ignores the easy background.
Tip
When one class fills most of the picture, trust IoU or Dice for the small class over pixel accuracy.
Quick check
A scan is 95% background. A network labels everything as background. Which score will look best?
Where it is used
The TensorFlow tutorial says image segmentation has many applications in medical imaging, self-driving cars and satellite imaging, to name a few. Wikipedia fills in the medical side: segmentation is used to locate tumours and other pathologies, measure tissue volumes, study anatomy and plan surgery. It also lists locating objects in satellite images, such as roads, forests and crops.
U-Net's own field is biomedical. Wikipedia says it is used for segmenting organs and body systems in CT and MRI scans, including brain and liver images, and that it has also been used on micrographs of materials in the physical sciences.
Tip
A mask is a labelled picture, so it can feed the next step: measure an area, count objects, or highlight them for a person to check.
Recap
What it is
A label for every pixel. Semantic marks classes, instance separates objects, panoptic does both.
How networks do it
Shrink to understand, grow back to draw. U-Net adds skip connections to bring fine detail across.
How we score it
Pixel accuracy is easy but can hide a miss. IoU per class and Dice judge the overlap of shapes.
Next: recurrent neural networks, models with a memory.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.