Machine Learning Foundations / Advanced / 03.34
Image & Video Recognition
Finding things in a picture and following them through a video: boxes, scores, overlap, and how a computer keeps track.
14 min readFree
From one label to many boxes
The last lesson, Image Classification, ended with a promise: finding things and following them. Classification gives a whole picture one label, such as "dog". Real photos are busier. A street has cars, bikes and people, all at once, and we often want to know where each one is. This lesson is about that, and then about doing it across a video.
Think of a crowd photo with a teacher walking around the class. She does not just say "this is a school photo". She points at each child, says the name, and draws a ring around them. Then, in a video, she follows each child from one moment to the next. A computer does the same in two jobs: detect, then track.
By the end you will know what a detector hands back, how to measure whether two boxes agree, how a computer removes repeated boxes, and how it follows one thing across frames. You will run all three in short Python programs.
Key words
- Object detection
- Finding things in a picture and saying both what each one is and where it is.
- Bounding box
- A rectangle drawn around a thing. Four numbers: left, top, right, bottom.
- Confidence (score)
- A number from 0 to 1 for how sure the detector is about one box. 0.9 means very sure.
- IoU
- Short for intersection over union. A number from 0 to 1 for how much two boxes overlap.
- Non-maximum suppression
- A clean-up step that keeps the best box and drops the repeats around it. Often written NMS.
- Frame
- One still picture out of a video. A video is many frames shown one after another.
- Tracking
- Following the same thing from frame to frame, so it keeps the same name.
Detection: what and where
Wikipedia describes object detection as finding instances of things of a certain class, such as humans, buildings or cars, in images and videos. For every thing it finds, a detector hands back three items: a box, a class and a confidence.
A box
Where the thing is, as four numbers: left, top, right, bottom.
A class
What it is: car, person, dog. The detector picks from a list it learned.
A confidence
How sure it is, from 0 to 1. A good system lets you ignore the unsure boxes.
The original R-CNN idea, in the words of Wikipedia, was to take an input image and produce a set of bounding boxes, each holding an object and its category. So this is classification plus a pointing finger.
Tip
A picture can have zero things or twenty. That is why a detector returns a list of boxes, not one answer.
Do two boxes agree? Intersection over union
We need a way to say how well two boxes match. It matters twice: when we score a detector against a human's box, and when a detector has drawn the same dog three times. The tool is IoU. Wikipedia calls it the intersection size divided by the union size, and notes it is the same as the Jaccard index.
Picture two sheets of paper lying on a table, partly on top of each other. The intersection is the part where they overlap. The union is all the table covered by either sheet. IoU is the overlap divided by the covered table.
In words: take the area where the two boxes overlap. Divide it by the total area they cover together. The second form says the same thing, because the total is the area of A plus the area of B, minus the overlap, which would otherwise be counted twice.
Wikipedia gives the range: IoU is never below 0 and never above 1. If the boxes share nothing, it is 0. If they are the same box, it is 1. In object detection, a box is often counted as a hit when its IoU with the human-drawn box is above a threshold such as 0.5.
Let us compute it. A box is four numbers. The width of the overlap is the smaller right edge minus the larger left edge, and the same for height. If either is negative, the boxes do not touch, so we use 0.
def iou(a, b):
# a, b are boxes: (left, top, right, bottom)
w = min(a[2], b[2]) - max(a[0], b[0])
h = min(a[3], b[3]) - max(a[1], b[1])
inter = max(0, w) * max(0, h)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / (area_a + area_b - inter)
pairs = [
("same box", (0, 0, 10, 10), (0, 0, 10, 10)),
("shifted a bit", (0, 0, 10, 10), (2, 0, 12, 10)),
("half hidden", (0, 0, 10, 10), (5, 0, 15, 10)),
("corner touch", (0, 0, 10, 10), (8, 8, 18, 18)),
("far apart", (0, 0, 10, 10), (20, 20, 30, 30)),
]
for name, a, b in pairs:
print(f"{name:14} IoU = {iou(a, b):.2f}")Output
same box IoU = 1.00 shifted a bit IoU = 0.67 half hidden IoU = 0.33 corner touch IoU = 0.02 far apart IoU = 0.00
Read the lines like a ruler. Two boxes of the same size, one pushed 2 steps sideways, still score 0.67. Pushed halfway, they fall to 0.33. Touching only at a corner gives 0.02, and boxes far apart give exactly 0.
Cleaning up repeats: non-maximum suppression
Detectors are eager. Around one dog they often draw several boxes, each a little different, each with its own score. Showing all of them would look like a dog with a halo of rectangles. Wikipedia's R-CNN page says the remaining candidates typically suffer from heavy duplication, and that a step called non-maximum suppression (NMS) resolves it.
The idea is a talent show vote. The best act on stage wins, and anyone who did almost the same act is sent home. Then the next-best act that is left gets a turn.
- 1
1. Sort
Line up all boxes from the highest score to the lowest.
- 2
2. Keep the best
Take the first box and keep it. It is the winner for its area.
- 3
3. Drop its copies
Compare every other box with the winner using IoU. Drop any box whose IoU is above a limit, here 0.5.
- 4
4. Repeat
Take the best box that is left and go back to step 2. Stop when nothing is left.
Here are six made-up boxes: a dog found three times, a cat found twice, and a sofa found once. Each has a score. We run the four steps and print every choice.
def iou(a, b):
w = min(a[2], b[2]) - max(a[0], b[0])
h = min(a[3], b[3]) - max(a[1], b[1])
inter = max(0, w) * max(0, h)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / (area_a + area_b - inter)
# (name, box, score): two dogs and one cat, found several times each
boxes = [
("dog A", (10, 10, 60, 60), 0.90),
("dog B", (14, 12, 64, 62), 0.80),
("dog C", (8, 14, 58, 66), 0.70),
("cat A", (100, 20, 140, 70), 0.85),
("cat B", (104, 22, 146, 74), 0.60),
("sofa", (30, 90, 130, 140), 0.55),
]
def nms(items, limit=0.5):
left = sorted(items, key=lambda it: it[2], reverse=True)
kept = []
while left:
best = left.pop(0)
kept.append(best)
print("keep", best[0], best[2])
rest = []
for it in left:
v = iou(best[1], it[1])
if v > limit:
print(f" drop {it[0]} (score {it[2]}), IoU with {best[0]} = {v:.2f}")
else:
rest.append(it)
left = rest
return kept
survivors = nms(boxes)
print("survivors:", [s[0] for s in survivors])Output
keep dog A 0.9 drop dog B (score 0.8), IoU with dog A = 0.79 drop dog C (score 0.7), IoU with dog A = 0.76 keep cat A 0.85 drop cat B (score 0.6), IoU with cat A = 0.70 keep sofa 0.55 survivors: ['dog A', 'cat A', 'sofa']
Three boxes survive, one per thing. Dog B and dog C were dropped because they overlapped the best dog box by 0.79 and 0.76, both above 0.5. Cat B was dropped at 0.70. The sofa overlapped nothing, so it lived, even with the lowest score.
Tip
The limit matters. Set it too low and two real dogs standing close together may be merged into one. Set it too high and the repeats stay. Teams tune it on their own pictures.
Three families of detectors
You will meet three names again and again. You do not need to build them. You only need to know what each one is known for.
R-CNN family
Wikipedia says the original R-CNN used selective search to propose as many as 2,000 regions and ran a network on each. Fast R-CNN ran the network once on the whole image, and Faster R-CNN moved the region proposing inside the network.
YOLO
"You Only Look Once", introduced in 2015. Wikipedia says it needs only one forward pass through the network, and that it divides the image into regions and predicts boxes and probabilities for each. It is a series of real-time detectors.
SSD
Single Shot MultiBox Detector. Its paper says it uses a single deep network, with default boxes of several shapes and sizes, and drops the separate step of proposing regions. The authors report accuracy comparable to methods that use that step, and much faster.
A "forward pass" is just one trip of the picture through the network. R-CNN needed one trip per region. YOLO and SSD need one trip per picture, which is why they are called single-shot. That is the whole difference in speed.
Video is many pictures in a row
A video is a flip book. Each page is a frame, and flipping fast gives the feeling of movement. Say a video runs at 30 frames in every second. Then the computer has only a thirtieth of a second, about 33 thousandths, to handle each frame if it wants to keep up live.
Detection works frame by frame: run the detector on each one. But that alone gives you boxes with no memory. In frame 5 there is a dog, in frame 6 there is a dog, and nothing says it is the same dog. That link is tracking.
Tracking one thing across frames
Wikipedia defines video tracking as locating a moving object, or several, over time using a camera, and says its objective is to associate target objects in consecutive video frames. It adds that this gets hard when objects move fast relative to the frame rate.
A simple way to link frames uses what you just learned. A car does not jump across the screen in one thirtieth of a second, so its new box should overlap its old box a lot. For each known thing, look at the next frame, pick the box with the highest IoU, and accept it if the overlap is high enough.
def iou(a, b):
w = min(a[2], b[2]) - max(a[0], b[0])
h = min(a[3], b[3]) - max(a[1], b[1])
inter = max(0, w) * max(0, h)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / (area_a + area_b - inter)
# What a detector found in each of 5 frames: boxes only, no names.
frames = [
[(10, 10, 30, 30), (80, 50, 100, 70)],
[(14, 10, 34, 30), (78, 52, 98, 72)],
[(18, 11, 38, 31), (76, 54, 96, 74), (150, 5, 170, 25)],
[(23, 12, 43, 32), (74, 56, 94, 76)],
[(28, 12, 48, 32)],
]
tracks = {1: frames[0][0], 2: frames[0][1]} # id -> last box
print("frame 0: id 1 and id 2 start")
for n, found in enumerate(frames[1:], start=1):
free = list(found)
for tid, last in tracks.items():
if not free:
print(f"frame {n}: id {tid} not found")
continue
best = max(free, key=lambda b: iou(last, b))
v = iou(last, best)
if v >= 0.3:
tracks[tid] = best
free.remove(best)
print(f"frame {n}: id {tid} matched, IoU = {v:.2f}")
else:
print(f"frame {n}: id {tid} not found")
for b in free:
print(f"frame {n}: new box {b}, no track close enough")Output
frame 0: id 1 and id 2 start frame 1: id 1 matched, IoU = 0.67 frame 1: id 2 matched, IoU = 0.68 frame 2: id 1 matched, IoU = 0.61 frame 2: id 2 matched, IoU = 0.68 frame 2: new box (150, 5, 170, 25), no track close enough frame 3: id 1 matched, IoU = 0.55 frame 3: id 2 matched, IoU = 0.68 frame 4: id 1 matched, IoU = 0.60 frame 4: id 2 not found
Id 1 moves right, about 4 or 5 steps a frame, and its IoU stays between 0.55 and 0.67, so it is followed through all five frames. Id 2 drifts down and left and is followed until frame 3, then vanishes in frame 4. A real system would wait a few frames before giving up, because things hide behind other things. The box that appears in frame 2 matches nobody, so it would get a new id.
Tip
Real trackers add a motion model, a guess of where the thing will be next. Wikipedia says tracking systems usually use one when objects move fast or turn. Our program only looks at overlap, which is why it can lose fast things.
From tracking to actions
Knowing where a person is does not say what they are doing. Activity recognition aims to recognise the actions and goals of one or more agents from a series of observations. With video, Wikipedia says the work is often split into four steps: detecting people, tracking them, recognising the activity, and then a higher-level judgement of what is going on. Because one picture of a raised arm could be a wave or a throw, the computer looks at a series of frames, not just one.
Where it is used, and the privacy line
Wikipedia lists vehicle counting, activity recognition and face detection among the uses of object detection, along with tracking a ball in a football match or a bat in cricket. Video tracking is used for traffic control and security and surveillance, among other things. For a designer this means a camera can count cars at a crossing, or follow a ball so a broadcast can show its path.
Here is the line to hold. A box around a car is one thing. A box around a person, followed all day, is another. Wikipedia mentions research on monitoring behaviour with users' consent, and says gait recognition can recognise a person by how they walk even in disguise. So if you ever design with this, ask first who is being watched, whether they know, and whether you need to keep names or faces at all. Counting people without recording who they are is a better default than following them.
Quick checks
Quick check
Two boxes have an IoU of 0. What does that mean?
Quick check
Why do detectors need non-maximum suppression?
Quick check
In the tracking program, what makes a box in the next frame count as the same object?
What to remember
A detector answers what and where with a list of boxes, each with a class and a confidence. IoU measures how much two boxes agree. NMS keeps the best box and drops its repeats. R-CNN, YOLO and SSD are three families, and the main difference is how many trips the picture makes through the network. A video is a row of frames, tracking links the same thing across them, and activity recognition reads the movement. Keep the privacy line close: count when you can, follow only when you must.
Next: image segmentation, labelling every pixel.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.