esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.27

Image Understanding

Getting facts out of pictures and video: describing, reading text, pulling fields into JSON, and checking what the model tells you.

13 min readFree

From pictures to facts

The last lesson, Multimodal AI, ended with a promise: next, image understanding, getting facts out of pictures and video. Here it is.

Think of a friend who looks at a photo of your shopping receipt and reads it out to you. "Corner Cafe, March 2nd, nine seventy-five." A vision model does the same job. You hand it a picture and a question. It hands back words.

By the end you will know what these models can do, how to send an image, a few tips that help, and how to get fields out as JSON. You will also run a small program that checks a model's answers, and a second one that shows how video is cut into frames.

Key words

Vision model
A model that accepts images as input, not just text, and answers in words.
OCR
Optical character recognition: turning the text in a picture into real text a computer can use.
Extraction
Pulling a few named facts out of something messy, like the store and the total from a receipt.
JSON
A plain-text way to write facts as labelled values, like "total": 9.75. Programs read it easily.
Validation
Checking an answer against rules you wrote, before you trust it.
Frame
One still picture taken from a video. A video is many frames shown quickly.

What a vision model can do

The OpenAI guide sums it up in one line: "Use a vision-capable model to describe images, read visible text, and answer questions about objects, shapes, colors, or textures." It adds a warning in the next sentence: account for the model's limitations when using its answers.

The Wikipedia page on optical character recognition explains the reading part. It defines OCR as "the electronic or mechanical conversion of images of typed, handwritten or printed text into machine-encoded text". A vision model can read the text and also answer questions about the picture around it.

  • Describe

    "What is in this picture?" You get a short description, a caption or alt text for a design.

  • Read and answer

    "What does the sign say?" or "Which chart bar is tallest?" The model reads text and looks at shapes and colours.

  • Extract and compare

    "Give me the store, date and total as JSON." Or send two pictures and ask what changed between them.

Extraction is the one that builds products. Fields can go straight into a spreadsheet or a database. The rest of this lesson leans on it.

How an image goes in

Chat models take a list of messages. For a vision model, one message can hold several parts: some text, and one or more images. The OpenAI guide says you can provide an image using "either a fully qualified image URL or a Base64-encoded data URL". Base64 is a way to write the bytes of a file as plain text so they fit inside a request.

  1. 1

    Pick the picture

    A clear file in a supported format. The OpenAI guide lists PNG, JPEG, WEBP and non-animated GIF. The Anthropic page lists JPEG, PNG, GIF and WebP.

  2. 2

    Add it to the message

    Put the image in the message as a link or as base64 text, next to your question.

  3. 3

    Ask a clear question

    Say what you want back: a description, a short answer, or fields in a set shape.

  4. 4

    Read the reply and check it

    The model answers in text. Treat that text as a draft until a check has passed.

Here is the request example from the OpenAI guide. It is shown exactly as the docs print it, with no output, because we do not run it here. Running it needs a paid account and a key.

Text
# From the OpenAI "Images and vision" guide, shown without its output (not run here)
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg",
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

Notice the content list. One part is text and one is an image link. You can send several images the same way. The OpenAI guide warns that "images count as tokens and will be billed accordingly". The Tokenization lesson in Machine Learning Foundations explains tokens; here it is enough to know a picture has a price, like words do.

The Anthropic page gives the size rule in one line. It says an image costs the width divided by 28, rounded up, times the height divided by 28, rounded up, in visual tokens. In words: cut the picture into 28-pixel squares and count them.

tokens=⌈w28⌉×⌈h28⌉\text{tokens} = \left\lceil \frac{w}{28} \right\rceil \times \left\lceil \frac{h}{28} \right\rceil

Check it with that page's own table. A 1000 by 1000 pixel image is 36 squares across and 36 down, so 1,296 tokens. The table says 1,296. That page also says images above a size limit are shrunk first. The limit it lists is 1,568 pixels on the long edge for most models, and 2,576 pixels for Claude 4.7 and later models.

Tips that help

Small habits change results a lot. These come from the docs we read, and one is marked as our own.

  • Image first, then the question

    The Anthropic page says "Claude works best when images come before text." Images placed after text still perform well, it adds, but prefer image-then-text when you can.

  • Label several images

    With more than one picture, the same page suggests a short text label before each: "Image 1:", "Image 2:". Then your question can say "how is Image 2 different from Image 1?"

  • Keep text legible

    Both guides ask for clear, readable text. Anthropic says heavy JPEG compression can make text difficult to read. OpenAI says to enlarge small text.

Size matters too. Anthropic says images under 200 pixels can lead to mistakes, and a huge image is shrunk, which can make text less legible. So crop to the part that matters instead of sending a whole 4K screenshot.

OpenAI also has a dial named "detail". Its values are low, high, original or auto, and auto is the default. For reading fine print the guide says to use "original" when the model supports it.

Tip

Our own habit, not a rule from the docs: ask for JSON with named fields, and say what to do when something cannot be read, like "use null". A fixed shape is easy to check by code. A free-form paragraph is not.

Pulling fields into JSON

Now the worked example. You are building an expense app. Users photograph receipts. A vision model reads each photo and returns JSON. Before that JSON goes into anyone's records, your code checks it.

We have no model here, and we do not need one. The code below uses a stand-in, fake_model, that ignores the picture and returns three fixed answers. It is a stand-in so the code runs free. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.

Receipt A is correct. Receipt B has the date written the wrong way round. Receipt C has line items that do not add up to the total. The checker looks for four things: valid JSON, all required fields, a date in year-month-day form, and line items that add to the total within 0.01.

Python
import json
from datetime import datetime
from decimal import Decimal

# Stand-in for a vision model: it ignores the picture and returns fixed text.
def fake_model(messages):
    name = messages[0]["image"]
    answers = {
        "receipt_a.jpg": '{"store": "Corner Cafe", "date": "2024-03-02", "items": [{"name": "Latte", "price": 4.50}, {"name": "Muffin", "price": 3.25}, {"name": "Tea", "price": 2.00}], "total": 9.75}',
        "receipt_b.jpg": '{"store": "Book Nook", "date": "03/04/2024", "items": [{"name": "Notebook", "price": 6.40}, {"name": "Pen", "price": 1.60}], "total": 8.00}',
        "receipt_c.jpg": '{"store": "Fresh Mart", "date": "2024-03-05", "items": [{"name": "Rice", "price": 12.00}, {"name": "Beans", "price": 7.50}], "total": 21.50}',
    }
    return answers[name]

REQUIRED = ["store", "date", "items", "total"]

def check(text):
    reasons = []
    try:
        data = json.loads(text, parse_float=Decimal)
    except json.JSONDecodeError:
        return ["not valid JSON"]
    for field in REQUIRED:
        if field not in data:
            reasons.append("missing " + field)
    if reasons:
        return reasons
    try:
        datetime.strptime(data["date"], "%Y-%m-%d")
    except ValueError:
        reasons.append("date " + data["date"] + " is not YYYY-MM-DD")
    added = sum(item["price"] for item in data["items"])
    if abs(added - data["total"]) > Decimal("0.01"):
        reasons.append(f"items add to {added} but total says {data['total']}")
    return reasons

for name in ["receipt_a.jpg", "receipt_b.jpg", "receipt_c.jpg"]:
    messages = [{"image": name}, {"text": "Return JSON with store, date (YYYY-MM-DD), items and total."}]
    problems = check(fake_model(messages))
    print(name, "PASS" if not problems else "FAIL: " + "; ".join(problems))

Output

receipt_a.jpg PASS
receipt_b.jpg FAIL: date 03/04/2024 is not YYYY-MM-DD
receipt_c.jpg FAIL: items add to 19.50 but total says 21.50

Read the output. Receipt A passes. Receipt B fails with a reason: "03/04/2024" is not year-month-day, and it could even mean March 4th or April 3rd, so the code refuses to guess. Receipt C fails because its items add to 19.50 while the total says 21.50.

Two details are worth a second look. The code reads prices as Decimal, a number type that does exact decimal sums. Plain floats can give 0.1 + 0.2 = 0.30000000000000004, which would make a sum check unfair. And the check returns reasons, not just yes or no, so a person can see what went wrong.

The missing-field and bad-JSON paths did not fire in this run. They are there for the day a real model leaves out a field or wraps its JSON in chatty words.

Checking what you get back

Vision models make mistakes. Both docs say so plainly, and their lists of limits are useful as a test plan.

  • Small, rotated or odd images

    OpenAI says the model may misinterpret rotated or upside-down text, and struggles with panoramic and fisheye images. Anthropic says it might make mistakes on low-quality, rotated or very small images.

  • Counting and positions

    Both say counts of objects can be approximate. OpenAI says precise spatial work, like identifying chess positions, is a struggle.

  • Medical and high stakes

    Both say not to use it to interpret specialized scans like CT scans or as medical advice. Anthropic adds: "Always carefully review and verify Claude's image interpretations, especially for high-stakes use cases."

So build checks into the flow. The receipt checker is one: rules a human would use, written as code. Others work the same way.

  1. 1

    Validate the shape

    Parse the JSON. Reject missing fields and wrong types.

  2. 2

    Cross-check numbers

    Add up the line items. Compare with the total. Compare the date with a sensible range.

  3. 3

    Send doubtful ones to a person

    A failed check should go to a human review queue, not be silently fixed or thrown away.

Quick check

A receipt's line items add to 19.50, but the model says the total is 21.50. What should your app do?

Privacy: faces, IDs and documents

A picture often holds more than you meant to send. A receipt can show part of a card number. A photo of a desk can show a face, a letter or a passport on the side. Once it goes in a request, it is data you are responsible for.

Some facts from the docs. Anthropic says Claude "cannot be used to name people in images and refuses to do so". Anthropic also says it cannot delete images you have uploaded in a normal request. If you use its Files API, you can delete them there. It states that it does not use uploaded images to train models, and points to its privacy policy for details. OpenAI says image use "must comply with our usage policies".

  • Send less

    Crop to the part you need. A crop of the total line does not carry a face or an address.

  • Mask what you do not need

    Cover card numbers, ID numbers and faces in your own code before the image leaves your server.

  • Tell people and ask first

    If users upload documents, say in plain words where the picture goes and who can see it. Get a clear yes. Read your provider's data rules before you launch.

The three tips above are our own advice, not quotes from the docs. If you would not want a stranger to see the picture, think hard before you send it.

Video: many pictures in a row

A video is a stack of still pictures, called frames, plus sound. Some models take a video file. Others take only images, and then you cut frames yourself and send them as several images. Both vision guides we read allow more than one image in a request.

The Gemini API page says its models can "describe, segment, and extract information from videos, answer questions about video content, and refer to specific timestamps within a video". It says you can ask about a moment using a time written as MM:SS, minutes then seconds, like 00:05.

The part to learn is sampling. The page says that by default video inputs are processed by "extracting frames at 1 FPS". FPS means frames per second. For descriptions it says the default "may miss details in videos with rapid motion or quick scene changes". You can set your own rate: the page's example sets fps to 0.5 with the comment "Sample 1 frame every 2 seconds".

More frames means more tokens and more cost. Fewer frames is cheaper but can miss a short event. The program below uses a made-up 9-second video. It lists the frame times for one frame every 2 seconds, then checks a made-up event, a door opening from 3.0 to 3.6 seconds.

Python
duration = 9
gap = 2

times = list(range(0, duration, gap))
print("every", gap, "seconds:", len(times), "frames")
for t in times:
    print(f"  frame at {t} s  ->  {t // 60:02d}:{t % 60:02d}")

event = (3.0, 3.6)
seen_2 = any(event[0] <= t <= event[1] for t in times)
seen_1 = any(event[0] <= t <= event[1] for t in range(duration))
print("door opens from 3.0 s to 3.6 s")
print("seen with a frame every 2 s?", seen_2)
print("seen with a frame every 1 s?", seen_1)

Output

every 2 seconds: 5 frames
  frame at 0 s  ->  00:00
  frame at 2 s  ->  00:02
  frame at 4 s  ->  00:04
  frame at 6 s  ->  00:06
  frame at 8 s  ->  00:08
door opens from 3.0 s to 3.6 s
seen with a frame every 2 s? False
seen with a frame every 1 s? True

Read it. Every 2 seconds gives 5 frames, at 0, 2, 4, 6 and 8 seconds. The door opens between the 2 s and the 4 s frames, so no frame catches it. One frame per second gives a frame at 3 s, which lands inside the event, so it is seen.

Quick check

You sample a video at one frame every 2 seconds. Which event is most likely to be missed?

What to remember

  • Ask in a shape you can check

    Image first, a clear question, JSON with named fields, and a plan for unreadable parts.

  • Never trust, always check

    Validate fields, cross-check totals, and send doubtful results to a person.

  • Send only what you need

    Crop, mask faces and ID numbers, and be clear with users about where pictures go.

Next: image generation, making pictures from words.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.