esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.26

AI that sees, hears and reads

Models that take pictures, sound and documents as well as text: how you send them, what they cost, where they slip, and what to build with them.

13 min readFree

Beyond words

The last lesson, Eval tools, ended with a promise: next, multimodal AI, models that work with images, audio and video. Here it is.

Until now, every lesson in this course sent words in and got words back. But users do not live in words only. They photograph receipts, record meetings and send screenshots and PDFs. A text-only model cannot help with that. A multimodal model can.

This lesson is the map: what multimodal means, how a picture travels inside a request, what it costs, where models slip, and what to build. The next lessons go deeper: images, then image generation, then speech. You will also build a message with a picture in it, using free Python.

Key words

Modality
A kind of data. Text is one modality. Images, audio and video are others.
Multimodal
Working with more than one modality, for example reading a photo and a question about it together.
Content part
One piece inside a message: a piece of text, or one image. A message holds a list of them.
Base64
A way to write any file, even a picture, using only ordinary letters and digits, so it fits inside text.
Token
The small unit a model counts in. Words are cut into tokens. The Tokenization lesson in Machine Learning Foundations goes deeper.

What multimodal means

Picture a friend who can only read letters. Give them eyes and ears, and they can also look at your photo and hear your voice note. That is the jump to multimodal.

Wikipedia describes multimodal learning as a type of deep learning that integrates and processes multiple types of data, which it calls modalities, such as text, audio, images or video.

A model can be multimodal in two directions. It can take more than text in, or it can put more than text out. The same page says GPT-4o can process and generate text, audio and images.

  • Many things in

    You send a photo, a recording or a document along with your question. The answer comes back as text.

  • Many things out

    The model makes a picture or speaks aloud. The Claude docs are clear that Claude is an image understanding model only and cannot generate or edit images. OpenAI's guide, by contrast, says GPT Image models can create new images or edit existing ones. Check what a model does, not just what it reads.

  • Mixed together

    Pictures and words interleaved in one request, such as a question, three photos, and a follow-up. Wikipedia describes one way to build it: pictures become image tokens that sit among the text tokens.

How a picture travels with your words

A message to a model is a list of parts. In a multimodal message you add an image part next to the text part, and the model reads them together.

A request is text, so a picture has to be written as text. The Claude docs list three ways to hand over an image: a base64 string inside the request, a URL to an image hosted online, or a file id from their Files API, where you upload once and refer to it many times. The OpenAI and Gemini docs list the same three ideas: a URL, base64 data, or an uploaded file.

  1. 1

    1. Get the picture as bytes

    Read a file, or make a tiny picture in code.

  2. 2

    2. Turn the bytes into base64 text

    Python's base64 module does this in one line.

  3. 3

    3. Add an image part to the content list

    The part says what kind it is, what format the picture has and carries the text string.

  4. 4

    4. Add a text part with your question

    Words tell the model what you want done with the picture.

  5. 5

    5. Send the list, and read the reply

    We stop before this step. Nothing is sent in this lesson.

Here is what the Claude docs show for the content list. It is copied as it appears there and not run, since running it would need a key and cost money.

Text
# From the Claude Platform Docs (Vision), shown without its output (not run here)
"content": [
    {
        "type": "image",
        "source": {
            "type": "base64",
            "media_type": image1_media_type,
            "data": image1_data,
        },
    },
    {"type": "text", "text": "Describe this image."},
],

Now build the same shape yourself. The code makes a tiny 2 by 2 picture with four pixels: red, green, blue and white. The zlib and struct modules from Python's standard library write the file. Then the code encodes it and puts it in a message with the field names from the docs above.

Python
import base64
import json
import struct
import zlib


def chunk(kind, data):
    body = kind + data
    return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))


def make_png(width, height, pixels):
    # Each row starts with a filter byte (0 = none), then red, green, blue per pixel.
    rows = b"".join(b"\x00" + bytes(row) for row in pixels)
    header = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
    return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header)
            + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b""))


# A made-up 2 by 2 picture: red, green / blue, white.
png = make_png(2, 2, [[255, 0, 0, 0, 255, 0], [0, 0, 255, 255, 255, 255]])
print("PNG bytes:", len(png))
print("Starts with:", png[:8])

b64 = base64.b64encode(png).decode("ascii")
print("Base64 characters:", len(b64))

message = {
    "role": "user",
    "content": [
        {
            "type": "image",
            "source": {"type": "base64", "media_type": "image/png", "data": b64},
        },
        {"type": "text", "text": "Describe this image."},
    ],
}

# Shorten the long string, only so it fits on screen.
shown = json.loads(json.dumps(message))
shown["content"][0]["source"]["data"] = b64[:20] + "..."
print(json.dumps(shown, indent=2))
print("Sent anywhere? No. This only built a dictionary.")

Output

PNG bytes: 75
Starts with: b'\x89PNG\r\n\x1a\n'
Base64 characters: 100
{
  "role": "user",
  "content": [
    {
      "type": "image",
      "source": {
        "type": "base64",
        "media_type": "image/png",
        "data": "iVBORw0KGgoAAAANSUhE..."
      }
    },
    {
      "type": "text",
      "text": "Describe this image."
    }
  ]
}
Sent anywhere? No. This only built a dictionary.

The picture is 75 bytes. As base64 it grows to 100 characters, because base64 spends four characters on every three bytes. The message is just a dictionary with an image part and a text part. We shortened the long string with three dots, only to fit your screen. Nothing was sent anywhere: no key, no network, no model.

A real photo is far bigger than 75 bytes, and base64 makes it about a third bigger. The Claude docs also warn that in a long chat each request resends the history, so base64 images ride along on every turn. Their advice: upload once with the Files API and use the file id.

Tip

Order matters, and providers disagree. The Claude docs say Claude works best when images come before text. The Gemini docs say that with one image and text, you should place the text prompt before the image. Read the page for the model you use, and test both orders.

Quick check

Why does a picture have to be written as base64 (or sent as a URL or file id)?

Pictures cost tokens

A model charges by the token, and a picture is turned into tokens too. The OpenAI guide says it plainly: images count as tokens and will be billed accordingly.

How many depends on the provider, the model and the picture size. Three examples from the docs:

  • Claude

    Works in patches: each is a 28 by 28 pixel block, called a visual token. Large images are scaled down first, which caps the cost.

  • OpenAI

    Some models cover the image with 32 by 32 pixel patches. A detail setting, with values low, high, original or auto, controls how much the picture is shrunk first. It defaults to auto.

  • Gemini

    258 tokens if both sides are 384 pixels or less. Larger pictures are cut into 768 by 768 tiles, each costing 258 tokens.

Bigger picture, more tokens. For Claude, the docs give the rule as a formula; the brackets mean round up.

tokens=⌈width/28⌉×⌈height/28⌉\text{tokens} = \lceil \text{width} / 28 \rceil \times \lceil \text{height} / 28 \rceil

In words: divide the width by 28 and round up, do the same for the height, then multiply the two. The code tries two sizes, then totals ten receipt photos. Prices change, so it uses a made-up price that belongs to no real model.

Python
import math

PRICE_PER_MILLION = 2.00  # a made-up price in made-up dollars, not any real model's


def patch_tokens(width, height):
    # The rule from the Claude vision docs: one token per 28 by 28 block.
    return math.ceil(width / 28) * math.ceil(height / 28)


for w, h in [(200, 200), (1000, 1000)]:
    t = patch_tokens(w, h)
    print(f"{w}x{h} picture: {t} tokens")

# Ten receipt photos, each 1000x1000, plus a 50-token question each time.
photos = 10
per_request = patch_tokens(1000, 1000) + 50
total = photos * per_request
print("Tokens for all ten:", total)
print("Made-up cost:", round(total / 1_000_000 * PRICE_PER_MILLION, 4), "made-up dollars")

Output

200x200 picture: 64 tokens
1000x1000 picture: 1296 tokens
Tokens for all ten: 13460
Made-up cost: 0.0269 made-up dollars

The 200 by 200 picture gives 8 times 8, which is 64 tokens. The 1000 by 1000 picture gives 36 times 36, which is 1,296. Both match the Claude docs. With a 50-token question each, ten requests use 10 times 1,346, which is 13,460 tokens, or a few cents of made-up money. The shape of the sum matters, not the number.

The Gemini docs also name a setting, media_resolution, for the most tokens given to each input image or video frame. Higher resolution helps the model read fine text, but adds tokens and delay: detail against cost.

Where models still slip

Seeing is not understanding. The Claude and OpenAI docs both list limits. These matter most when you build.

  • Small or blurry text

    OpenAI says to enlarge text inside the image to help it read. Claude says it might make mistakes with low-quality, rotated or very small images, under 200 pixels.

  • Counting

    Both docs say counts are approximate, and worse with many small objects. Do not trust a model to count 80 screws.

  • Position and space

    OpenAI says the model struggles with precise spatial tasks, and gives chess positions as an example. Claude says its coordinates and locations are approximate and should be checked.

OpenAI also lists graphs with varied line styles, panoramic and fisheye images, and non-Latin text such as Japanese or Korean. Neither doc suits complex medical scans, and Claude says it will not name people in photos or tell whether a picture was made by AI. Neither reads the file name or hidden data in the file.

Tip

The Claude docs advise carefully reviewing image readings, especially for high-stakes uses, and keeping human oversight for sensitive tasks.

Size limits differ too. Per request, Claude lists up to 100 images for models with a 200k-token context window and up to 600 for others, OpenAI up to 1,500 images and 512 MB, and Gemini up to 3,600 images. Formats differ: Claude lists JPEG, PNG, GIF and WebP, and Gemini adds HEIC and HEIF.

Tip

Capabilities in this lesson were checked on 16 November 2026. Limits and field names change often, so open the vendor's current page before you build.

Quick check

You want to count the screws in a photo of a full toolbox. What is the safest plan?

What to build with it

These are product ideas, not claims from the docs. Each takes something users already have, like a photo or a recording, and saves them typing.

  • Receipts and forms

    Photograph a receipt and get the shop, lines and total as data. The Claude docs point to extracting content from forms as a typical task.

  • Describing images for accessibility

    Write alt text, the short description a screen reader speaks, for pictures that have none.

  • Checking product photos

    Ask whether a seller's photo is blurry or missing a required angle, before it goes live.

  • Meeting transcripts

    Turn a recording into text, then into notes and tasks. This needs audio; the Gemini docs say that API supports prompting with text, image, audio and video data.

  • Video summaries

    Send a video, get a short summary. The media_resolution note above mentions video frames.

  • Screens and documents

    Read a screenshot or a scanned page, so people stop retyping.

The limits above show up here: a receipt has small text. So the model's reading should never be the last word. Check it with plain code.

The program below does that for receipts. It builds a message with a text part and an image part, then hands it to fake_model. This is a stand-in so the code runs free: it returns a fixed answer and never looks at the picture. A real model is slower, costs money for every token, can give a different answer on each run, and can fail or refuse. Then the code does what real code should: it adds up the items and compares them with the total the model reported.

Python
import json


def fake_model(messages):
    # A stand-in with a fixed answer, so this runs free. It never looks at the picture.
    # A real model is slower, costs money for every token, can answer differently
    # each run, and can fail or refuse.
    return json.dumps({
        "shop": "Corner Cafe",
        "items": [["Tea", 3.50], ["Cake", 4.25], ["Water", 1.25]],
        "total": 9.00,
    })


messages = [{"role": "user", "content": [
    {"type": "text", "text": "Read this receipt. Reply with JSON: shop, items, total."},
    {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "..."}},
]}]

reply = fake_model(messages)
data = json.loads(reply)
print("Shop:", data["shop"])

added = round(sum(price for _, price in data["items"]), 2)
print("Items add up to:", added)
print("Receipt total says:", data["total"])
print("Check passed:", added == data["total"])

wrong = dict(data, total=9.50)
print("A misread total passes?", round(sum(p for _, p in wrong["items"]), 2) == wrong["total"])

Output

Shop: Corner Cafe
Items add up to: 9.0
Receipt total says: 9.0
Check passed: True
A misread total passes? False

The items add up to 9.0, the same as the total, so the check passes. The last line shows the use: had a model misread the total as 9.5, the check would fail, and a person would look at that receipt.

  1. 1

    1. Ask for a fixed shape

    Request JSON with named fields, not a free paragraph. The Structured output lesson covers this.

  2. 2

    2. Check what you can check

    Totals that add up, dates that exist, required fields that are filled.

  3. 3

    3. Send doubtful cases to a person

    A failed check is a signal, not a crash.

Quick check

In the receipt example, what does the check do?

What to remember

Multimodal models take or make more than text. Most products start with pictures in, text out. A picture travels as an image part next to a text part, written as base64, a URL or a file id. Field names and order differ between providers, so read the docs of your model.

Pictures cost tokens, more for bigger pictures. Models slip on small text, counting and positions. Check their reading with code, and keep a person in the loop for anything important.

Next: image understanding, getting facts out of pictures and video.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.