AI engineering / Advanced / 03.31
Frameworks for multimodal apps
How LangChain and LlamaIndex carry pictures and sound next to text, how a mixed-document search works, and when plain code is enough.
14 min readFree
Pictures and sound, with help
The last lesson, Text-to-Speech, ended with a promise: next, frameworks for multimodal apps. Here it is.
So far you have met models that read text, models that look at images and models that talk. A real product often needs several of them at once. Think of a repair app: a user types a question, your app finds the right manual page, shows the right photo and asks a model to explain it. That is a lot of plumbing.
Frameworks are code libraries that do some of the plumbing for you. This lesson looks at two of them, LangChain and LlamaIndex, and what each says about pictures, sound and mixed documents. Then you build the same idea yourself in plain Python, so you can judge when a framework is worth it.
Key words
- Multimodal
- Working with more than one kind of data, such as text, pictures, sound and video. Each kind is called a modality.
- Message
- One turn in a chat with a model: who is speaking (the role) and what they said (the content).
- Content block
- One piece inside a message, such as a block of text or one image. A message can hold a list of blocks.
- Caption
- A short sentence that describes a picture. Here it is also the text we search instead of the picture.
- Index
- A list built ahead of time so searching is fast, like the index at the back of a book.
- Multimodal RAG
- Retrieval-augmented generation over mixed documents: find the best text and pictures, then hand them to a model together.
- Vision-capable model
- A model that can look at an image you send it, not only read text.
Tip
Facts in this lesson were checked on 16 November 2026. These libraries change often, so read each snippet as a picture of the idea and check the current docs before you copy it.
What multimodal means
Wikipedia describes multimodal learning as a type of deep learning that integrates and processes multiple types of data, called modalities, such as text, audio, images or video.
The same page gives a simple reason this matters. People often caption an image to say something the image does not show by itself. And sometimes an image says what words cannot. So an app that handles only one kind of data misses half the picture. The Embeddings for builders lesson showed how meaning becomes numbers. Multimodal work asks that for pictures and sound too.
For you as a builder, there are three separate jobs hiding inside the word.
Send mixed input
Put text, an image or audio into one message, so a model can read them together.
Search mixed documents
Find the right paragraph and the right picture in a pile of files, then use them as context.
Return mixed output
Get text, speech or an image back. This lesson stays on the first two; the earlier lessons cover making images and speech.
LangChain's docs mostly help with the first job. LlamaIndex's docs spend more time on the second. We take them in that order.
LangChain: mixed messages
LangChain is a framework for building apps around chat models. Its Messages page defines multimodality as the ability to work with data that comes in different forms, such as text, audio, images and video. It says LangChain includes standard types for these data that can be used across providers. A provider is the company or service that runs the model.
That last part is the selling point. Each provider asks for pictures in its own shape. A standard shape means you write the message once and switch providers without rewriting it.
The page also says what a message can hold. A message's content can be a string, a list of content blocks in a provider-native format, or a list of LangChain's standard content blocks. Think of three ways to pack a lunchbox: a single wrapped sandwich, the school's own compartment box, or a standard box that fits every bag.
Here is the docs' own example for sending a picture. A message has a role and a list of blocks: one block of text, one block of image. The image can come from a web address (URL), from base64 data (the picture turned into plain text characters, plus a mime type that names the file kind), or from a file ID, a reference to a file the provider already stores for you.
# From the LangChain docs (Messages, Multimodal), shown without its output (not run here)
# From URL
message = {
"role": "user",
"content": [
{"type": "text", "text": "Describe the content of this image."},
{"type": "image", "url": "https://example.com/path/to/image.jpg"},
]
}Audio follows the same pattern. The docs show an audio block with a base64 string and a mime type of "audio/wav", or with a file ID. The page's tabs also cover PDF documents and video. It adds that extra keys can go in a block, and that some providers need them, so you must read your chosen provider's page.
Tip
The snippet above is a message you build, not a call to a model. Nothing is sent anywhere until you hand the message to a model. A real model is slower than a local script, costs money for every token, can give a different answer each run, and can fail or refuse.
LlamaIndex: mixed documents
LlamaIndex is a framework built around your own documents. Its Multi-modal page says it offers capabilities to build not only language-based applications but also multi-modal applications, combining language and images.
For retrieval-augmented generation, the page says that all the core RAG concepts, indexing, retrieval and synthesis, can be extended into the image setting. Wikipedia describes RAG as a technique that lets a language model retrieve and use information from external data sources: the model first refers to a chosen set of documents, then answers. The LlamaIndex page then lists four places where an image can appear.
- 1
The question
The input could be text or an image.
- 2
The stored knowledge
The knowledge base can hold text or images.
- 3
The context for the answer
The inputs to response generation can be text or an image.
- 4
The final reply
The final response can be text or an image.
The Multi-modal models page gets practical. It says LlamaIndex's built-in MultiModalVectorStoreIndex supports building separate vector stores for image and text embedding vector stores. A vector store, from the Choosing a vector store lesson, is a database that finds items by meaning. So text and pictures get their own shelves.
It adds that a MultiModalRetriever and a SimpleMultiModalQueryEngine support retrieval from text to text and image, and from image to image, plus simple ranking fusion functions for combining text and image results. Fusion here means merging two ranked lists into one.
Here is the docs' own snippet for building that index. It reads a folder that holds both text and image files, then creates the index. The storage settings that come before it in the docs are left out.
# From the LlamaIndex docs (Multi-modal models), shown without its output (not run here)
# Load text and image documents from local folder
documents = SimpleDirectoryReader("./data_folder/").load_data()
# Create the MultiModal index
index = MultiModalVectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
)One more idea from the use-case page is worth knowing. It calls it retrieval-augmented image captioning: first caption the image with a multi-modal model, then refine the caption by retrieving it from a text corpus. A corpus is a big collection of text. In plain words: a model says what it sees, and then a search through your own notes fixes the details.
Quick check
In LangChain's docs, what does a standard content block buy you?
A multimodal RAG pipeline in plain steps
Strip the framework names away and every multimodal RAG app does the same seven things. Learn the shape once and any tool's docs become easy to read.
- 1
Extract
Pull the text and the images out of each document, such as a PDF. Keep the page number for each piece.
- 2
Split the text
Cut long text into chunks, as in the Chunking lesson. Each image stays whole.
- 3
Caption or embed the images
Either ask a vision-capable model to write a caption for each picture, or turn the picture into an embedding, a list of numbers that stands for its meaning, using a model made for pictures.
- 4
Store with a link back
Save each piece with its source: file name, page number and, for images, the image file name. This lets you show the user where an answer came from.
- 5
Retrieve both
Search for the best text chunks and the best images for the question.
- 6
Build the message
Put the chosen text and the chosen images into one message, with the question last.
- 7
Ask a vision-capable model
Send the message. The model reads the words and looks at the pictures, then answers.
Two choices in that list matter most. Step 3 decides how pictures are searched. A caption is plain text, so your normal text search works and you can read what was stored. An embedding can match a picture to a question directly, but it is harder to inspect. Step 4 matters because pictures without a source are hard to trust.
Wikipedia notes a limit that still applies: according to a source it quotes, RAG is not a direct solution to made-up answers, because the model can still hallucinate around the source material in its response. Hallucinate means state something that is not true, in a confident voice. Retrieval helps, but you still test the answers.
Tip
The Tokenization lesson in Machine Learning Foundations goes deeper on how text is counted. Models count the size of your message, and a message with several images and chunks is bigger than a plain question. Keep the number of retrieved parts small.
Build a toy multimodal index
Now you build the pipeline yourself, small. The program has five made-up parts of a bike repair manual: three text parts and two image entries. It follows steps 3 to 6 above.
One honest point first. A real app would send each picture to a vision model to get its caption. Here the captions are typed by hand, so they stand in for what a vision model would write. No picture file exists, only its name.
The search is simple on purpose: it counts how many meaningful words the question shares with each part, after dropping small words like "the". Image parts are searched by their caption but keep a pointer to their file name. At the end, the program builds a message list with text parts and image references, and prints it.
import re
# Five made-up parts of a bike repair manual. Image captions stand in for a vision model.
parts = [
{"id": 1, "kind": "text", "page": 3, "text": "To fix a flat tyre, remove the wheel and take the tube out."},
{"id": 2, "kind": "image", "page": 3, "file": "tube-removal.png", "caption": "a hand pulling a bike tube out of the tyre"},
{"id": 3, "kind": "text", "page": 4, "text": "Check the tyre for sharp stones before you fit a new tube."},
{"id": 4, "kind": "image", "page": 5, "file": "brake-cable.png", "caption": "a brake cable running along the frame"},
{"id": 5, "kind": "text", "page": 5, "text": "Brake pads wear out. Replace them when the grooves are gone."},
]
STOP = {"a", "the", "to", "do", "i", "how", "is", "it", "of", "you", "before", "when", "out"}
def words(s):
return {w for w in re.findall(r"[a-z]+", s.lower()) if w not in STOP}
# Index: every part becomes searchable text. Images are searched by their caption.
index = []
for p in parts:
searchable = p["text"] if p["kind"] == "text" else p["caption"]
index.append({"part": p, "words": words(searchable)})
def search(question, top=3):
q = words(question)
scored = [(len(q & item["words"]), item["part"]) for item in index]
scored = [s for s in scored if s[0] > 0]
scored.sort(key=lambda s: (-s[0], s[1]["id"]))
return [p for _, p in scored[:top]]
def build_messages(question, hits):
content = []
for p in hits:
if p["kind"] == "text":
content.append({"type": "text", "text": f"(page {p['page']}) {p['text']}"})
else:
content.append({"type": "image", "file": p["file"]})
content.append({"type": "text", "text": "Question: " + question})
return [
{"role": "system", "content": "Answer only from the parts given."},
{"role": "user", "content": content},
]
def fake_model(messages):
# Stand-in: it never looks at anything. A real model would read the text and the pictures.
return "(fixed reply from the stand-in)"
question = "How do I get the tube out of a flat tyre?"
hits = search(question)
print("hits:", [(p["id"], p["kind"]) for p in hits])
messages = build_messages(question, hits)
for m in messages:
print(m["role"] + ":")
if isinstance(m["content"], str):
print(" " + m["content"])
else:
for c in m["content"]:
print(" " + str(c))
print(fake_model(messages))Output
hits: [(1, 'text'), (2, 'image'), (3, 'text')]
system:
Answer only from the parts given.
user:
{'type': 'text', 'text': '(page 3) To fix a flat tyre, remove the wheel and take the tube out.'}
{'type': 'image', 'file': 'tube-removal.png'}
{'type': 'text', 'text': '(page 4) Check the tyre for sharp stones before you fit a new tube.'}
{'type': 'text', 'text': 'Question: How do I get the tube out of a flat tyre?'}
(fixed reply from the stand-in)Read the output from the top. The search returned three parts: part 1 (text), part 2 (an image) and part 3 (text). Part 1 shares the most words with the question: flat, tyre, tube. Parts 2 and 3 share two each, and ties are broken by the lowest id. The brake cable and the brake pads shared nothing, so they were left out.
Then the message. The user turn is a list of four blocks: three that came from the search, then the question. The image block holds only the file name, tube-removal.png. A real app would swap that name for the picture itself, as a URL, base64 data or file ID, the same three ways LangChain's docs list. Notice also that the text block for part 3, from page 4, came along because it matched on words. A search is not a judge of quality, so even good ones return a weak part now and then.
The last line is the stand-in model. It returns the same fixed sentence every time and never looks at the text or the picture. It exists only so the program runs free, with no key and no network. A real vision model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.
Tip
Try it yourself: change the question to "How do I fix a brake?" and see which parts come back. Then add a sixth part, an image with a caption you write, and check that it can be found.
When to skip the framework
You just built the heart of a multimodal search in about 55 lines. That is a useful test. A framework earns its place when it saves you work you would otherwise write and maintain, and it costs you when it hides what happens. The Frameworks for AI apps lesson made the same point about text-only apps.
Use a framework when
You read many file types, you want ready-made connectors for vector stores and providers, or you expect to swap providers. The standard message types and the built-in multimodal index in the docs are exactly this kind of saving.
Skip it when
You have one model, one kind of document and a small amount of data. A plain script like the toy above is easier to read, test and debug.
Think twice when
Your team must explain every step to a customer or a reviewer. Hidden layers make that harder. If you do use a framework, keep a trace of what each step received and returned.
Another reason to be careful: these libraries change quickly. The LlamaIndex page you read here describes itself as covering a space that is "actively being explored right now", and the snippets depend on the exact package versions installed. Pin your versions and re-read the docs before an upgrade.
A simple rule: build the first version with plain code, as in this lesson. When you find yourself rewriting the same chunk of plumbing a third time, look for the framework that already has it.
Quick check
You have 40 short text notes, one photo folder and one model. What is a sensible start?
Quick check
In the toy index, how is an image found by a question?
Next: AI coding tools, assistants that help you build all of this.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.