esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Core / 02.22

Frameworks for AI apps

What LLM app frameworks bundle, what they cost, and when plain code or a provider's SDK is enough. Plus a 15-line toy that shows how they work.

10 min readFree

The libraries that bundle all of this

The last lesson, Multi-agent systems, ended with a promise: next, frameworks, the libraries that bundle all of this. Here they are.

Over this course you built many small parts by hand: a way to find notes, a way to build a prompt, a call to a model, a tool, a loop. A framework is a ready-made box of those parts, with the wiring done for you. Think of a meal kit. The box has the chopped vegetables and the sauce, and you still cook. It saves time. It also means you cook the way the box expects.

By the end you will know what four well-known frameworks say they are for, what any framework gives you and what it costs, and when you should skip one. You will also run a 15-line teaching toy that shows what a framework does under the hood.

Key words

Framework
A library that gives you ready-made parts and a set way to connect them, so you write less glue code.
Chain
Steps that run one after another, where each step's result feeds the next. Find notes, build a prompt, ask the model.
Pipeline
Another word for a chain of steps. Some tools use one word, some the other.
Connector
A ready-made piece that reads from somewhere: a file, a database, a web page, or a model provider.
Trace
A record of what each step received and returned, in order. It is the receipt of one run.
Lock-in
When your code leans so hard on one tool that moving away is painful.
SDK
A provider's own code kit for calling its models. The Using SDKs directly lesson covers it.

Four frameworks, in their own words

Many frameworks exist. We look at four from the course plan. This lesson only reports what each project says about itself on its own docs pages, checked on 16 November 2026. It does not rank them. Each one aims at a different job.

  • Langchain

    Its overview says "LangChain provides `create_agent`: a minimal, highly configurable harness." It adds that the harness is "everything around the model loop: the prompt, the tools, and any middleware that shapes behavior."

  • LlamaIndex

    Its docs call the framework "a Python toolkit for building LLM-powered agents over your data." Its parts include data connectors, indexes and vector stores, query and chat engines, agents and tools, and workflows.

  • Haystack

    Its intro calls it "an open-source AI framework for building production-ready AI Agents, powerful RAG applications and scalable multimodal search systems." Its core is components and pipelines.

  • RAGFlow

    Its README calls it "a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities". Its quickstart starts servers with Docker Compose, so it behaves more like an app you run than a library you import.

  • One more fact: LangChain's neighbours

    The LangChain page names two relatives. LangGraph is "our low-level orchestration framework", and LangSmith is for tracing: it says to use LangSmith "to trace, debug, and evaluate agents built with any of these frameworks."

  • One more fact: Haystack's choice

    Haystack says a pipeline "is a graph you wire yourself, so it runs the same way every time", while an agent "hands step-by-step control to the LLM".

A quick table puts the four side by side. It only repeats what the pages above say.

Key words

Langchain: focus
A configurable agent harness: prompt, tools, middleware. Tracing through a sister product, LangSmith.
LlamaIndex: focus
Building agents over your own data, with connectors, indexes, query engines and workflows.
Haystack: focus
Pipelines of reusable components for RAG, agents and search. Docs list deployment, evaluation, tracing and logging as next steps.
RAGFlow: focus
An open-source RAG engine you run with Docker Compose, with agent features on top.

Tip

Framework facts here were checked on 16 November 2026. These APIs change often, so treat every snippet as a picture of the idea and check the current docs before you copy it.

One snippet from each quickstart

Here is the first example from each project's own docs, copied as shown. They are labelled "not run here": most of them need a paid model and a key, and this course never asks you to spend money. Read them for the shape, not the details.

Langchain first. You write a normal Python function, hand it to create_agent as a tool, and call invoke with a message.

Text
# From the LangChain overview docs, shown without its output (not run here)
# pip install -qU langchain "langchain[openai]"
from langchain.agents import create_agent

def get_weather(city: str) -> str:
    """Get weather for a given city."""
    return f"It's always sunny in {city}!"

agent = create_agent(
    model="openai:gpt-6.1-sol",
    tools=[get_weather],
    system_prompt="You are a helpful assistant",
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "What's the weather in San Francisco?"}]}
)
print(result["messages"][-1].content_blocks)

LlamaIndex next. Same idea, a plain function as a tool, wrapped by its FunctionAgent. The tutorial notes it uses async code, which just lets Python wait for slow calls without freezing.

Text
# From the LlamaIndex starter tutorial, shown without its output (not run here)
import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

# Define a simple calculator tool
def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

# Create an agent workflow with our calculator tool
agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can multiply two numbers.",
)

async def main():
    # Run the agent
    response = await agent.run("What is 1234 * 4567?")
    print(str(response))

# Run the agent
if __name__ == "__main__":
    asyncio.run(main())

Haystack wires named components together. Each add_component line places a part, and each connect line says whose output goes into whose input. The retriever, prompt_builder and llm parts are created earlier in the same docs page.

Text
# From the Haystack Get Started page, shown without its output (not run here)
rag_pipeline = Pipeline()
rag_pipeline.add_component("retriever", retriever)
rag_pipeline.add_component("prompt_builder", prompt_builder)
rag_pipeline.add_component("llm", llm)
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder", "llm")

RAGFlow's quickstart is not Python code at all. You download it, pick a release, and start the services.

Text
# From the RAGFlow README, shown without its output (not run here)
git clone https://github.com/infiniflow/ragflow.git
cd ragflow/docker
git checkout v1.0.0
docker compose -f docker-compose.yml up -d

Spot the pattern. Three of the four give you named parts and a way to connect them. That is the one idea worth understanding, so the next section builds it from scratch.

What a framework gives you

From the docs above, frameworks offer three kinds of help.

  1. 1

    Connectors

    Ready-made code to load files, talk to many model providers and use many stores. You skip writing the same adapter again.

  2. 2

    Ready pieces

    Common RAG and agent parts already built: a retriever, a prompt builder, a tool-using agent. You pick, you wire.

  3. 3

    Hooks for tracing

    A place to see what happened in a run, which matters once something goes wrong.

The Build RAG by hand lesson wrote a retriever and a prompt builder yourself. A framework swaps those for parts someone else already tested.

What a framework costs

  • More layers to debug

    When the answer is wrong, you now ask: is it my data, my prompt, or something inside the framework? The extra layer hides the prompt that was really sent.

  • Fast-changing APIs

    These projects move quickly. A tutorial from last year may not run today. That is why this lesson tells you to re-check the docs.

  • Lock-in

    If your whole app is written in one framework's shapes, switching later means rewriting. Keep your own logic in plain functions and the framework at the edges.

Quick check

Your app calls one model once and shows the reply. What is usually enough?

A tiny framework you can run

Frameworks sound magic until you build one. This is a teaching toy, not a real framework. The Chain class below is about 15 lines. It runs your step functions in order, passes each result to the next step, and writes a trace line for every step. That is the core of what chains and pipelines do.

It rebuilds the three-step RAG flow from earlier lessons: retrieve a note, build a prompt, ask a model. The model is a stand-in called fake_model that returns fixed text, so the code runs free. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The stand-in does none of that. Do not mistake it for the real thing.

Python
class Chain:
    def __init__(self, *steps):
        self.steps = steps
        self.trace = []

    def run(self, value):
        for step in self.steps:
            try:
                value = step(value)
            except Exception as err:
                self.trace.append((step.__name__, "failed", str(err)))
                raise
            self.trace.append((step.__name__, "ok", str(value).replace("\n", " / ")[:44]))
        return value


NOTES = [
    "Refunds are paid within 5 days.",
    "Shipping to Spain takes 3 days.",
    "Support is open on weekdays.",
]


def retrieve(question):
    words = set(question.lower().strip("?").split())
    scored = [(len(words & set(n.lower().strip(".").split())), n) for n in NOTES]
    best = max(scored)[1]
    return {"question": question, "note": best}


def build_prompt(data):
    return "Answer from this note only.\nNote: " + data["note"] + "\nQuestion: " + data["question"]


def fake_model(prompt):
    # Stand-in so the code runs free: always the same reply.
    return "Answer (stand-in): " + prompt.split("Note: ")[1].split("\n")[0]


rag = Chain(retrieve, build_prompt, fake_model)
print(rag.run("How many days do refunds take?"))
for name, status, preview in rag.trace:
    print(name, status, "|", preview)

def broken(prompt):
    raise ValueError("model timed out")

bad = Chain(retrieve, build_prompt, broken)
try:
    bad.run("Is support open?")
except ValueError:
    pass
print(bad.trace[-1])

Output

Answer (stand-in): Refunds are paid within 5 days.
retrieve ok | {'question': 'How many days do refunds take?
build_prompt ok | Answer from this note only. / Note: Refunds 
fake_model ok | Answer (stand-in): Refunds are paid within 5
('broken', 'failed', 'model timed out')

Read the output. The first line is the answer. The next three lines are the trace: one per step, with its name, whether it worked, and a short preview of what it returned. The second step's preview shows the prompt that was built, which is the thing you most want to see when an answer looks wrong.

The last line is a failure. We swapped the final step for one that always raises an error. The chain wrote down which step failed and why, then let the error out. Real frameworks do the same job with more care: retries, timeouts, saving traces to a dashboard.

Tip

The toy has no retries, no streaming, no async and no real memory. Real frameworks add those. The shape is the same: ordered steps, passed values, a trace.

Quick check

In the toy, what does the trace let you do?

When you do not need one

Pick the smallest tool that does the job.

  • Provider SDK is enough

    One model, a few calls, maybe a tool or two. The SDK plus a short loop of your own code covers it.

  • Hand-rolled is enough

    A small RAG over a few hundred notes, like the Build RAG by hand lesson. You can read every line, so you can fix every line.

  • A framework earns its place

    Many data sources, many model providers, tracing across a team, or pieces you would otherwise rewrite and keep tested yourself.

A fair test: could you explain, in a minute, what your app does between the user's message and the answer? If a framework makes that harder instead of easier, it is not paying for itself yet.

Quick check

Which sign suggests a framework could help?

What to remember

  1. 1

    A framework is a meal kit

    Ready parts plus set wiring. Faster to start, less say in how it is cooked.

  2. 2

    Gives and costs

    It gives connectors, ready RAG and agent pieces, and tracing hooks. It costs extra layers, changing APIs and lock-in.

  3. 3

    Start small

    Use the SDK or plain code first. Reach for a framework when the glue code grows.

Next: agent SDKs, ready-made kits for building agents.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.