esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.14

Streaming Responses

Why an answer shown word by word feels faster, how it travels over the web, and how to handle each piece, errors and a stop button.

13 min readFree

Showing the answer as it is written

The last lesson, ReAct, ended with a promise: next, streaming responses, showing the answer as it is written. Here it is.

Think about two ways to get a letter. In the first, the post office holds every page until the whole letter is ready, then hands it over in one go. In the second, they give you each page the moment it is done. You can start reading page one while page two is still being written.

A model can work the second way. This is called streaming. It is a big part of why a chat assistant feels alive instead of frozen.

By the end you will know why streaming makes an app feel faster, how the pieces travel over the web, what to do with each piece that arrives, what to do when something goes wrong or the reader presses stop, and one catch that surprises most beginners. You will run three short Python programs. None of them calls a real model.

Key words

Streaming
Getting the answer in small pieces as it is made, instead of waiting for the whole answer.
Chunk
One small piece of a streamed answer. Often a word, part of a word, or a few words.
Time to first token
How long you wait before the first piece shows up. A token is the small unit of text a model reads and writes; the Tokens lesson explains it.
Total time
How long until the very last piece has arrived.
Server-sent events (SSE)
A simple way for a server to keep a connection open and push text to the page, piece by piece.
End signal
A marker that says the answer is finished, so the app knows to stop waiting.

Why streaming matters

A model writes its answer one small piece at a time, left to right. That is how it works inside. Streaming lets you see those pieces as they appear, instead of hiding them until the last one is done.

OpenAI's guide for its API puts the problem plainly: "When generating long outputs, waiting for a response can take time." Then it names the fix: "Streaming responses lets you start printing or processing the beginning of the model’s output while it continues generating the full response."

Three ideas are worth keeping apart.

  • Time to first token

    How long until anything shows up. With no streaming, it is the same as the total time. With streaming, it can be very short.

  • Total time

    How long until the answer is complete. Streaming does not make the model write faster. The last word still lands at about the same moment.

  • Perceived speed

    How fast it feels. A page that starts filling in right away feels quicker than a blank page that makes you wait, even when both finish together.

Think of a restaurant. If the kitchen sends every dish out at once, you sit at an empty table until the slowest dish is ready. If it sends each dish as it is done, you start eating early. The meal takes the same time. It just does not feel like waiting.

Tip

Streaming improves the wait, not the work. If your app needs the whole answer before it can do anything (say, to run a calculation on it), streaming will not make that part faster.

Quick check

What does streaming mostly improve?

How the pieces travel

On the web, a normal request is like a letter exchange. The page asks, the server replies once, and the connection is done. For streaming, the server keeps the connection open and sends text down it little by little.

The usual way to do this is server-sent events, or SSE. Both OpenAI and Anthropic describe their streaming that way. OpenAI's guide says it focuses on "HTTP streaming (stream=true) over server-sent events (SSE)", and Anthropic's page says you can "incrementally stream the response using server-sent events (SSE)".

MDN, the web's reference site, explains the idea. It is a one-way street: "This is a one-way connection, so you can't send events from a client to a server." The server talks. The page listens.

The format is plain text

An event stream is just text. MDN says the server must reply with the type text/event-stream, and that "Each notification is sent as a block of text terminated by a pair of newlines." So a blank line means "that piece is complete".

Inside a block, each line has a name, a colon, and a value. The line you will see most is data:, which carries the text. MDN adds that a line starting with a colon is a comment and is ignored, and that a server can send one now and then to keep the connection alive.

In the browser, a built-in tool called EventSource listens for these blocks. MDN notes that by default, "if the connection between the client and server closes, the connection is restarted", and that you end it yourself by calling the close() method.

Real services add their own details on top. Anthropic's stream, for example, uses named events such as message_stop for the last one, and sends occasional ping events. OpenAI's uses its own event names too. So always read the docs of the service you use.

Reading the event text yourself

You usually let a library do this, but it helps to see the format once. This program takes a short block of SSE-style text and reads it the way a browser would: blocks are separated by a blank line, lines starting with a colon are ignored, and data: lines carry the words.

One honest note. The end marker [DONE] in this example is made up for the lesson. Real services pick their own way to say "finished". The text here is not from any model. It is a fixed string.

Python
raw = (
    "data: Streaming\n\n"
    "data: shows\n\n"
    "data: words\n\n"
    ": keep-alive comment\n\n"
    "data: early.\n\n"
    "data: [DONE]\n\n"
)
# [DONE] is a made-up end marker for this example. Real services choose their own.

text = ""
for event in raw.split("\n\n"):
    for line in event.split("\n"):
        if line.startswith(":") or not line:
            continue
        field, _, value = line.partition(":")
        value = value.lstrip(" ")
        if field == "data" and value == "[DONE]":
            print("end marker seen")
        elif field == "data":
            text += value + " "
            print("got:", value)

print("final text:", text.strip())

Output

got: Streaming
got: shows
got: words
got: early.
end marker seen
final text: Streaming shows words early.

Each data: line gave one piece. The comment was skipped. The end marker was caught, so the program knew to stop. The final text was built by adding the pieces in order.

Try it with a stand-in

Time to see the idea run. The next program has a function called fake_model_stream. It is a stand-in. It is not a real model. It always gives the same sentence and sends it one word at a time, with a tiny pause between words. That keeps the code free and runs on your own computer.

A real model is different. It is slower, it costs money for every token it writes, it can give a different answer each run, and it can fail or refuse. We use a stand-in so you can see the streaming idea clearly, nothing more.

The program keeps a clock. It notes when the first piece arrives, adds up the pieces, prints each one on the same line as it comes, and checks the clock at the end.

Python
import time

def fake_model_stream(question):
    # Stand-in: always gives the same answer, one word at a time.
    answer = "Streaming sends the answer in small pieces while it is still being written."
    for word in answer.split():
        time.sleep(0.01)
        yield word + " "

start = time.perf_counter()
first_chunk_at = None
text = ""
for chunk in fake_model_stream("What is streaming?"):
    if first_chunk_at is None:
        first_chunk_at = time.perf_counter() - start
    text += chunk
    print(chunk, end="", flush=True)
total = time.perf_counter() - start

print()
print("chunks:", len(text.split()))
print("first piece arrived before the end:", first_chunk_at < total)
print("first piece took under a third of the total:", first_chunk_at < total / 3)

Output

Streaming sends the answer in small pieces while it is still being written. 
chunks: 13
first piece arrived before the end: True
first piece took under a third of the total: True

The sentence appeared as one line, built word by word. Then the program counted 13 chunks. The first piece arrived before the end, and it took under a third of the total time. If you waited for everything, you would see nothing until the last word. With streaming, you could start reading after the first one.

The program prints true or false, not raw seconds, because exact times change on every run and every computer.

What to do with each piece

Receiving pieces is easy. Handling them well takes a few habits.

  1. 1

    Append the text

    Keep a string in your app. Every time a chunk arrives, add it to the end and redraw. Never replace the text with the newest chunk, or the reader sees only the last word.

  2. 2

    Watch for the end signal

    The stream tells you when it is finished. Until then, show that the answer is still being written. When it arrives, turn off the cursor and enable any buttons that wait for a full answer.

  3. 3

    Handle errors

    A stream can break halfway: the network drops, or the service sends an error. Keep what arrived, tell the reader plainly what happened, and offer a retry.

  4. 4

    Handle the stop button

    Let the reader cancel. Stop reading, close the connection, and keep the text that already arrived.

  5. 5

    Ignore what you do not know

    Services add new event types over time. Skip the ones you do not recognise instead of crashing.

Errors in the middle

With a normal request, the answer either arrives or it does not. With a stream, it can arrive halfway and then fail. This is a new situation, and you should plan for it.

Anthropic's streaming page says so directly: "The API might occasionally send errors in the event stream." Its example is an overloaded_error, which means the service was too busy. In the browser, MDN says EventSource raises an error event, and you can react by setting an onerror handler.

Anthropic's page also says "your code should handle unknown event types gracefully", because new types may be added. A beginner's version of that sentence is: if you see something you do not understand, skip it and keep going.

Tip

Decide what a half answer means before you ship. Showing it with a note like "The answer was cut off. Try again?" is honest. Showing it as if it were complete is not.

When the reader presses stop

Sometimes the first two lines are enough. The reader sees the answer is going the wrong way and wants to stop. A good stop button does three things: it stops reading new chunks, it closes the connection so the server can stop too, and it keeps the text already shown. Throwing it away would feel like losing work.

The same goes for the browser tab. MDN's example server checks whether the client has left and ends its loop, so it does not keep working for nobody. A stream you forgot to close wastes effort on both sides.

The catch: structured output

Plain text is easy to stream. Structured output is not. Say you ask a model for a small JSON answer, like a city and a number of days. The pieces arrive broken at odd places, such as the text {"city": "Par, with the rest coming later.

A half JSON message is not valid. If you try to read it too soon, your program fails. Anthropic's docs describe the same thing for tool calls: "the deltas are partial JSON strings", and you can "accumulate the string deltas and parse the JSON once you receive a content_block_stop event".

The next program shows it. The generator is again a stand-in with fixed pieces. Part one tries to read the text after every piece. Part two pretends the reader pressed stop after two pieces.

Python
import json

def fake_json_stream():
    # Stand-in: sends one JSON answer in broken-up pieces.
    for piece in ['{"city": "Par', 'is", "days": ', '3}']:
        yield piece

# Part 1: the JSON only makes sense at the end.
buffer = ""
for piece in fake_json_stream():
    buffer += piece
    try:
        json.loads(buffer)
        print("valid after:", repr(buffer))
    except json.JSONDecodeError:
        print("not valid yet:", repr(buffer))

# Part 2: the reader presses stop after two pieces.
stop_pressed = False
shown = ""
for count, piece in enumerate(fake_json_stream(), start=1):
    shown += piece
    if count == 2:
        stop_pressed = True
    if stop_pressed:
        break

print("stopped early, kept:", repr(shown))
try:
    json.loads(shown)
except json.JSONDecodeError:
    print("cannot parse a half answer")

Output

not valid yet: '{"city": "Par'
not valid yet: '{"city": "Paris", "days": '
valid after: '{"city": "Paris", "days": 3}'
stopped early, kept: '{"city": "Paris", "days": '
cannot parse a half answer

Only the final piece made the JSON valid. The first two tries failed, and the stopped version could never be read at all. So the rule is: show streamed text freely, but check and use structured data only after the end signal. If the reader stops early, treat the result as unusable, not as a smaller answer.

There is a related problem with safety checks. OpenAI's guide warns that streaming "makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate". If your app needs to check the answer before showing it, streaming and checking pull in opposite directions.

Quick check

A streamed JSON answer has arrived only halfway. What should your code do?

Design tips for the screen

Streaming is as much a design job as a code job. These habits make a streamed answer feel calm and in your control.

  • Show a cursor

    A small blinking bar at the end of the text tells the reader the answer is still arriving. Remove it the moment the end signal comes.

  • Always allow stop

    A visible stop button while text is arriving gives control back. Turn it into a retry button if something fails.

  • Do not jump the layout

    Text that grows pushes things around. Reserve space for the answer, keep buttons in the same place, and only follow the newest line if the reader is already at the bottom.

Tip

Test with a slow connection, not just your fast one. Pieces that arrive in bursts, or stall for a second, show you the ugly cases before your readers find them.

Recap

  1. 1

    Streaming sends the answer in pieces

    The reader sees the start early. The total time is about the same.

  2. 2

    SSE is the common road

    Plain text, blocks split by a blank line, data: lines carrying the words, and an end signal.

  3. 3

    Append, watch the end, handle trouble

    Add each chunk, know when it is finished, and plan for errors and stop.

  4. 4

    Wait before you parse

    Structured output only makes sense once it is complete.

Quick check

Which button should a streaming answer always have?

Next: context engineering, choosing what the model gets to see.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.