AI engineering / Beginner / 01.20
State & Historical Context
A model forgets everything between calls. Your app has to store the chat and send it back each time, without letting it grow out of control.
13 min readFree
How a chat remembers what was said
The last lesson, Context Security, ended with a promise: next, state and historical context, how a chat remembers what was said. Here is the surprise. It does not remember. Your app does the remembering.
Think of a shop assistant with a new face every time you walk in. Each visit, they know nothing about you. If you want them to continue yesterday's talk, you must hand them your notes. A model works like that assistant. Every call starts fresh.
By the end you will know why that is, where your app can keep the notes, what belongs in them, why long notes cost money, and three ways to keep them short. You will also run two small programs that store chats for two different users.
Key words
- State
- What has happened so far: the messages, the settings, the facts you have learned about the user.
- Stateless
- Keeping nothing between requests. Each request is read on its own, as if it were the first.
- History
- The list of earlier messages in a chat, in order.
- Turn
- One message in a chat, from either the user or the model.
- Conversation ID
- A label, such as c-101, that points to one chat. Every message of that chat is stored under it.
- User ID
- A label for one person. It lets you keep one person's chats apart from another's.
- Pinned fact
- A short fact you always send, even after the old messages are dropped.
- Token
- A small piece of text, like a word or part of a word. The Tokens lesson covers it, and the Tokenization lesson in Machine Learning Foundations goes deeper.
Models are stateless
The word for this is stateless. Wikipedia describes a stateless protocol as one in which "the receiver does not retain session state from previous requests." It adds that the sender passes along what is needed "in such a way that every request can be understood in isolation."
The same idea shows up in the OpenAI docs on conversation state, which say: "While each text generation request is independent and stateless, you can still implement multi-turn conversations by providing additional messages as parameters to your text generation request."
Read that twice. Each request stands alone. A multi-turn chat is not something the model keeps. It is something you build by sending the earlier messages again.
So when a chat app seems to remember your name, the app is doing it. It saved your earlier messages and put them back in the next request.
Tip
Wikipedia also names a cost of this design: stateless protocols "may decrease network performance by increasing the repetitive data sent in a series of requests." Chats pay that cost every turn, and the cost is counted in tokens.
Where the state lives
If the model keeps nothing, someone must. You have three common places. They differ in who holds the notes.
In the browser
The page keeps the messages in memory or local storage and sends them with each request. Simple and cheap. But close the tab or switch device and the chat is gone, and a user can change what is stored.
In your database
Your server saves every turn under a conversation ID. It survives closing the tab, works on any device, and you control it. You also have to build it and look after it.
In the provider's stored chat
Some providers can hold the chat for you. The OpenAI docs describe a Conversations API that persists state as "a long-running object with its own durable identifier." You pass that ID and the provider supplies the earlier messages.
None of these is the winner for every product. A quick demo can live in the browser. An app people come back to usually needs a database. A provider's stored chat saves you from building storage, but it ties your chat history to that provider's feature.
A stored chat does not make the model remember, either. It only moves the notes to someone else's shelf. The same OpenAI page says: "Even when using previous_response_id, all previous input tokens for responses in the chain are billed as input tokens in the API." So the earlier messages are still read, and still counted, on every call.
Quick check
A chat app seems to remember your name from five messages ago. Who really remembers it?
What to store
A chat is more than text. The OpenAI docs say stored conversations hold "items, which can be messages, tool calls, tool outputs, and other data." Your own store should think the same way.
Key words
- Messages
- What the user said and what the model answered, each with its role (user or assistant).
- Tool results
- What a tool returned, such as a weather lookup or an order status. The model needs them to explain its next step. The Function Calling lesson covers tools.
- Decisions
- Choices made along the way: the user picked the blue plan, or approved a refund. Store them as plain facts.
- User settings
- Language, tone, units. These rarely change, so they fit well in a pinned line.
Store each item with a time and the conversation ID. Later you can show the chat again, debug a wrong answer, or count the cost. Do not store more than you need. Chats can hold private details, and anything saved must be protected. The Context Security lesson explains why.
IDs: one per chat, one per person
Two labels keep everything tidy. A conversation ID says which chat a message belongs to. A user ID says whose chat it is.
Think of a coat check. Your ticket number finds your coat, not someone else's. Without tickets, coats get mixed up. Without IDs, one user can be shown another user's chat. That is a serious bug, not a small one.
- 1
Make a conversation ID when a chat starts
Use a fresh, hard-to-guess value. Store it with the user ID.
- 2
Save every turn under that ID
Both the user's message and the model's answer.
- 3
Check the owner on every request
Before you load a chat, confirm its user ID matches the person asking.
- 4
Build the next request from the stored chat
Take what you need, in order, and send it.
A person can have many conversation IDs, one per chat. A conversation belongs to exactly one person. That is the whole map.
History grows, and so does the bill
Here is the catch. If you send everything each time, the request gets longer with every turn. The Anthropic docs describe it: "As the conversation advances through turns, each user message and assistant response accumulates within the context window, and previous turns are preserved completely."
Picture a shared notebook where each new page must be photocopied along with all the pages before it. By page twenty you are copying twenty pages to add one line.
Three things go wrong as it grows. The request gets slower. It costs more, because you pay for every token you send. And it can fill the context window, the most text a model can read in one request. The Context Window lesson covers that limit.
There is a fourth problem, and it is less obvious. The same Anthropic page says "more context isn’t automatically better. As token count grows, accuracy and recall degrade." So a very long history can make answers worse, not better.
Tip
Memorise this: the bill for a chat does not grow by a small step each turn. It grows by the whole history each turn. Keeping the history short is a cost decision as much as a technical one.
Three ways to keep history short
You do not have to send every message. You choose what goes in. The Anthropic page notes that chat apps do something similar: they "can also manage the context window on a rolling ‘first in, first out’ basis." That means the oldest messages leave first.
Keep the last N turns
Send only the most recent few messages. Easy and predictable. The risk: something important said early, like the user's name, falls off the end.
Summarise the older turns
Replace a long stretch of old messages with a short summary. You keep the meaning and lose the detail. Compaction gets its own lesson later.
Pin key facts
Keep a short list of facts that must never be lost, and send it every time. Cheap, and it protects exactly what keep-the-last-N would drop.
Most real apps mix them. They pin a few facts, summarise the far past, and keep the recent turns word for word. Let us build the simplest mix: pinned facts plus the last four turns.
Worked example: a tiny chat store
First, proof that a model only knows what it is sent. Below is a stand-in model, a few lines of plain Python that fakes a model so the code runs free. It reads only the messages we hand it.
A real model is different in ways that matter. It is slower, it costs money for every token, it can give a different answer each run, and it can fail or refuse. This stand-in never does any of that.
# A stand-in "model": it only reads the messages it is handed.
def fake_model(messages):
name = None
for m in messages:
text = m["text"].lower()
if m["role"] == "user" and "my name is " in text:
name = text.split("my name is ")[1].split()[0].strip(".!?").title()
question = messages[-1]["text"].lower()
if "what is my name" in question:
return f"Your name is {name}." if name else "Sorry, I do not know your name."
return "Nice to meet you."
# Call 1: send only the newest message.
only_last = [{"role": "user", "text": "What is my name?"}]
print("Last message only:", fake_model(only_last))
# Call 2: send the whole conversation so far.
whole = [
{"role": "user", "text": "Hi, my name is Asha."},
{"role": "assistant", "text": "Nice to meet you."},
{"role": "user", "text": "What is my name?"},
]
print("Whole history: ", fake_model(whole))Output
Last message only: Sorry, I do not know your name. Whole history: Your name is Asha.
Same stand-in, same question. With only the last message, it has no name to give. With the whole history, it finds "Asha". The only difference is what we sent.
Now the store. It keeps chats in a dictionary keyed by conversation ID. Each chat records its user ID, its turns and a list of pinned facts. The function build_messages makes the list for the next call: one line of pinned facts, then the last four turns.
Again the model is a stand-in. It just echoes the last message, so we can focus on the storage. The word counts are a rough stand-in for tokens, since real tokens are not whole words. The printout counts one question and its answer as one turn, but the store keeps every message on its own, and "the last four turns" means the last four messages.
# One in-memory store, keyed by conversation id.
store = {}
def new_conversation(conv_id, user_id, pinned):
store[conv_id] = {"user": user_id, "turns": [], "pinned": pinned}
def add_turn(conv_id, role, text):
store[conv_id]["turns"].append({"role": role, "text": text})
def build_messages(conv_id, keep_last=4):
convo = store[conv_id]
facts = "Known facts: " + "; ".join(convo["pinned"])
recent = convo["turns"][-keep_last:]
return [{"role": "system", "text": facts}] + recent
def words(messages):
return sum(len(m["text"].split()) for m in messages)
# A stand-in model, so this runs free. It reads only what it is sent.
def fake_model(messages):
return "Noted: " + messages[-1]["text"].split(".")[0].lower() + "."
new_conversation("c-101", "user-asha", ["name is Asha", "likes short answers"])
new_conversation("c-202", "user-ravi", ["name is Ravi", "building a cafe app"])
chat = [
"Hi, my name is Asha.",
"I need a logo idea for my bakery.",
"Something warm and simple.",
"Maybe orange and cream colours.",
"Please keep the font friendly and round.",
"Now suggest a name for the bakery.",
]
for text in chat:
add_turn("c-101", "user", text)
add_turn("c-101", "assistant", fake_model(build_messages("c-101")))
sent = build_messages("c-101")
print(f"turn {len(store['c-101']['turns']) // 2}: stored {words(store['c-101']['turns'])} words, sent {words(sent)} words")
add_turn("c-202", "user", "Hi, my name is Ravi.")
print("c-202 turns:", len(store["c-202"]["turns"]), "| c-101 turns:", len(store["c-101"]["turns"]))
for m in build_messages("c-101"):
print(m["role"] + ":", m["text"])Output
turn 1: stored 11 words, sent 19 words turn 2: stored 28 words, sent 36 words turn 3: stored 37 words, sent 34 words turn 4: stored 48 words, sent 28 words turn 5: stored 63 words, sent 34 words turn 6: stored 78 words, sent 38 words c-202 turns: 1 | c-101 turns: 12 system: Known facts: name is Asha; likes short answers user: Please keep the font friendly and round. assistant: Noted: please keep the font friendly and round. user: Now suggest a name for the bakery. assistant: Noted: now suggest a name for the bakery.
Look at the numbers. After the first turn, the chat has stored 11 words. After the sixth turn it has stored 78. That is the history growing every turn.
Now look at what was sent. From turn 3 on it stayed between 28 and 38 words, because we kept only the last four turns plus the pinned line. The request stopped growing, even though the chat kept going.
The last output shows the trade. The chat began with "Hi, my name is Asha." That message is long gone from what we send, yet the pinned line still says "name is Asha". Pinning saved the one fact we cannot lose.
The second user, user-ravi, has chat c-202 with its own single turn. Asha's chat holds 12 turns and is untouched. Two IDs kept the two chats apart.
What this sketch leaves out
This store lives in memory, so it vanishes when the program stops. A real app writes to a database. It also counts words, not tokens, and it never checks that the person asking owns the chat. Those checks belong in any real version.
Cutting at a fixed number of turns is blunt. A better rule counts tokens and drops the oldest turns until the request fits a budget. Also take care not to cut a tool call from its tool result, because the model needs both to make sense of the step.
Quick check
Your chat sends the last 4 turns only. The user said their name in turn 1. What keeps the name from being lost?
Key points to take away
Your app is the memory
The model is stateless. You store the chat and send it back, whether in the browser, your database or a provider's stored chat.
Store more than text
Keep messages, tool results, decisions and user settings, each under a conversation ID and a user ID.
Keep the request lean
Send the last few turns, a summary of older ones and a short list of pinned facts, not the whole history every time.
Next: memory systems, remembering across chats and days.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.