esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Beginner / 01.21

Memory Systems

How an AI product can remember you across chats and days: what to save, where to keep it, how to find it again, and how to forget.

13 min readFree

Remembering across chats and days

The last lesson, State & Historical Context, ended with a promise: next, memory systems, remembering across chats and days. Here it is.

Picture a cafe where the barista greets you on every visit as if you had never walked in. Nice coffee, but you must say your name and your allergy again each time. Many AI chats work like that cafe. Close the chat, open a new one, and the model has no idea who you are.

A memory system fixes this. It is the part of your product that keeps some facts after a chat ends and brings them back later. By the end you will know the two kinds of memory, how a memory system works in four steps, where it can go wrong, and you will run a tiny one yourself.

Key words

Short-term memory
What the model can see right now: the messages in the current chat. It disappears when the chat ends.
Long-term memory
Facts and notes your product keeps outside the chat, so they are still there tomorrow.
Memory store
The place where long-term notes are kept, such as a file or a database table.
Retrieve
Look in the store and pull out the few notes that fit the current question.
Inject
Paste the retrieved notes into the prompt, so the model can see them.
Stale
Once true, but no longer true. Like a note that says "lives in Paris" after the person moved.

Short-term memory: the current chat

A model does not remember anything between requests. Each time you call it, it sees only what you send. In a chat, your product re-sends the whole conversation every turn, and that is what makes the model look like it remembers the last message.

That pile of messages is short-term memory. It lives in the context window, the amount of text a model can see at once. The Context Window lesson covers it in detail.

Think of a whiteboard in a meeting room. Everyone can read it while the meeting runs. When the meeting ends, someone wipes it clean. Anything you wanted to keep must be copied into a notebook before the wipe.

Short-term memory has two limits. It is small, because the window is limited. And it ends with the chat. The MemGPT paper states the first limit directly: large language models "are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis".

Long-term memory: the notebook

Long-term memory is the notebook from the whiteboard story. It is a store outside the chat. Your code writes notes into it, and your code reads notes back out when they are useful.

Notice who does the remembering. The model does not. Your product does. The model is like a very smart person who wakes up with no past each time, and you hand them a page of notes before they start.

You do not have to put the whole notebook in front of the model. That would fill the window fast. Instead you pick only the few notes that fit the question. The MemGPT paper uses this thought in its design. It draws on "hierarchical memory systems in traditional operating systems", where a small fast memory and a big slow one work together to give "the appearance of large memory resources through data movement between fast and slow memory".

In plain words: keep the small, fast space (the window) for what matters right now, keep the big, slow space (the store) for everything else, and move notes between them when needed. The paper describes its system as one that "intelligently manages different memory tiers" to provide extended context within a limited window.

What kinds of things can be remembered

Not all memories are the same. Two research papers give useful pictures.

The first is Cognitive Architectures for Language Agents, called CoALA. Its abstract says CoALA "describes a language agent with modular memory components". Modular means built from separate parts. So the paper's picture is not one big heap of notes, but several memories with different jobs, plus a structured set of actions for working with them.

The second is Generative Agents, a paper about software characters that live in a small simulated town. Its abstract describes an architecture that extends a language model "to store a complete record of the agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior".

That sentence names two kinds of memory. One is a record of what happened. The other is a reflection: a higher-level summary made from many records. From many records like "ate alone again" you might reflect: "this person likes quiet places".

For a product, it helps to sort notes into three piles. This grouping is our own plain-words choice. The first two piles echo the papers above.

  • Facts about the user

    Things that stay true for a while: a name, a language, a food allergy, a favourite seat.

  • Past events

    What happened and when: "asked for a refund on 3 November". The Generative Agents paper keeps such a record of experiences.

  • Reflections

    A short lesson drawn from many events: "this user prefers short answers". The same paper turns records into higher-level reflections.

The same abstract also says that the parts of its design, "observation, planning, and reflection", each "contribute critically" to how believable the agents act. You do not need all three piles on day one. Facts alone already make a product feel much friendlier.

How it works: four steps

Every memory system, small or large, does the same four jobs. Think of a good assistant who keeps a notebook.

  1. 1

    Write

    Decide what is worth saving. Most sentences are small talk. Saving all of them makes a messy notebook. A rule, or a model call, picks the few that matter.

  2. 2

    Store

    Put the note somewhere safe with a little extra: when it was saved and some tags (short labels) so you can find it later.

  3. 3

    Retrieve

    When a new question arrives, look through the store and pull out the best few notes.

  4. 4

    Inject

    Paste those notes into the prompt, usually in the system message, before the model answers.

Step three has two common styles. Keyword search looks for shared words: if the question and the note both contain "allergic", they match. It is simple, fast and easy to explain. But it misses notes that say the same thing in different words.

Search by meaning solves that. It turns each note and each question into a list of numbers called an embedding, and finds notes whose numbers sit close together. Then "allergy" can find a note about peanuts. The Embeddings lesson goes deeper, so here we use plain keyword search. The four steps stay the same either way.

A tiny memory store you can run

Here is a small memory system in plain Python. It follows the four steps. Read the comments as you go.

Write: a rule saves a sentence only if it starts with "Remember" or contains "my name is". Store: each note gets a number, a timestamp and tags (its useful words). Retrieve: each note scores one point for every question word it shares, and the best two win. Inject: the top notes are pasted into the system message. There is also a `forget` function that deletes a note by its number.

To keep results the same on every run, the timestamps are typed in by hand rather than read from the clock.

Python
import re

SKIP = {"the", "a", "an", "is", "am", "my", "i", "to", "of", "do", "what", "that", "have", "and", "me", "you", "remember"}
store = []
next_id = 1


def words_of(text):
    return [w for w in re.findall(r"[a-z']+", text.lower()) if w not in SKIP]


def worth_saving(sentence):
    s = sentence.lower().strip()
    return s.startswith("remember") or "my name is" in s


def save(sentence, when):
    global next_id
    if not worth_saving(sentence):
        print("  skip   :", sentence)
        return
    note = {"id": next_id, "text": sentence, "when": when, "tags": set(words_of(sentence))}
    store.append(note)
    next_id += 1
    print("  saved  :", note["id"], when, sorted(note["tags"]))


def retrieve(question, k=2):
    q = set(words_of(question))
    scored = [(len(q & note["tags"]), note) for note in store]
    scored = [(score, note) for score, note in scored if score > 0]
    scored.sort(key=lambda pair: (-pair[0], -pair[1]["id"]))
    return scored[:k]


def forget(note_id):
    global store
    store = [note for note in store if note["id"] != note_id]
    print("  forgot : memory", note_id)


def build_prompt(question, found):
    notes = "\n".join("- " + note["text"] for _, note in found)
    system = "You are a helpful cafe assistant."
    if notes:
        system += "\nThings you remember about this user:\n" + notes
    return [("system", system), ("user", question)]


def fake_model(messages):
    # Stand-in: it just repeats the first remembered line it was given.
    system = messages[0][1]
    lines = [l for l in system.split("\n") if l.startswith("- ")]
    return "From my notes: " + lines[0][2:] if lines else "I do not know that yet."


def ask(question):
    print("ASK:", question)
    found = retrieve(question)
    print("  found  :", [(score, note["id"]) for score, note in found])
    messages = build_prompt(question, found)
    print("  prompt :", messages[0][1].replace("\n", " | "))
    print("  answer :", fake_model(messages))


print("Day 1")
save("Hi there, good morning.", "2026-11-16 09:00")
save("My name is Asha.", "2026-11-16 09:01")
save("Remember that I am allergic to peanuts.", "2026-11-16 09:02")
save("Remember I like window seats.", "2026-11-16 09:03")
print("Day 2 (a brand new chat, the store is still there)")
ask("What is my name?")
ask("Do I have an allergy?")
ask("What am I allergic to?")
ask("Tell me my name and what I like.")
forget(1)
ask("What is my name?")

Output

Day 1
  skip   : Hi there, good morning.
  saved  : 1 2026-11-16 09:01 ['asha', 'name']
  saved  : 2 2026-11-16 09:02 ['allergic', 'peanuts']
  saved  : 3 2026-11-16 09:03 ['like', 'seats', 'window']
Day 2 (a brand new chat, the store is still there)
ASK: What is my name?
  found  : [(1, 1)]
  prompt : You are a helpful cafe assistant. | Things you remember about this user: | - My name is Asha.
  answer : From my notes: My name is Asha.
ASK: Do I have an allergy?
  found  : []
  prompt : You are a helpful cafe assistant.
  answer : I do not know that yet.
ASK: What am I allergic to?
  found  : [(1, 2)]
  prompt : You are a helpful cafe assistant. | Things you remember about this user: | - Remember that I am allergic to peanuts.
  answer : From my notes: Remember that I am allergic to peanuts.
ASK: Tell me my name and what I like.
  found  : [(1, 3), (1, 1)]
  prompt : You are a helpful cafe assistant. | Things you remember about this user: | - Remember I like window seats. | - My name is Asha.
  answer : From my notes: Remember I like window seats.
  forgot : memory 1
ASK: What is my name?
  found  : []
  prompt : You are a helpful cafe assistant.
  answer : I do not know that yet.

Important: `fake_model` is a stand-in so the code runs free, with no key and no internet. It is not a real model. It only repeats the first remembered line it was given. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse.

Now read the output. On Day 1, the greeting was skipped, because it does not start with "Remember" and has no "my name is". Three notes were saved, each with tags. On Day 2 there is a brand new chat, but the store is still there.

"What is my name?" matched one note, number 1, with a score of 1 (the word "name"). The prompt now holds that note, so the stand-in could answer. Without the note it could only say "I do not know that yet."

"Do I have an allergy?" found nothing. The note says "allergic", the question says "allergy". Different words, so no match. That is the weakness of keyword search. The next question, "What am I allergic to?", used the exact word and found note 2.

"Tell me my name and what I like." matched two notes, each with a score of 1. Both were injected, which is the top-2 rule at work. When scores tie, the code puts the newer note first, so note 3 came before note 1.

Last, `forget(1)` removed the name note. The same question afterwards got "I do not know that yet." Forgetting works because the note is gone from the store, not because anyone told the model to ignore it.

Quick check

Quick check

Where does the long-term memory in this lesson's code live?

What can go wrong

A notebook that never forgets and never checks itself causes trouble. Three risks matter most.

  • Wrong or sensitive notes

    A rule or a model can save a wrong fact, such as a joke taken seriously. It can also save something private, like a health detail or an address.

  • Stale notes

    People move, change jobs and change their minds. An old note can push the model to a confident but outdated answer.

  • No way to see or delete

    If users cannot see what is remembered, or cannot remove it, they lose trust in the product.

Here are habits that lower each risk. They are good design practice, not rules from the papers.

  1. 1

    Save less

    Be strict about what counts as worth saving. Our rule saved only sentences the user marked with "Remember" or that stated a name. Asking first, or saving only on request, keeps the user in charge.

  2. 2

    Skip sensitive data

    Do not save things like passwords, card numbers or health details unless you truly need them and know the rules where your users live.

  3. 3

    Date every note

    That is why our notes carry a timestamp. A date lets you show the newest note first, or drop notes that are too old.

  4. 4

    Let users see and forget

    Show the list of notes in a plain screen. Put a delete button on every line. Our `forget` function is the heart of that button.

Tip

A good test for any memory feature: could a user open a page, read everything the product remembers about them, and wipe it in one click? If not, it is not finished.

One more point: a retrieved note is only a hint. If the user says "I moved to Rome", the new message should win over an old note. Your prompt can say that: "If the user's message and a note disagree, trust the message."

One more check

Quick check

A note says "I live in Paris" and is two years old. What is the safest way to handle it?

What to remember

  • Two kinds of memory

    Short-term is the current chat in the context window. Long-term is a store your product keeps.

  • Four steps

    Write, store, retrieve, inject. Each is a small piece of code you can read and test.

  • Three risks

    Wrong or private notes, stale notes, and users who cannot see or delete what is kept.

You will meet memory again in the later lessons on context compaction and multi-agent context sharing. The idea holds: the model forgets, so your product has to remember with care.

Next: context compaction, shrinking a long history without losing the plot.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.