SyntaxPilot
Courses/AI: Zero to Agents/Lesson 30

AI: Zero to Agents · Lesson 30 of 40 · 11 min

Memory

queued

These are the notes for AI episode 30. Our agent can loop, call tools and plug into MCP servers, but open a new chat and it has forgotten everything. You'll see why (the model is stateless), why keeping every message overflows the context window, why a sliding window forgets the wrong things, and then invent summary memory (short-term), a memory store with remember() and recall() (long-term), and scoring memories by recency, importance and relevance. Every Python example runs as-is in the SyntaxPilot compiler; outputs and errors are real (Python 3.13). The models are toys; the 200,000-token limit and 30 tokens per message are example numbers.

Real-life example: Raju bhaiya's chai stall

At the chai stall In an AI agent
"Asha didi! Adrak chai, less sugar?" the agent recalls a saved preference
A new stall: "What would you like?" a new chat with an empty history
Raju keeps only what's worth keeping save facts, not every word
He crosses out the old note when she switches to green tea update stale memories

Problem 1: a new chat forgets

def model(history):              # toy LLM: answers from the history it gets
    for m in history:
        if "chai" in m:
            return "Your usual: " + m.split(". ")[1]
    return "Sorry, what is your usual?"

monday = ["I'm Asha. Adrak chai, less sugar", "My usual, please"]
print("Monday:", model(monday))

friday = ["My usual, please"]    # a NEW chat starts with an empty history
print("Friday:", model(friday))

Output:

Monday: Your usual: Adrak chai, less sugar
Friday: Sorry, what is your usual?

The model is stateless (episode 23): it only knows the messages you send in this call. A new chat starts with an empty history.

Problem 2: keep everything forever?

LIMIT = 200_000                  # our model's context window, in tokens
TOKENS_PER_MSG = 30

def call_model(history):
    tokens = len(history) * TOKENS_PER_MSG
    if tokens > LIMIT:
        raise ValueError(f"prompt is too long: {tokens} tokens > {LIMIT}")
    return tokens

history = []
for day in range(1, 366):
    history += ["msg"] * 40      # 40 messages a day, ALL kept forever
    if day in (1, 30, 100, 200):
        print(f"day {day}: {call_model(history)} tokens in every call")

Output:

day 1: 1200 tokens in every call
day 30: 36000 tokens in every call
day 100: 120000 tokens in every call
Traceback (most recent call last):
  File "/tmp/tmpxitursto/main.py", line 14, in <module>
    print(f"day {day}: {call_model(history)} tokens in every call")
                        ~~~~~~~~~~^^^^^^^^^
  File "/tmp/tmpxitursto/main.py", line 7, in call_model
    raise ValueError(f"prompt is too long: {tokens} tokens > {LIMIT}")
ValueError: prompt is too long: 240000 tokens > 200000

The context window is the most text a model can read at once: its short-term (working) memory. Keeping every message forever overflows it, and before that you pay for every token in every call, and responses get slower.

Problem 3: keep only the last few (sliding window)

chat = ["Hi, I'm Asha", "Hello Asha!", "Adrak chai, less sugar", "Done!",
        "And one samosa", "Done!", "Is it raining?", "No idea, sorry!",
        "What's my name?"]

def model(messages):             # toy LLM: can only use what it SEES
    for m in messages:
        if "I'm " in m:
            return "You're " + m.split("I'm ")[1].split(";")[0] + "!"
    return "Sorry, I don't know your name."

WINDOW = 4                       # keep only the last 4 messages
print("whole chat :", model(chat))
print("last 4 only:", model(chat[-WINDOW:]))

Output:

whole chat : You're Asha!
last 4 only: Sorry, I don't know your name.

A sliding window never overflows, but it forgets by age, not by importance: the name said first falls out first.

The fix for short-term memory: summary + window

chat = ["Hi, I'm Asha", "Hello Asha!", "Adrak chai, less sugar", "Done!",
        "And one samosa", "Done!", "Is it raining?", "No idea, sorry!",
        "What's my name?"]

def model(messages):
    for m in messages:
        if "I'm " in m:
            return "You're " + m.split("I'm ")[1].split(";")[0] + "!"
    return "Sorry, I don't know your name."

def summarise(old):              # toy: a real app asks the LLM to do this
    keep = [m for m in old if "I'm" in m or "chai" in m]
    return "Summary: " + "; ".join(keep)

WINDOW = 4
old, recent = chat[:-WINDOW], chat[-WINDOW:]
prompt = [summarise(old)] + recent
for m in prompt: print("  ", m)
print("model:", model(prompt))

Output:

   Summary: Hi, I'm Asha; Adrak chai, less sugar
   Done!
   Is it raining?
   No idea, sorry!
   What's my name?
model: You're Asha!

Old messages are replaced by a short summary; the last few stay word for word. A real app asks the LLM: "Summarise this conversation; keep names and preferences." The catch: a summary can drop details, so tell the summariser what matters.

The fix for long-term memory: a store outside the chat

memory = []                      # saved OUTSIDE the chat (a file or DB)

def remember(fact): memory.append(fact)

def recall(question, k=1):       # the facts sharing the most words
    q = set(question.lower().strip("?").split())
    score = lambda f: len(q & set(f.lower().split()))
    return sorted(memory, key=score, reverse=True)[:k]

# --- session 1 (Monday) ---
remember("Asha likes adrak chai with less sugar")
remember("Asha sits at the window table")

# --- session 2 (Friday): a brand-new chat, empty history ---
print("recalled:", recall("Which chai does Asha like?"))

Output:

recalled: ['Asha likes adrak chai with less sugar']

Long-term memory lives outside the model (a file or a database), lasts between chats, and only the matching facts are put into the prompt. Real apps match by meaning using embeddings (episode 13), just like RAG (episode 24).

Problem 4: stale memories

memory = []
def remember(fact): memory.append(fact)
def recall(question, k=1):
    q = set(question.lower().strip("?").split())
    score = lambda f: len(q & set(f.lower().split()))
    return sorted(memory, key=score, reverse=True)[:k]

remember("Asha likes adrak chai with less sugar")      # March
remember("Asha sits at the window table")
remember("Asha stopped chai, now likes green tea")      # October

print("recalled:", recall("Which chai does Asha like?"))

Output:

recalled: ['Asha likes adrak chai with less sugar']

The old fact shares more words with the question, so it wins. Matching alone never asks "is it still true?"

The fix: score recency + importance + relevance

memory = []                      # each memory: text, hour saved, importance
def remember(text, hour, importance):
    memory.append({"text": text, "hour": hour, "imp": importance})

def recall(question, now, k=1):
    q = set(question.lower().strip("?").split())
    def score(m):
        relevance  = len(q & set(m["text"].lower().split())) / len(q)
        recency    = 0.995 ** (now - m["hour"])     # fades every hour
        importance = m["imp"] / 10                  # rated 1 to 10
        return relevance + recency + importance
    for m in memory: print(f"  {score(m):.2f}  {m['text']}")
    return max(memory, key=score)["text"]

remember("Asha likes adrak chai with less sugar", hour=0, importance=5)
remember("Asha sits at the window table", hour=10, importance=2)
remember("Asha stopped chai, now likes green tea", hour=5000, importance=8)
print("recalled:", recall("Which chai does Asha like?", now=5100))

Output:

  0.90  Asha likes adrak chai with less sugar
  0.40  Asha sits at the window table
  1.61  Asha stopped chai, now likes green tea
recalled: Asha stopped chai, now likes green tea
Part Question it answers In the Generative Agents paper (2023)
Relevance does it match the question? cosine similarity of embeddings
Recency how fresh is it? exponential decay, factor 0.995 per hour
Importance how much does it matter? the LLM rates each memory 1 (brushing teeth) to 10 (a breakup)

The three scores are normalised and added (all weights 1); the top memories that fit the context window go into the prompt. You just invented agent memory.

Four kinds of agent memory (CoALA, 2023)

Kind Definition (CoALA) Example
Working "active and readily available information ... for the current decision cycle" the context window right now
Episodic "experience from earlier decision cycles" "Asha's order was late on Friday"
Semantic "an agent's knowledge about the world and itself" "Asha likes green tea"
Procedural "implicit knowledge stored in the LLM weights" and "explicit knowledge written in the agent's code" how to take an order

Who decides what to save?

  • Memory as tools (episode 28): give the model remember and recall tools; it calls remember when the user says "I'm vegetarian".
  • Background saving: after each chat, the app (or an LLM call) extracts the facts worth keeping.
  • Either way, save little: only what will matter next time.
 User: "I'm vegetarian" ──► model calls remember("vegetarian") ──► memory store
 Next chat: question ──► recall(question) ──► top facts ──► into the prompt

A remember tool

import json
memory = []
TOOLS = [{"name": "remember",
          "description": "Save a lasting fact about the user.",
          "input_schema": {"type": "object",
                           "properties": {"fact": {"type": "string"}},
                           "required": ["fact"]}}]

def model(message):              # toy LLM: saves only lasting facts
    if "vegetarian" in message:
        return '{"name": "remember", "input": {"fact": "is vegetarian"}}'
    return "Sure, one masala chai coming up!"

for msg in ["I'm vegetarian, by the way", "One masala chai please"]:
    reply = model(msg)
    if reply.startswith("{"):    # a tool call: our code runs it
        call = json.loads(reply)
        memory.append(call["input"]["fact"])
        reply = "Noted!"
    print(f"{msg!r:30} -> {reply}")
print("memory:", memory)

Output:

"I'm vegetarian, by the way"   -> Noted!
'One masala chai please'       -> Sure, one masala chai coming up!
memory: ['is vegetarian']

The model chose what to save (a lasting fact, not a one-off order); our code did the saving, like any tool call.

Problem 5: whose memory is it?

memory = []                      # ONE list for every customer

def remember(fact): memory.append(fact)
def recall(): return memory      # toy: give back everything saved

remember("likes adrak chai, less sugar")      # said by Asha
remember("allergic to peanuts")               # said by Asha

# --- later, Rahul opens the app ---
print("Rahul's prompt includes:", recall())

Output:

Rahul's prompt includes: ['likes adrak chai, less sugar', 'allergic to peanuts']

One shared list means Asha's private facts (including an allergy) reach Rahul's prompt. Every memory needs an owner.

The fix: one notebook per user, and a real delete

memory = {}                      # one notebook PER user

def remember(user, fact): memory.setdefault(user, []).append(fact)
def recall(user): return memory.get(user, [])
def forget_all(user): memory.pop(user, None)   # "delete my data"

remember("asha", "likes adrak chai, less sugar")
remember("asha", "allergic to peanuts")
remember("rahul", "likes cold coffee")

print("Rahul:", recall("rahul"))
print("Asha :", recall("asha"))
forget_all("asha")
print("Asha after delete:", recall("asha"))

Output:

Rahul: ['likes cold coffee']
Asha : ['likes adrak chai, less sugar', 'allergic to peanuts']
Asha after delete: []

Memory and privacy

  • Never store passwords, OTPs or card numbers.
  • Tell users what is remembered; let them see, edit and delete it.
  • Keep each user's memory separate.

Glossary

Term Meaning Example from this episode
Stateless the model keeps nothing between calls Friday's new chat
History the list of messages you send each call monday = [...]
Context window the most tokens a model can read at once 200,000 (example)
Token a chunk of text the model reads (episode 12) 30 per message (example)
Short-term memory what's in the context window now summary + last 4
Sliding window keep only the last N messages chat[-4:]
Summary memory old turns compressed into a short note summarise(old)
Long-term memory a store outside the chat that lasts between sessions memory = []
remember / recall save a fact / fetch matching facts recall("Which chai...")
Stale memory a saved fact that is no longer true "adrak chai"
Relevance how well a memory matches the question shared words / embeddings
Recency how fresh a memory is 0.995 ** hours
Importance how much a memory matters, 1 to 10 green tea switch = 8
Working memory active info for the current step the prompt
Episodic memory past experiences "order was late on Friday"
Semantic memory facts about the world and the user "likes green tea"
Procedural memory how to do things: weights, code, prompts the agent's code
Embedding a list of numbers capturing meaning (episode 13) used for relevance
Memory tool a remember tool the model calls to save a fact {"name": "remember", ...}
Memory owner the one user a saved fact belongs to memory["asha"]

Common mistakes

Mistake What you see Why Fix
Keeping every message ValueError: prompt is too long: 240000 tokens > 200000 the context window is limited summary + last N
A bare sliding window Sorry, I don't know your name. it forgets by age keep a summary of what matters
Recall by match only the stale "adrak chai" wins old facts still match score recency + importance + relevance, or update facts
Saving everything slow, noisy recall the store fills with small talk save only useful facts
Storing secrets privacy risk memory is personal data never store passwords, OTPs, card numbers
One memory list for all users Rahul's prompt includes: ['likes adrak chai, less sugar', 'allergic to peanuts'] memories have no owner one notebook per user; a real delete

Mini project: a chai-stall memory

Build remember(text, hour, importance), recall(question, now, k=2) with the three-part score, and forget(word). Simulate three customers over a week, change one preference, and check that recall returns the new one.

Practice

1. Add forget() to the store, save five facts, and recall the top 2.

Solution
memory = []
def remember(fact): memory.append(fact)
def forget(word):                # delete every fact that mentions word
    global memory
    memory = [f for f in memory if word.lower() not in f.lower()]
def recall(question, k=2):
    q = set(question.lower().strip("?").split())
    score = lambda f: len(q & set(f.lower().split()))
    return sorted(memory, key=score, reverse=True)[:k]

for fact in ["Asha likes green tea", "Asha sits at the window table",
             "Asha is allergic to peanuts", "Asha's order was late on Friday",
             "Rahul likes cold coffee"]:
    remember(fact)
print("top 2:", recall("What does Asha like to drink?"))
forget("late")
print("after forget:", len(memory), "facts")

Output:

top 2: ['Asha is allergic to peanuts', 'Asha likes green tea']
after forget: 4 facts

Notice the top result: "allergic to peanuts" matched the question's word "to". Counting shared words is crude; real apps compare meanings with embeddings.

2. Summarise a 12-message chat, keeping only the facts, plus the last 4 messages.

Solution
chat = [f"message {i}" for i in range(1, 13)]
chat[0] = "Hi, I'm Rahul"
chat[2] = "Cold coffee, no sugar"

def summarise(old):              # toy summariser: keep the facts
    keep = [m for m in old if "I'm" in m or "coffee" in m]
    return "Summary: " + "; ".join(keep) + f" (+{len(old) - len(keep)} small talk)"

WINDOW = 4
prompt = [summarise(chat[:-WINDOW])] + chat[-WINDOW:]
for m in prompt: print(m)

Output:

Summary: Hi, I'm Rahul; Cold coffee, no sugar (+6 small talk)
message 9
message 10
message 11
message 12

3. Why should an allergy get importance 10?

Answer

Because forgetting it could harm the user. A high importance keeps it near the top even when it is old or only loosely matches the question.

Quick quiz 1

"Asha's order was 20 minutes late last Friday, and she was upset." Which kind of memory?

Answer

B: episodic memory: an experience from an earlier interaction.

Quick quiz 2

Monday: "I like green tea" (nothing saved). Friday, new chat: "My usual, please." What does the model know?

Answer

B: nothing. Chatting doesn't change the model's weights; without a memory store or the old history in the prompt, a new chat starts empty.

Interview questions

  1. Why do LLM apps "forget" between chats? The model is stateless; it only sees the messages in the current request.
  2. Short-term vs long-term memory in agents? Short-term is the context window (recent messages, summaries); long-term is an external store (database, vector store) recalled into the prompt when relevant.
  3. Why not send the whole history every time? It overflows the context window, costs tokens on every call and slows responses.
  4. Sliding window vs summary memory? A window keeps the last N messages and drops older ones by age; a summary compresses older turns so key facts survive.
  5. How do you rank memories? Combine relevance (embedding similarity), recency (decay over time) and importance (a 1-10 rating), as in Generative Agents.
  6. Name four kinds of agent memory. Working, episodic, semantic, procedural (CoALA).
  7. Privacy rules for memory? Don't store secrets, show users what's stored, let them edit and delete it, isolate users.

Summary

  • A new chat starts empty: the model is stateless.
  • Keeping everything overflows the context window; a sliding window forgets by age.
  • Summary + last N = good short-term memory.
  • A store outside the chat with remember/recall = long-term memory.
  • Rank memories by recency + importance + relevance.
  • Memory kinds: working, episodic, semantic, procedural.
  • Memory is personal data: store little, let users see and delete it.

Next: episode 31, Planning and reflection: thinking before acting, and checking your own work.

Sources: Park et al., "Generative Agents: Interactive Simulacra of Human Behavior", arXiv 2304.03442 (2023); Sumers, Yao, Narasimhan, Griffiths, "Cognitive Architectures for Language Agents", arXiv 2309.02427 (2023).