These are the notes for AI episode 30. Our agent can loop, call tools and plug into MCP servers, but open a new chat and it has forgotten everything. You'll see why (the model is stateless), why keeping every message overflows the context window, why a sliding window forgets the wrong things, and then invent summary memory (short-term), a memory store with remember() and recall() (long-term), and scoring memories by recency, importance and relevance. Every Python example runs as-is in the SyntaxPilot compiler; outputs and errors are real (Python 3.13). The models are toys; the 200,000-token limit and 30 tokens per message are example numbers.
Real-life example: Raju bhaiya's chai stall
| At the chai stall | In an AI agent |
|---|---|
| "Asha didi! Adrak chai, less sugar?" | the agent recalls a saved preference |
| A new stall: "What would you like?" | a new chat with an empty history |
| Raju keeps only what's worth keeping | save facts, not every word |
| He crosses out the old note when she switches to green tea | update stale memories |
Problem 1: a new chat forgets
def model(history): # toy LLM: answers from the history it gets
for m in history:
if "chai" in m:
return "Your usual: " + m.split(". ")[1]
return "Sorry, what is your usual?"
monday = ["I'm Asha. Adrak chai, less sugar", "My usual, please"]
print("Monday:", model(monday))
friday = ["My usual, please"] # a NEW chat starts with an empty history
print("Friday:", model(friday))
Output:
Monday: Your usual: Adrak chai, less sugar
Friday: Sorry, what is your usual?
The model is stateless (episode 23): it only knows the messages you send in this call. A new chat starts with an empty history.
Problem 2: keep everything forever?
LIMIT = 200_000 # our model's context window, in tokens
TOKENS_PER_MSG = 30
def call_model(history):
tokens = len(history) * TOKENS_PER_MSG
if tokens > LIMIT:
raise ValueError(f"prompt is too long: {tokens} tokens > {LIMIT}")
return tokens
history = []
for day in range(1, 366):
history += ["msg"] * 40 # 40 messages a day, ALL kept forever
if day in (1, 30, 100, 200):
print(f"day {day}: {call_model(history)} tokens in every call")
Output:
day 1: 1200 tokens in every call
day 30: 36000 tokens in every call
day 100: 120000 tokens in every call
Traceback (most recent call last):
File "/tmp/tmpxitursto/main.py", line 14, in <module>
print(f"day {day}: {call_model(history)} tokens in every call")
~~~~~~~~~~^^^^^^^^^
File "/tmp/tmpxitursto/main.py", line 7, in call_model
raise ValueError(f"prompt is too long: {tokens} tokens > {LIMIT}")
ValueError: prompt is too long: 240000 tokens > 200000
The context window is the most text a model can read at once: its short-term (working) memory. Keeping every message forever overflows it, and before that you pay for every token in every call, and responses get slower.
Problem 3: keep only the last few (sliding window)
chat = ["Hi, I'm Asha", "Hello Asha!", "Adrak chai, less sugar", "Done!",
"And one samosa", "Done!", "Is it raining?", "No idea, sorry!",
"What's my name?"]
def model(messages): # toy LLM: can only use what it SEES
for m in messages:
if "I'm " in m:
return "You're " + m.split("I'm ")[1].split(";")[0] + "!"
return "Sorry, I don't know your name."
WINDOW = 4 # keep only the last 4 messages
print("whole chat :", model(chat))
print("last 4 only:", model(chat[-WINDOW:]))
Output:
whole chat : You're Asha!
last 4 only: Sorry, I don't know your name.
A sliding window never overflows, but it forgets by age, not by importance: the name said first falls out first.
The fix for short-term memory: summary + window
chat = ["Hi, I'm Asha", "Hello Asha!", "Adrak chai, less sugar", "Done!",
"And one samosa", "Done!", "Is it raining?", "No idea, sorry!",
"What's my name?"]
def model(messages):
for m in messages:
if "I'm " in m:
return "You're " + m.split("I'm ")[1].split(";")[0] + "!"
return "Sorry, I don't know your name."
def summarise(old): # toy: a real app asks the LLM to do this
keep = [m for m in old if "I'm" in m or "chai" in m]
return "Summary: " + "; ".join(keep)
WINDOW = 4
old, recent = chat[:-WINDOW], chat[-WINDOW:]
prompt = [summarise(old)] + recent
for m in prompt: print(" ", m)
print("model:", model(prompt))
Output:
Summary: Hi, I'm Asha; Adrak chai, less sugar
Done!
Is it raining?
No idea, sorry!
What's my name?
model: You're Asha!
Old messages are replaced by a short summary; the last few stay word for word. A real app asks the LLM: "Summarise this conversation; keep names and preferences." The catch: a summary can drop details, so tell the summariser what matters.
The fix for long-term memory: a store outside the chat
memory = [] # saved OUTSIDE the chat (a file or DB)
def remember(fact): memory.append(fact)
def recall(question, k=1): # the facts sharing the most words
q = set(question.lower().strip("?").split())
score = lambda f: len(q & set(f.lower().split()))
return sorted(memory, key=score, reverse=True)[:k]
# --- session 1 (Monday) ---
remember("Asha likes adrak chai with less sugar")
remember("Asha sits at the window table")
# --- session 2 (Friday): a brand-new chat, empty history ---
print("recalled:", recall("Which chai does Asha like?"))
Output:
recalled: ['Asha likes adrak chai with less sugar']
Long-term memory lives outside the model (a file or a database), lasts between chats, and only the matching facts are put into the prompt. Real apps match by meaning using embeddings (episode 13), just like RAG (episode 24).
Problem 4: stale memories
memory = []
def remember(fact): memory.append(fact)
def recall(question, k=1):
q = set(question.lower().strip("?").split())
score = lambda f: len(q & set(f.lower().split()))
return sorted(memory, key=score, reverse=True)[:k]
remember("Asha likes adrak chai with less sugar") # March
remember("Asha sits at the window table")
remember("Asha stopped chai, now likes green tea") # October
print("recalled:", recall("Which chai does Asha like?"))
Output:
recalled: ['Asha likes adrak chai with less sugar']
The old fact shares more words with the question, so it wins. Matching alone never asks "is it still true?"
The fix: score recency + importance + relevance
memory = [] # each memory: text, hour saved, importance
def remember(text, hour, importance):
memory.append({"text": text, "hour": hour, "imp": importance})
def recall(question, now, k=1):
q = set(question.lower().strip("?").split())
def score(m):
relevance = len(q & set(m["text"].lower().split())) / len(q)
recency = 0.995 ** (now - m["hour"]) # fades every hour
importance = m["imp"] / 10 # rated 1 to 10
return relevance + recency + importance
for m in memory: print(f" {score(m):.2f} {m['text']}")
return max(memory, key=score)["text"]
remember("Asha likes adrak chai with less sugar", hour=0, importance=5)
remember("Asha sits at the window table", hour=10, importance=2)
remember("Asha stopped chai, now likes green tea", hour=5000, importance=8)
print("recalled:", recall("Which chai does Asha like?", now=5100))
Output:
0.90 Asha likes adrak chai with less sugar
0.40 Asha sits at the window table
1.61 Asha stopped chai, now likes green tea
recalled: Asha stopped chai, now likes green tea
| Part | Question it answers | In the Generative Agents paper (2023) |
|---|---|---|
| Relevance | does it match the question? | cosine similarity of embeddings |
| Recency | how fresh is it? | exponential decay, factor 0.995 per hour |
| Importance | how much does it matter? | the LLM rates each memory 1 (brushing teeth) to 10 (a breakup) |
The three scores are normalised and added (all weights 1); the top memories that fit the context window go into the prompt. You just invented agent memory.
Four kinds of agent memory (CoALA, 2023)
| Kind | Definition (CoALA) | Example |
|---|---|---|
| Working | "active and readily available information ... for the current decision cycle" | the context window right now |
| Episodic | "experience from earlier decision cycles" | "Asha's order was late on Friday" |
| Semantic | "an agent's knowledge about the world and itself" | "Asha likes green tea" |
| Procedural | "implicit knowledge stored in the LLM weights" and "explicit knowledge written in the agent's code" | how to take an order |
Who decides what to save?
- Memory as tools (episode 28): give the model
rememberandrecalltools; it callsrememberwhen the user says "I'm vegetarian". - Background saving: after each chat, the app (or an LLM call) extracts the facts worth keeping.
- Either way, save little: only what will matter next time.
User: "I'm vegetarian" ──► model calls remember("vegetarian") ──► memory store
Next chat: question ──► recall(question) ──► top facts ──► into the prompt
A remember tool
import json
memory = []
TOOLS = [{"name": "remember",
"description": "Save a lasting fact about the user.",
"input_schema": {"type": "object",
"properties": {"fact": {"type": "string"}},
"required": ["fact"]}}]
def model(message): # toy LLM: saves only lasting facts
if "vegetarian" in message:
return '{"name": "remember", "input": {"fact": "is vegetarian"}}'
return "Sure, one masala chai coming up!"
for msg in ["I'm vegetarian, by the way", "One masala chai please"]:
reply = model(msg)
if reply.startswith("{"): # a tool call: our code runs it
call = json.loads(reply)
memory.append(call["input"]["fact"])
reply = "Noted!"
print(f"{msg!r:30} -> {reply}")
print("memory:", memory)
Output:
"I'm vegetarian, by the way" -> Noted!
'One masala chai please' -> Sure, one masala chai coming up!
memory: ['is vegetarian']
The model chose what to save (a lasting fact, not a one-off order); our code did the saving, like any tool call.
Problem 5: whose memory is it?
memory = [] # ONE list for every customer
def remember(fact): memory.append(fact)
def recall(): return memory # toy: give back everything saved
remember("likes adrak chai, less sugar") # said by Asha
remember("allergic to peanuts") # said by Asha
# --- later, Rahul opens the app ---
print("Rahul's prompt includes:", recall())
Output:
Rahul's prompt includes: ['likes adrak chai, less sugar', 'allergic to peanuts']
One shared list means Asha's private facts (including an allergy) reach Rahul's prompt. Every memory needs an owner.
The fix: one notebook per user, and a real delete
memory = {} # one notebook PER user
def remember(user, fact): memory.setdefault(user, []).append(fact)
def recall(user): return memory.get(user, [])
def forget_all(user): memory.pop(user, None) # "delete my data"
remember("asha", "likes adrak chai, less sugar")
remember("asha", "allergic to peanuts")
remember("rahul", "likes cold coffee")
print("Rahul:", recall("rahul"))
print("Asha :", recall("asha"))
forget_all("asha")
print("Asha after delete:", recall("asha"))
Output:
Rahul: ['likes cold coffee']
Asha : ['likes adrak chai, less sugar', 'allergic to peanuts']
Asha after delete: []
Memory and privacy
- Never store passwords, OTPs or card numbers.
- Tell users what is remembered; let them see, edit and delete it.
- Keep each user's memory separate.
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Stateless | the model keeps nothing between calls | Friday's new chat |
| History | the list of messages you send each call | monday = [...] |
| Context window | the most tokens a model can read at once | 200,000 (example) |
| Token | a chunk of text the model reads (episode 12) | 30 per message (example) |
| Short-term memory | what's in the context window now | summary + last 4 |
| Sliding window | keep only the last N messages | chat[-4:] |
| Summary memory | old turns compressed into a short note | summarise(old) |
| Long-term memory | a store outside the chat that lasts between sessions | memory = [] |
| remember / recall | save a fact / fetch matching facts | recall("Which chai...") |
| Stale memory | a saved fact that is no longer true | "adrak chai" |
| Relevance | how well a memory matches the question | shared words / embeddings |
| Recency | how fresh a memory is | 0.995 ** hours |
| Importance | how much a memory matters, 1 to 10 | green tea switch = 8 |
| Working memory | active info for the current step | the prompt |
| Episodic memory | past experiences | "order was late on Friday" |
| Semantic memory | facts about the world and the user | "likes green tea" |
| Procedural memory | how to do things: weights, code, prompts | the agent's code |
| Embedding | a list of numbers capturing meaning (episode 13) | used for relevance |
| Memory tool | a remember tool the model calls to save a fact |
{"name": "remember", ...} |
| Memory owner | the one user a saved fact belongs to | memory["asha"] |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Keeping every message | ValueError: prompt is too long: 240000 tokens > 200000 |
the context window is limited | summary + last N |
| A bare sliding window | Sorry, I don't know your name. |
it forgets by age | keep a summary of what matters |
| Recall by match only | the stale "adrak chai" wins | old facts still match | score recency + importance + relevance, or update facts |
| Saving everything | slow, noisy recall | the store fills with small talk | save only useful facts |
| Storing secrets | privacy risk | memory is personal data | never store passwords, OTPs, card numbers |
| One memory list for all users | Rahul's prompt includes: ['likes adrak chai, less sugar', 'allergic to peanuts'] |
memories have no owner | one notebook per user; a real delete |
Mini project: a chai-stall memory
Build remember(text, hour, importance), recall(question, now, k=2) with the three-part score, and forget(word). Simulate three customers over a week, change one preference, and check that recall returns the new one.
Practice
1. Add forget() to the store, save five facts, and recall the top 2.
Solution
memory = []
def remember(fact): memory.append(fact)
def forget(word): # delete every fact that mentions word
global memory
memory = [f for f in memory if word.lower() not in f.lower()]
def recall(question, k=2):
q = set(question.lower().strip("?").split())
score = lambda f: len(q & set(f.lower().split()))
return sorted(memory, key=score, reverse=True)[:k]
for fact in ["Asha likes green tea", "Asha sits at the window table",
"Asha is allergic to peanuts", "Asha's order was late on Friday",
"Rahul likes cold coffee"]:
remember(fact)
print("top 2:", recall("What does Asha like to drink?"))
forget("late")
print("after forget:", len(memory), "facts")
Output:
top 2: ['Asha is allergic to peanuts', 'Asha likes green tea']
after forget: 4 facts
Notice the top result: "allergic to peanuts" matched the question's word "to". Counting shared words is crude; real apps compare meanings with embeddings.
2. Summarise a 12-message chat, keeping only the facts, plus the last 4 messages.
Solution
chat = [f"message {i}" for i in range(1, 13)]
chat[0] = "Hi, I'm Rahul"
chat[2] = "Cold coffee, no sugar"
def summarise(old): # toy summariser: keep the facts
keep = [m for m in old if "I'm" in m or "coffee" in m]
return "Summary: " + "; ".join(keep) + f" (+{len(old) - len(keep)} small talk)"
WINDOW = 4
prompt = [summarise(chat[:-WINDOW])] + chat[-WINDOW:]
for m in prompt: print(m)
Output:
Summary: Hi, I'm Rahul; Cold coffee, no sugar (+6 small talk)
message 9
message 10
message 11
message 12
3. Why should an allergy get importance 10?
Answer
Because forgetting it could harm the user. A high importance keeps it near the top even when it is old or only loosely matches the question.
Quick quiz 1
"Asha's order was 20 minutes late last Friday, and she was upset." Which kind of memory?
Answer
B: episodic memory: an experience from an earlier interaction.
Quick quiz 2
Monday: "I like green tea" (nothing saved). Friday, new chat: "My usual, please." What does the model know?
Answer
B: nothing. Chatting doesn't change the model's weights; without a memory store or the old history in the prompt, a new chat starts empty.
Interview questions
- Why do LLM apps "forget" between chats? The model is stateless; it only sees the messages in the current request.
- Short-term vs long-term memory in agents? Short-term is the context window (recent messages, summaries); long-term is an external store (database, vector store) recalled into the prompt when relevant.
- Why not send the whole history every time? It overflows the context window, costs tokens on every call and slows responses.
- Sliding window vs summary memory? A window keeps the last N messages and drops older ones by age; a summary compresses older turns so key facts survive.
- How do you rank memories? Combine relevance (embedding similarity), recency (decay over time) and importance (a 1-10 rating), as in Generative Agents.
- Name four kinds of agent memory. Working, episodic, semantic, procedural (CoALA).
- Privacy rules for memory? Don't store secrets, show users what's stored, let them edit and delete it, isolate users.
Summary
- A new chat starts empty: the model is stateless.
- Keeping everything overflows the context window; a sliding window forgets by age.
- Summary + last N = good short-term memory.
- A store outside the chat with remember/recall = long-term memory.
- Rank memories by recency + importance + relevance.
- Memory kinds: working, episodic, semantic, procedural.
- Memory is personal data: store little, let users see and delete it.
Next: episode 31, Planning and reflection: thinking before acting, and checking your own work.
Sources: Park et al., "Generative Agents: Interactive Simulacra of Human Behavior", arXiv 2304.03442 (2023); Sumers, Yao, Narasimhan, Griffiths, "Cognitive Architectures for Language Agents", arXiv 2309.02427 (2023).