SyntaxPilot
Courses/AI: Zero to Agents/Lesson 24

AI: Zero to Agents · Lesson 24 of 40 · 11 min

RAG: giving AI your documents

queued

These are the notes for AI episode 24. A model only knows what it saw in training, so it guesses about your shop, your company or your notes. RAG (retrieval-augmented generation) fixes that: retrieve the most relevant pieces of your documents, augment the prompt with them, and let the model generate an answer from them. You'll see why pasting everything doesn't scale, why keyword search misses synonyms, how embeddings search by meaning, how to chunk documents (size and overlap), how a threshold lets the bot say "I don't know", and real techniques: contextual retrieval, hybrid search (BM25 + embeddings) and reranking. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The embedding is a toy with four hand-made meaning groups; real embeddings are learned by a model (episode 13).

The problem: a confident guess

TRAINED_ON = {"refund": "Most shops accept refunds within 30 days.",
              "open": "Most shops open at 10 am."}     # learned from the web

def llm(question):            # toy LLM: only knows what it saw in training
    for topic, guess in TRAINED_ON.items():
        if topic in question.lower():
            return guess
    return "Yes, I think so!"

print("Q: What is Ramesh's refund policy?")
print("A:", llm("What is Ramesh's refund policy?"))
print("Shop manual says: refunds only within 2 days, with the bill.")

Output:

Q: What is Ramesh's refund policy?
A: Most shops accept refunds within 30 days.
Shop manual says: refunds only within 2 days, with the bill.

The model never saw Ramesh's manual (it isn't on the internet, and it was written after training), so it answers with what is typical. A confident made-up answer is a hallucination.

Real-life example: a new employee and a 1,000-page manual

What the employee does In AI
Guesses: "30 days, I think?" an LLM without your data: hallucination
Reads the whole manual for every question pasting every document into the prompt
Uses the index, reads only the refund page retrieval: find the relevant chunks
Answers from the page and shows "page 42" a grounded answer with a citation

Obvious fix 1: paste the whole manual

pages = 1000                    # manuals, recipes, staff rules, supplier deals
tokens_per_page = 500           # rough: one page of text
questions = 500                 # WhatsApp questions per day

manual = pages * tokens_per_page
print(f"whole manual        : {manual:>11,} tokens per question")
print(f"pasted every time   : {manual * questions:>11,} tokens per day")
relevant = 2 * 100              # 2 useful paragraphs, ~100 tokens each
print(f"only what's relevant: {relevant * questions:>11,} tokens per day")
print(f"that is {manual // relevant:,} times less")

Output:

whole manual        :     500,000 tokens per question
pasted every time   : 250,000,000 tokens per day
only what's relevant:     100,000 tokens per day
that is 2,500 times less

Pasting works for small documents. Anthropic's guide on contextual retrieval says that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can just include all of it in the prompt. Ramesh's 1,000 pages are bigger, and every question would pay for all of it.

CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
          "Timings: we open at 7 am and close at 10 pm.",
          "Menu: chai 15, samosa 20, vada pav 25.",
          "Delivery: we do not deliver, pick up only."]
def words(text):
    return set(text.lower().replace("?", " ").replace(",", " ")
               .replace(".", " ").replace(":", " ").split())

def keyword_score(question, chunk):      # how many words they share
    return len(words(question) & words(chunk))

question = "Can I get my money back?"
for chunk in CHUNKS:
    print(keyword_score(question, chunk), "|", chunk)

Output:

0 | Refunds: only within 2 days of purchase, with the bill.
0 | Timings: we open at 7 am and close at 10 pm.
0 | Menu: chai 15, samosa 20, vada pav 25.
0 | Delivery: we do not deliver, pick up only.

Why it failed: the customer said "money back", the manual says "refunds". Keyword search matches spelling, not meaning.

Search by meaning: embeddings

  • An embedding is a list of numbers that captures the meaning of a text (episode 13).
  • Texts with similar meaning get vectors that point in similar directions.
  • Cosine similarity compares directions: 1 = same direction (same meaning), 0 = unrelated.
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
          "menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text):         # toy embedding: one number per meaning group
    ws = re.findall(r"[a-z]+", text.lower())
    return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b):        # 1.0 = same direction, 0.0 = unrelated
    size = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
          "Timings: we open at 7 am and close at 10 pm.",
          "Menu: chai 15, samosa 20, vada pav 25.",
          "Delivery: we do not deliver, pick up only."]
question = "Can I get my money back?"
print("question:", embed(question))
for c in CHUNKS:
    print(f"{cosine(embed(question), embed(c)):.2f} | {embed(c)} | {c[:26]}")

Output:

question: [2, 0, 0, 0]
1.00 | [2, 0, 0, 0] | Refunds: only within 2 day
0.00 | [0, 5, 0, 0] | Timings: we open at 7 am a
0.00 | [0, 0, 5, 0] | Menu: chai 15, samosa 20, 
0.00 | [0, 0, 0, 3] | Delivery: we do not delive

This is semantic search. Real systems compute the embedding of every chunk ahead of time and store them in a vector database, built to find the closest vectors quickly, even among millions.

Retrieve, augment, generate

import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
          "menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text):         # toy embedding: one number per meaning group
    ws = re.findall(r"[a-z]+", text.lower())
    return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b):        # 1.0 = same direction, 0.0 = unrelated
    size = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
          "Timings: we open at 7 am and close at 10 pm.",
          "Menu: chai 15, samosa 20, vada pav 25.",
          "Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1):                   # R: the most similar chunks
    return sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
q = "Can I get my money back?"
print(f"Answer ONLY from the context. If it isn't there, say I don't know.\n"
      f"Context: {retrieve(q)}\nQuestion: {q}")

Output:

Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Refunds: only within 2 days of purchase, with the bill.']
Question: Can I get my money back?

This prompt is sent to the model with the API call from episode 23, and the model answers "only within 2 days, with the bill" because the answer is in front of it.

question ──► embed ──► find top-k chunks (retrieve)
                              │
                              ▼
   instruction + chunks + question (augment) ──► LLM ──► answer (generate)
Step What happens
Retrieve find the k chunks most similar to the question
Augment put them in the prompt as context, with "answer only from the context"
Generate the LLM writes the answer from that context

Why it helps:

  • Grounding: the answer is based on text you supplied, so the model reads instead of guessing.
  • Easy updates: change the document, not the model. No retraining.
  • Citations: you know which chunk was used, so you can show the source.

When nothing is relevant

import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
          "menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text):         # toy embedding: one number per meaning group
    ws = re.findall(r"[a-z]+", text.lower())
    return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b):        # 1.0 = same direction, 0.0 = unrelated
    size = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
          "Timings: we open at 7 am and close at 10 pm.",
          "Menu: chai 15, samosa 20, vada pav 25.",
          "Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1):                   # R: the most similar chunks
    return sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
q = "Do you sell pizza?"
print(f"Answer ONLY from the context. If it isn't there, say I don't know.\n"
      f"Context: {retrieve(q)}\nQuestion: {q}")

Output:

Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Refunds: only within 2 days of purchase, with the bill.']
Question: Do you sell pizza?

Top-k retrieval always returns something, even when every score is 0. Irrelevant context can confuse the model. Add a minimum score (a threshold):

import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
          "menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text):         # toy embedding: one number per meaning group
    ws = re.findall(r"[a-z]+", text.lower())
    return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b):        # 1.0 = same direction, 0.0 = unrelated
    size = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
          "Timings: we open at 7 am and close at 10 pm.",
          "Menu: chai 15, samosa 20, vada pav 25.",
          "Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1, min_score=0.3):    # only chunks that really match
    ranked = sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
    return [c for c in ranked if cosine(embed(q), embed(c)) >= min_score]
for q in ["Do you sell pizza?", "When do you close?"]:
    print(q, "->", retrieve(q) or "nothing found: say I don't know")

Output:

Do you sell pizza? -> nothing found: say I don't know
When do you close? -> ['Timings: we open at 7 am and close at 10 pm.']

An honest "I don't know" is better than a confident wrong answer.

Chunking: size and overlap

manual = ("Refunds are allowed only within 2 days of purchase. You must show "
          "the bill. We open at 7 am and close at 10 pm. We do not deliver.")
words = manual.split()
size, overlap = 10, 0                  # words per chunk, words shared

step = size - overlap
chunks = [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
for n, chunk in enumerate(chunks):
    print(n, "|", chunk)

Output:

0 | Refunds are allowed only within 2 days of purchase. You
1 | must show the bill. We open at 7 am and
2 | close at 10 pm. We do not deliver.

Chunk 0 ends with "You" and chunk 1 starts with "must show the bill": the rule was cut in half. Overlap repeats the last few words of the previous chunk:

manual = ("Refunds are allowed only within 2 days of purchase. You must show "
          "the bill. We open at 7 am and close at 10 pm. We do not deliver.")
words = manual.split()
size, overlap = 10, 4                  # words per chunk, words shared

step = size - overlap
chunks = [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
for n, chunk in enumerate(chunks):
    print(n, "|", chunk)

Output:

0 | Refunds are allowed only within 2 days of purchase. You
1 | days of purchase. You must show the bill. We open
2 | the bill. We open at 7 am and close at
3 | am and close at 10 pm. We do not deliver.
4 | We do not deliver.
Chunk size Problem
too small a chunk loses its meaning; ideas get split
too big you stuff in text that isn't needed; more tokens

Real systems often use chunks of a few hundred tokens and split on headings and paragraphs, not just word counts.

Lost context in a chunk

import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
          "menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text):         # toy embedding: one number per meaning group
    ws = re.findall(r"[a-z]+", text.lower())
    return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b):        # 1.0 = same direction, 0.0 = unrelated
    size = math.hypot(*a) * math.hypot(*b)
    return sum(x * y for x, y in zip(a, b)) / size if size else 0.0

chunk = "Section 4.2: it is allowed only within 2 days."
q = "Can I get my money back?"
print(f"plain chunk     : {cosine(embed(q), embed(chunk)):.2f}")
with_context = "Refund policy. " + chunk     # add where it came from
print(f"chunk + context : {cosine(embed(q), embed(with_context)):.2f}")

Output:

plain chunk     : 0.00
chunk + context : 1.00

The chunk says "it is allowed" but never says what: the heading "Refund policy" was left behind when the page was chopped. Putting a short note of context in front of the chunk before embedding it fixes the search.

Contextual retrieval (Anthropic, 19 September 2024)

Anthropic's example: a chunk saying "The company's revenue grew by 3% over the previous quarter" doesn't say which company or which quarter. Their fix is to have a model write a short context for each chunk before it is embedded. Reported results (failure to find the right chunk in the top 20):

Method Failed retrievals Reduction
plain embeddings 5.7% -
contextual embeddings 3.7% 35% fewer
+ contextual BM25 (hybrid search) 2.9% 49% fewer
+ reranking 1.9% 67% fewer
  • BM25 is a classic keyword-ranking method; it catches exact words such as names and codes that embeddings can miss.
  • Hybrid search combines keyword and meaning search.
  • Reranking: a second model re-scores the top candidates and keeps the best.
  • They also found that passing 20 chunks worked better than 10 or 5.

Where RAG came from

Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", arXiv, 22 May 2020 (NeurIPS 2020). They combined:

  • parametric memory: knowledge stored in a pre-trained model's weights (parameters), and
  • non-parametric memory: a dense vector index of Wikipedia, searched by a neural retriever.

They reported state-of-the-art results on three open-domain question-answering tasks and more specific, diverse and factual generation, and named the problems RAG addresses: updating a model's knowledge and showing where an answer came from (provenance).

Glossary

Term Meaning Example from this episode
RAG retrieval-augmented generation: retrieve, augment, generate the refund answer
Knowledge base the documents the system searches Ramesh's manual
Hallucination a confident, made-up answer "within 30 days"
Knowledge cutoff the date after which a model has no training data the manual was written later
Context stuffing pasting whole documents into the prompt 500,000 tokens a question
Chunk a small piece of a document "Refunds: only within 2 days..."
Chunk size how big each chunk is 10 words here; real: a few hundred tokens
Overlap words shared by neighbouring chunks overlap 4
Keyword search matching the same words 0 matches for "money back"
Embedding numbers that capture meaning [2, 0, 0, 0]
Cosine similarity how alike two vectors' directions are (0 to 1 here) 1.00 for the refund chunk
Semantic search search by meaning "money back" finds "Refunds"
Vector database stores embeddings and finds the closest ones fast holds every chunk's vector
Retrieval / retriever finding the most relevant chunks retrieve(q, k=1)
Top-k the k best-scoring chunks k = 1
Augment add the retrieved chunks to the prompt "Context: [...]"
Grounding basing the answer on supplied text "answer ONLY from the context"
Threshold a minimum score to count as relevant min_score=0.3
Citation showing which source an answer used "page 42"
Contextual retrieval adding a short context to each chunk before embedding "Refund policy. " + chunk
BM25 a classic keyword-ranking method catches exact names and codes
Hybrid search keyword + meaning search together BM25 + embeddings
Reranking a second model re-scores the top candidates 5.7% → 1.9% failures
Parametric memory what a model stores in its weights "most shops: 30 days"
Non-parametric memory an external index the model can search a vector index of Wikipedia

Common mistakes

Code / situation Exact error or symptom Why Fix
keyword search for "money back" every score 0: the refund chunk is missed same meaning, different words embeddings or hybrid search
retrieve("Do you sell pizza?") returns the refund rule top-k always returns something a minimum score; else "I don't know"
cosine without the size check, on an all-zero vector ZeroDivisionError: float division by zero an empty embedding has length 0 return 0.0 when the size is 0
chunks with no overlap a rule split across two chunks fixed-size cuts ignore sentences overlap; split on headings and paragraphs
a chunk without its heading score 0 for the right chunk the chunk lost its context add the context to each chunk

Mini project: a tiny RAG bot with citations

Give each chunk an id ("[1] Refunds: ..."). Write answer(q): retrieve the top 2 chunks above min_score=0.3; if none, return "Sorry, I don't know."; otherwise build the prompt from episode 24 and return it together with the ids of the chunks used ("Sources: [1]"). Test it on five questions, including one that isn't in the manual.

Practice

  1. Add a chunk "Parking: free parking behind the shop." and ask "Where can I park?". What happens? Fix it.
  2. Set k=2 and ask "When do you close, and is chai cheap?". Which two chunks come back?
  3. Try overlap 2 and 6. How many chunks do you get?
  4. Find a question that keyword search answers but our toy embedding doesn't.
Solutions

1. With the original four meaning groups, every score is 0: the toy embedding has no "parking" meaning. Add a fifth group "park parking car bike":

0.00 | Refunds: only within 2 days of purchase, with the bill.
0.00 | Timings: we open at 7 am and close at 10 pm.
0.00 | Menu: chai 15, samosa 20, vada pav 25.
0.00 | Delivery: we do not deliver, pick up only.
1.00 | Parking: free parking behind the shop.

Real embedding models learn meanings for almost every word, so this happens far less, but rare words, product codes and names are still a weak spot (that's what BM25 helps with).

2.

Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Timings: we open at 7 am and close at 10 pm.', 'Menu: chai 15, samosa 20, vada pav 25.']
Question: When do you close, and is chai cheap?

3. Overlap 2 gives 4 chunks; overlap 6 gives 7 chunks:

0 | Refunds are allowed only within 2 days of purchase. You
1 | purchase. You must show the bill. We open at 7
2 | at 7 am and close at 10 pm. We do
3 | We do not deliver.
0 | Refunds are allowed only within 2 days of purchase. You
1 | within 2 days of purchase. You must show the bill.
2 | purchase. You must show the bill. We open at 7
3 | the bill. We open at 7 am and close at
4 | at 7 am and close at 10 pm. We do
5 | close at 10 pm. We do not deliver.
6 | We do not deliver.

More overlap means more chunks (more storage and embedding work) but fewer ideas cut in half.

4. With the parking chunk added, ask "Is it free?". Keyword search scores the parking chunk 1 (the shared word "free"); the toy embedding scores every chunk 0.00, because "free" is in none of its meaning groups. This is why real systems combine both (hybrid search).

Quick quiz 1

Ramesh changes refunds from 2 days to 3. How does the RAG bot learn it? A: retrain the model · B: edit the document in the knowledge base · C: raise max_tokens · D: it can't

Answer

B. RAG reads the documents at question time; update the document and the next answer uses it.

Quick quiz 2

Question: "What time do you shut?" Chunk: "Timings: we close at 10 pm." Which search finds it? A: keyword search · B: meaning search with embeddings · C: neither · D: only the LLM, no search

Answer

B. "Shut" and "close" are different words with the same meaning.

Interview questions

  1. What is RAG and why use it instead of fine-tuning? Retrieve relevant documents at query time and put them in the prompt; knowledge is easy to update, answers are grounded and can cite sources, and no retraining is needed.
  2. Walk through a RAG pipeline. Split documents into chunks, embed and index them in a vector store; at query time embed the question, retrieve top-k (optionally hybrid + rerank), build a prompt with the context and instructions, generate, and return citations.
  3. How do you choose chunk size and overlap? Trade-off between context and precision; typically a few hundred tokens, split on structure (headings, paragraphs), with some overlap; evaluate retrieval quality to tune.
  4. What happens when nothing relevant is retrieved? Top-k still returns chunks; use a similarity threshold and instruct the model to say it doesn't know.
  5. What is hybrid search and why use it? Combining keyword ranking (e.g. BM25) with embeddings: embeddings catch synonyms, BM25 catches exact names, numbers and codes.
  6. What is reranking? A second, more accurate model re-scores the retrieved candidates so the best ones go into the prompt.
  7. When is RAG unnecessary? When the whole knowledge base fits comfortably in the prompt (Anthropic's guide: under about 200,000 tokens).

Summary

  • A model doesn't know your documents, so it guesses (hallucination).
  • Pasting everything is fine for small knowledge bases, but it doesn't scale.
  • Keyword search misses synonyms; embeddings + cosine similarity search by meaning.
  • RAG = retrieve the best chunks, augment the prompt, generate the answer.
  • A threshold lets the bot say "I don't know"; citations show the source.
  • Good chunks need sensible size, overlap and context; hybrid search and reranking find them more reliably.
  • Next episode: evaluating AI apps.