SyntaxPilot
Courses/Vibe Coding/Lesson 21
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Vibe Coding · Lesson 21 of 40

Project: AI study buddy with your own notes

retrieval in plain wordsembeddings (one-line idea)chunking documentsanswering only from your notesciting sourceswhen it still hallucinates

Vibe Coding: build real apps with AI · Episode 21. Every agent run below is real: Claude Code 2.1.322 (model claude-opus-5-5) in print mode (claude -p --permission-mode acceptEdits), on 12 November 2026 (India time), in /home/asha/projects/study-buddy (a Git repository). The agent could not run commands without approval, so it could not test its own code; we ran and tested everything.

The APIs used in this episode (both on the same computer, both OpenAI-compatible, both started with a made-up key demo-key-not-real; no real key is used anywhere):

  • Chat: llama-cpp-python 0.3.36 serving Qwen2.5-1.5B-Instruct (Apache-2.0, 1.54 billion parameters, Q4_K_M file), context window 4,096 tokens. Same as Episode 20.
  • Embeddings: a second llama-cpp-python server started with --embedding true, serving bge-small-en-v1.5 by BAAI (MIT licence; 384 numbers per text; the Q8_0 GGUF file published by CompendiumLabs on Hugging Face).
python3 -m llama_cpp.server --model qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --model_alias qwen2.5-1.5b-instruct --api_key demo-key-not-real --n_ctx 4096 --port 8080
python3 -m llama_cpp.server --model bge-small-en-v1.5-q8_0.gguf \
  --model_alias bge-small-en-v1.5 --embedding true --api_key demo-key-not-real --n_ctx 512 --port 8081

The notes: 11 short chapters written for this video in the style of a class-6 science notebook (food, components of food, fibre to fabric, separation of substances, nutrition in plants, soil, the water cycle, weather and climate, electricity, light and shadows, magnets): 16,670 characters, 3,662 tokens on Qwen's tokenizer. Every answer, score, token count and time below is real output. Facts about RAG and context windows quote the Claude Platform glossary, read on 12 November 2026.

The problem: the night before the exam

Asha has 11 chapters of notes and questions in her own words: "Why does the outside of a cold glass get wet?" Ctrl+F needs the exact word, and the exam checks what is in her notes, not what a website says. She wants an answer from her notes, and where it is written, so she can check.

Obvious way 1: ask the model directly (real)

No notes given, temperature 0:

Q: According to my class notes, how many years of weather make a climate?
A: "According to my class notes, the answer is 30 years."

Q: Which states grow cotton, according to my notes?
A: "I'm sorry, but I cannot provide the information you requested ..."

Her notes (Chapter 8) say: "Climate is the average weather pattern of a place over a long time, usually 25 years or more." The model never saw her notes, gave a number from its training and claimed it came from her notes. That is a hallucination: a confident answer that is not true to the source.

Obvious way 2: paste all the notes into every question (real)

Q: How many years of weather make a climate?   (all 11 chapters pasted before it)
A: "A climate is the average weather pattern of a place over a long time,
    usually 25 years or more."
usage: prompt_tokens 3692 · completion_tokens 23     time: 50.4 seconds

Correct, but 3,692 of the model's 4,096-token context window and 50 seconds for a one-line answer. Every question pays for the whole book, and one more chapter next term would not fit at all.

Why both fail: with no notes the model guesses; with all the notes it reads the whole book every time. We want to find the two or three paragraphs that answer the question and send only those.

Retrieval, generation, citing sources (an open-book exam)

Open-book exam Study buddy
Flip to the right page (don't reread the book) Retrieval: find the paragraphs that match the question
Write the answer in your words, from that page Generation: the LLM writes the answer from those paragraphs
Write "see page 74" so the teacher can check Citing sources: show which paragraph it used

Together this is RAG, retrieval augmented generation. Claude's glossary: "a technique that combines information retrieval with language model generation to improve the accuracy and relevance of the generated text, and to better ground the model's response in evidence", and "the effectiveness of RAG depends on the quality and relevance of the external knowledge base and the knowledge that is retrieved".

Chunking

A chunk is a small piece of a document. Cutting a document into chunks is chunking, like making flash cards from a textbook. Asha's notes split at blank lines give 61 chunks (paragraphs), each labelled with its chapter ("Chapter 7: The Water Cycle"); the label becomes the source we show.

  • Too big: wastes the context window and mixes several ideas.
  • Too small (half a sentence): loses its meaning.
  • One paragraph about one idea is a good start. (The AI's code also splits any paragraph longer than 1,200 characters at sentence ends.)

Why keyword search misses the answer (run in the compiler)

const chunks = [   // three real paragraphs from Asha's notes, shortened
  { id: "Ch 1 #4", text: "Animals can be grouped by what they eat. Herbivores eat " +
      "only plants or plant products. Carnivores eat other animals. Omnivores eat both." },
  { id: "Ch 2 #3", text: "Vitamins and minerals protect us from diseases. Vitamin C is " +
      "found in amla, lemons and oranges." },
  { id: "Ch 2 #5", text: "If a person does not get enough of a nutrient for a long time, " +
      "they can get a deficiency disease. Lack of vitamin C causes scurvy." },
];
const SMALL = new Set(["what", "if", "i", "don", "t", "for", "or", "the", "a", "of"]);
const words = (s) => s.toLowerCase().match(/[a-z]+/g).filter((w) => !SMALL.has(w));
// The obvious search: count the question's words in each paragraph
function keywordSearch(question) {
  const q = new Set(words(question));
  return chunks.map((c) => ({ id: c.id, hits: words(c.text).filter((w) => q.has(w)) }))
               .sort((a, b) => b.hits.length - a.hits.length);
}
const question = "What happens if I don't eat oranges or amla for months?";
for (const r of keywordSearch(question))
  console.log("  ", r.id, "->", r.hits.length, "matches", JSON.stringify(r.hits));
   Ch 1 #4 -> 4 matches ["eat","eat","eat","eat"]
   Ch 2 #3 -> 2 matches ["amla","oranges"]
   Ch 2 #5 -> 0 matches []

The winner is a paragraph about herbivores; the paragraph with the answer (scurvy) shares no words with the question. Words are not meaning.

Embeddings (one-line idea)

An embedding is a list of numbers that gives a piece of text an address on a map of meaning: texts about the same idea land close together, even with different words (like sweet shops in the same lane). To search, give the question an address too and pick the closest chunks.

Real request to the embeddings API:

curl http://localhost:8081/v1/embeddings -H "Authorization: Bearer $LLM_API_KEY" \
  -d '{"model": "bge-small-en-v1.5",
       "input": "Why does the outside of a cold glass get wet?"}'
"embedding": [-0.184, -0.075, 0.381, -0.108, 0.453, ... , -0.446, 1.171]   (384 numbers)
"usage": {"prompt_tokens": 13, "total_tokens": 13}

An embedding model is a different kind of model from a chat model: it returns numbers, not text. The numbers can't be read by people; only distances between them mean anything.

Cosine similarity (toy numbers, run in the compiler)

// Toy "meaning coordinates" with 3 numbers (real ones have 384)
// [about water and weather, about food and the body, about magnets]
const notes = {
  "condensation (Ch 7)": [0.9, 0.1, 0.0],
  "scurvy (Ch 2)":       [0.1, 0.9, 0.1],
  "compass (Ch 11)":     [0.1, 0.0, 0.9],
};
const question = [0.8, 0.2, 0.1];   // "Why does a cold glass get wet?"

// Cosine similarity: 1 = pointing the same way (same meaning), 0 = unrelated
function cosine(a, b) {
  const dot = a.reduce((s, x, i) => s + x * b[i], 0);
  const len = (v) => Math.sqrt(v.reduce((s, x) => s + x * x, 0));
  return dot / (len(a) * len(b));
}

for (const [name, vec] of Object.entries(notes))
  console.log(name.padEnd(20), cosine(question, vec).toFixed(3));
condensation (Ch 7)  0.984
scurvy (Ch 2)        0.357
compass (Ch 11)      0.226

The three numbers here are made up to show the idea; the real embedding model's 384 numbers are learned and have no simple labels.

The AI builds it (agent run 1)

claude -p "Build me a study buddy in this folder. My class notes are the .txt files in notes/ (11 chapters). I want a page where I type a question and get an answer that comes ONLY from my notes, with the source shown (which chapter), and if the answer is not in my notes it must say so instead of guessing. All the notes together are too long to send with every question (my model reads only 4096 tokens), so only send the parts that matter. I have two OpenAI-compatible APIs: chat (LLM_BASE_URL, LLM_MODEL) and embeddings (EMBED_BASE_URL, EMBED_MODEL), and the key is LLM_API_KEY; all are in .env. Keep it simple: plain HTML and JavaScript, a small Node server, no frameworks." --permission-mode acceptEdits
Write  server.js, public/index.html, public/app.js
Write  .gitignore   (.env, .cache/, node_modules/), README.md
Bash   node test.js                  → "requires approval"
Reply: "Your 11 chapters are split into about 60 paragraphs ...
   the server finds up to 5 paragraphs closest to it ...
   The 'close enough' cut-off (MIN_SCORE) is a guess. I set it
   to 0.6 but couldn't test it against your search model."

How it works (from its code):

  1. Split each chapter into paragraphs, each labelled with its chapter.
  2. Embed every paragraph once (with its chapter title in front) and save the numbers in .cache/embeddings.json, so restarts are fast.
  3. Retrieve: embed the question (with the instruction bge models expect in front of search queries), score every chunk with cosine similarity, keep up to 5 chunks scoring at least MIN_SCORE that fit in the context window. If none is close enough, answer "I couldn't find this in your notes" without asking the model.
  4. Generate with strict rules and numbered excerpts:
const SYSTEM_PROMPT = `You are a study helper for a student. Answer the question using ONLY the numbered excerpts from the student's class notes.
Rules:
1. Use only facts written in the excerpts. Never use outside knowledge, even if you know the answer.
2. End each sentence with the number of the excerpt it came from in square brackets, like [2].
3. If the excerpts do not contain the answer, reply with exactly: NOT IN NOTES
4. Keep the answer short and simple (1 to 4 sentences).`;
  1. Show the answer with [Ch 7] style citations, the chapter chips, and "Show the note text used" with the cited paragraphs marked. An answer with no valid citation is not shown.

It works (real)

  • "Why does the outside of a cold glass get wet?" → an answer citing Chapter 7; the cited paragraph: "...You see the same thing when drops form on the outside of a cold glass of water."
  • "How many years of weather make a climate?" → "25 years or more [Ch 8]" (Obvious way 1 said 30).

Questions her notes can't answer (real server log)

Indexed 61 chunks from 11 chapters.
Q: What is the capital of France?          top=0.451  → not asked
Q: Who won the cricket world cup in 2011?  top=0.471  → not asked
Q: Who discovered the water cycle?         top=0.675  → model: "NOT IN NOTES"

On topic is not the same as "has the answer": the water cycle paragraphs score high, but none says who discovered it, so the model correctly replies NOT IN NOTES.

AI mistake 1: a guessed cut-off hid a real answer

"What happens if I don't eat oranges or amla for months?" → "I couldn't find this in your notes." The best chunk (Chapter 2, scurvy) scored 0.586, just under the AI's guessed MIN_SCORE of 0.6. The right paragraph was found, then thrown away. (The AI had said the number was a guess.)

AI mistake 2: a source with no answer

"What was the shepherd's name in the magnet story?" → the whole answer was "[Ch 11]". The model replied only [1]; the code checked that a citation was present but never that an answer was.

The fix (agent run 2)

claude -p "I ran it with my real servers. Two bugs. 1) \"What happens if I don't eat oranges or amla for months?\" says \"I couldn't find this in your notes\", but Chapter 2 talks about vitamin C, amla and scurvy. The server log shows top=0.586, under your MIN_SCORE of 0.6. Real top scores I saw: cold glass question 0.744, climate question 0.788, magnet story 0.619, oranges and amla 0.586, but \"What is the capital of France?\" only 0.451. 2) \"What was the shepherd's name in the magnet story?\" shows just \"[Ch 11]\" as the answer. The model replied only \"[1]\", with no words, and your code accepted it because it had a citation. Fix both: pick a cut-off that fits these real scores, and never show an answer that is only citations (ask the model once more, telling it to write the answer in words; if it still fails, say not found). Change nothing else." --permission-mode acceptEdits

It set MIN_SCORE to 0.52 ("sits in that gap ... The gap is small, so it's worth checking a few more off-topic questions") and added hasWords(): a reply that is only citations is sent back once with a reminder. Our retest:

Q: What happens if I don't eat oranges or amla for months?   top=0.586
A: "[Ch 2] If you don't eat oranges or amla for months, you can get
    scurvy because vitamin C is essential for your health."
Q: What was the shepherd's name in the magnet story?
   1st reply: "[1]"   retry: "The shepherd's name in the magnet story was Magnes."
   (no citation)  → "I couldn't find this in your notes."

The retry was right (Magnes is in Chapter 11) but had no citation, so the app chose the safe side: no source, no answer.

The cut-off is a trade-off

Too high Too low
Real answers come back "not found" (amla at 0.6) Unrelated chunks reach the model, which has more chances to invent

Scores belong to one embedding model: measure them with your own in-notes and off-topic questions, then pick the cut-off in the gap (France 0.451 · cricket 0.471 · amla 0.586 · climate 0.788).

When it still makes things up (real)

  1. The right fact, the wrong source. "How do mushrooms get their food?" → "Mushrooms get their food through saprotrophic nutrition. [Ch 1]". The mushroom sentence is in the Chapter 5 paragraph; the model cited excerpt [2], a Chapter 1 paragraph about roti and kheer.
  2. A fact from outside her notes. "What does a compass needle point to and why?" → "A compass needle points to the north and south directions because it is influenced by Earth's magnetic field. [Ch 11]". True in science, but not in her notes: rule 1 broken, with a source attached.
  3. Not found, although it's there. The shepherd (no citation on a correct retry), and in our tests "Which crops grow well in loamy soil?" (the model replied "Wheat and gram grow well in loamy soil." without a citation, so it was hidden).

The fix (agent run 3): check the cited words

claude -p "Two answers looked right but were not fully from my notes. 1) \"How do mushrooms get their food?\" answered \"Mushrooms get their food through saprotrophic nutrition. [Ch 1]\" but the mushroom paragraph is in Chapter 5; the model cited excerpt [2], a Chapter 1 paragraph that never mentions mushrooms. 2) \"What does a compass needle point to and why?\" answered \"...because it is influenced by Earth's magnetic field. [Ch 11]\" but my notes never mention Earth's magnetic field. Add a simple check: for each sentence of the answer, look at the important words (skip small words like the, is, of) and check they appear in the excerpt it cites. If a sentence has important words that are not in its cited excerpt, still show the answer but add a warning under it: \"Check this: some words are not in the cited note\", and list those words. Change nothing else." --permission-mode acceptEdits

It added findUnsupportedWords() and an amber warning box, and said: "The check matches exact words, so the warning will also appear when the model correctly rewords the notes ... Treat it as a prompt to look at the note, not proof that the answer is wrong."

Real retest:

Question Warning shown Verdict
Mushrooms mushrooms, get, saprotrophic, nutrition Real catch: wrong source
Compass influenced, earth, field Real catch: outside fact
Cold glass vapor, excess, dew "dew" real; "vapor" false alarm (notes say "vapour")
Oranges and amla don, eat, months, essential, health Mostly words echoed from the question; "essential", "health" not in notes
Pinhole camera none Every word is in the note

Lessons: a source is not proof; always show the exact note text so the student can check in seconds; a word check tells you where to look; bigger models follow the rules much better than a 1.5-billion-parameter model, but never perfectly.

Project history:

48a42b6 Warn when answer words are not in the cited note (Claude Code, prompt 3)
ff9e96f Cut-off 0.52, retry citation-only answers (Claude Code, prompt 2)
2c396c0 Study buddy: chunks, embeddings, answers with sources (Claude Code, prompt 1)
a2070a6 Term 1 class notes (11 chapters)

.env and .cache/ are in .gitignore: no key and no generated numbers are committed.

The finished app (all files)

.env (never committed; the values are for the local servers in this video):

LLM_BASE_URL=http://localhost:8080/v1
LLM_MODEL=qwen2.5-1.5b-instruct
EMBED_BASE_URL=http://localhost:8081/v1
EMBED_MODEL=bge-small-en-v1.5
LLM_API_KEY=<your key>

.gitignore:

.env
.cache/
node_modules/

package.json:

{
  "name": "study-buddy",
  "version": "1.0.0",
  "private": true,
  "description": "Ask questions about your class notes; answers come only from the notes.",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "engines": {
    "node": ">=21.7"
  }
}

server.js (Node 21.7+, no dependencies; npm start):

// Study Buddy: answers questions using ONLY the class notes in notes/.
// Plain Node (>= 21.7), no dependencies. Run: node server.js
'use strict';

const http = require('node:http');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');

// ---------- config ----------
if (fs.existsSync(path.join(__dirname, '.env'))) process.loadEnvFile(path.join(__dirname, '.env'));

const env = (k, d) => (process.env[k] !== undefined && process.env[k] !== '' ? process.env[k] : d);
const CFG = {
  port: Number(env('PORT', 3000)),
  llmBase: env('LLM_BASE_URL', '').replace(/\/+$/, ''),
  llmModel: env('LLM_MODEL', ''),
  embedBase: env('EMBED_BASE_URL', '').replace(/\/+$/, ''),
  embedModel: env('EMBED_MODEL', ''),
  apiKey: env('LLM_API_KEY', ''),
  // bge models expect this instruction in front of search queries (not in front of passages).
  queryPrefix: env('EMBED_QUERY_PREFIX', 'Represent this sentence for searching relevant passages: '),
  // The chat model's whole window. Prompt + excerpts + answer must fit inside it.
  contextTokens: Number(env('MODEL_CONTEXT_TOKENS', 4096)),
  answerTokens: Number(env('ANSWER_MAX_TOKENS', 400)),
  topK: Number(env('TOP_K', 5)),
  // Excerpts scoring below this are treated as unrelated. If even the best one is
  // below it, we answer "not in your notes" without asking the model at all.
  // 0.52 sits in the gap seen with the real servers: covered questions scored 0.586-0.788,
  // "What is the capital of France?" scored 0.451.
  minScore: Number(env('MIN_SCORE', 0.52)),
};

for (const k of ['llmBase', 'llmModel', 'embedBase', 'embedModel']) {
  if (!CFG[k]) { console.error(`Missing config: ${k} (check .env)`); process.exit(1); }
}

const NOTES_DIR = path.join(__dirname, 'notes');
const PUBLIC_DIR = path.join(__dirname, 'public');
const CACHE_FILE = path.join(__dirname, '.cache', 'embeddings.json');
const NOT_FOUND = "I couldn't find this in your notes.";

// Rough token estimate. English averages ~4 chars/token; using 3 keeps us safely under the limit.
const estTokens = (s) => Math.ceil(s.length / 3);

// ---------- notes -> chunks ----------
function loadChunks() {
  const files = fs.readdirSync(NOTES_DIR).filter((f) => f.endsWith('.txt')).sort();
  const chunks = [];
  for (const file of files) {
    const text = fs.readFileSync(path.join(NOTES_DIR, file), 'utf8').replace(/\r\n/g, '\n');
    const m = text.match(/Chapter\s+(\d+)\s*:\s*(.+?[.?!])(?=\s|$)/);
    const num = m ? Number(m[1]) : Number((file.match(/\d+/) || [0])[0]);
    const title = m ? m[2].replace(/\.$/, '').trim() : file.replace(/\.txt$/, '');
    const chapter = `Chapter ${num}: ${title}`;

    const paras = text.split(/\n\s*\n/).map((p) => p.replace(/\s+/g, ' ').trim()).filter(Boolean);
    for (const p of paras) {
      for (const piece of splitLong(p, 1200)) {
        chunks.push({ id: chunks.length, file, chapter, chapterNum: num, text: piece });
      }
    }
  }
  return chunks;
}

// Split very long paragraphs on sentence boundaries so each chunk stays small.
function splitLong(p, max) {
  if (p.length <= max) return [p];
  const sentences = p.match(/[^.!?]+[.!?]+(\s|$)|[^.!?]+$/g) || [p];
  const out = [];
  let cur = '';
  for (const s of sentences) {
    if (cur && (cur + s).length > max) { out.push(cur.trim()); cur = ''; }
    cur += s;
  }
  if (cur.trim()) out.push(cur.trim());
  return out;
}

// ---------- API calls ----------
async function postJSON(url, body) {
  const headers = { 'Content-Type': 'application/json' };
  if (CFG.apiKey) headers.Authorization = `Bearer ${CFG.apiKey}`;
  const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
  const txt = await res.text();
  if (!res.ok) throw new Error(`${url} -> HTTP ${res.status}: ${txt.slice(0, 300)}`);
  return JSON.parse(txt);
}

async function embed(texts) {
  const out = [];
  for (let i = 0; i < texts.length; i += 16) {
    const batch = texts.slice(i, i + 16);
    const r = await postJSON(`${CFG.embedBase}/embeddings`, { model: CFG.embedModel, input: batch });
    const data = [...r.data].sort((a, b) => (a.index ?? 0) - (b.index ?? 0));
    for (const d of data) out.push(normalize(d.embedding));
  }
  return out;
}

async function chat(messages) {
  const r = await postJSON(`${CFG.llmBase}/chat/completions`, {
    model: CFG.llmModel,
    messages,
    temperature: 0,
    max_tokens: CFG.answerTokens,
  });
  return (r.choices?.[0]?.message?.content || '').trim();
}

function normalize(v) {
  const n = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
  return v.map((x) => x / n);
}
const dot = (a, b) => { let s = 0; for (let i = 0; i < a.length; i++) s += a[i] * b[i]; return s; };

// ---------- index (with on-disk cache so restarts are fast) ----------
// Each chunk is embedded together with its chapter name, so a paragraph that never
// says "magnet" still matches a question about magnets.
const passageText = (c) => `${c.chapter}. ${c.text}`;
const cacheKey = (t) => crypto.createHash('sha256').update(`${CFG.embedModel}\n${t}`).digest('hex');

async function buildIndex() {
  const chunks = loadChunks();
  let cache = {};
  try { cache = JSON.parse(fs.readFileSync(CACHE_FILE, 'utf8')); } catch {}
  const missing = chunks.filter((c) => !cache[cacheKey(passageText(c))]);
  if (missing.length) {
    console.log(`Embedding ${missing.length} of ${chunks.length} note chunks...`);
    const vecs = await embed(missing.map(passageText));
    missing.forEach((c, i) => { cache[cacheKey(passageText(c))] = vecs[i]; });
  }
  const fresh = {};
  for (const c of chunks) { c.vec = cache[cacheKey(passageText(c))]; fresh[cacheKey(passageText(c))] = c.vec; }
  fs.mkdirSync(path.dirname(CACHE_FILE), { recursive: true });
  fs.writeFileSync(CACHE_FILE, JSON.stringify(fresh));
  const chapters = new Set(chunks.map((c) => c.chapter));
  console.log(`Indexed ${chunks.length} chunks from ${chapters.size} chapters.`);
  return chunks;
}

// ---------- answering ----------
const SYSTEM_PROMPT = `You are a study helper for a student. Answer the question using ONLY the numbered excerpts from the student's class notes.
Rules:
1. Use only facts written in the excerpts. Never use outside knowledge, even if you know the answer.
2. End each sentence with the number of the excerpt it came from in square brackets, like [2].
3. If the excerpts do not contain the answer, reply with exactly: NOT IN NOTES
4. Keep the answer short and simple (1 to 4 sentences).`;

function buildUserMessage(question, excerpts) {
  const ex = excerpts.map((c, i) => `[${i + 1}] (${c.chapter})\n${c.text}`).join('\n\n');
  return `Excerpts from my notes:\n\n${ex}\n\nQuestion: ${question}\n\nAnswer using only the excerpts, with [number] citations, or reply NOT IN NOTES.`;
}

// Pick the most relevant chunks that also fit in the model's window.
function selectExcerpts(question, ranked) {
  const overhead = estTokens(SYSTEM_PROMPT) + estTokens(buildUserMessage(question, [])) + 50; // 50 = chat template
  let budget = CFG.contextTokens - CFG.answerTokens - overhead;
  const picked = [];
  for (const r of ranked) {
    if (picked.length >= CFG.topK || r.score < CFG.minScore) break;
    const cost = estTokens(`[${picked.length + 1}] (${r.chunk.chapter})\n${r.chunk.text}\n\n`);
    if (cost > budget) continue; // too big; a smaller, slightly lower-ranked chunk may still fit
    picked.push(r);
    budget -= cost;
  }
  return picked;
}

// The agreed refusal, plus common ways a small model says "the excerpts don't cover this"
// (only when the subject is the excerpts/notes, so "Cuscuta does not contain chlorophyll" is fine).
const NOT_FOUND_RE = /not\s+in\s+(the\s+|my\s+|your\s+)?notes|(excerpts?|notes|text|context|passages?)\s+(do(es)?\s+not|don't|doesn't)\s+(contain|mention|say|include|provide|talk)|no\s+information\s+(about|on)|not\s+mentioned\s+in/i;

// True if the reply has actual words once the [n] citations are removed.
const hasWords = (s) => /\p{L}/u.test(s.replace(/\[\d+\]/g, ''));

// ---------- word check: are a sentence's important words in the excerpt it cites? ----------
const STOP_WORDS = new Set(`a an the is am are was were be been being of to in on at by for with from into onto
through about as and or but nor not no so if than then that this these those it its it's they them their theirs there
here he him his she her hers we us our ours you your yours i me my mine who whom whose which what when where why how
do does did done has have had having will would shall should can could may might must also too very just only
some any each every all both either neither other such more most less much many one own same up down out off over
under again further once while because until after before during against between`.split(/\s+/));

// Same light ending-trim on both sides, so "mushrooms" matches "mushroom" and "pointing" matches "points".
const stem = (w) => (w.length > 4 ? w.replace(/(ing|ed|es|s)$/, '') : w);
const wordsOf = (s) => (s.toLowerCase().replace(/\[\d+\]/g, ' ').match(/[\p{L}\p{N}]+/gu) || []);

// Returns the important words of each sentence that don't appear in the excerpt(s) that
// sentence cites. A sentence with no citation of its own is checked against every excerpt
// the answer cites.
function findUnsupportedWords(raw, excerpts, allCited) {
  // Split into sentences; a citation left on its own after the full stop ("... north. [3]")
  // belongs to the sentence before it.
  const sentences = [];
  for (const part of raw.split(/(?<=[.!?])\s+/)) {
    if (sentences.length && !hasWords(part)) sentences[sentences.length - 1] += ' ' + part;
    else sentences.push(part);
  }
  const missing = new Set();
  for (const s of sentences) {
    let cites = [...s.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])).filter((n) => n >= 1 && n <= excerpts.length);
    if (!cites.length) cites = allCited;
    const known = new Set(cites.flatMap((n) => wordsOf(`${excerpts[n - 1].chapter} ${excerpts[n - 1].text}`)).map(stem));
    for (const w of wordsOf(s)) {
      if (STOP_WORDS.has(w) || w.length < 2) continue;
      if (!known.has(stem(w))) missing.add(w);
    }
  }
  return [...missing];
}

async function answer(question, chunks) {
  const [qv] = await embed([CFG.queryPrefix + question]);
  const ranked = chunks.map((chunk) => ({ chunk, score: dot(qv, chunk.vec) }))
    .sort((a, b) => b.score - a.score);
  const picked = selectExcerpts(question, ranked);
  const debug = { topScores: ranked.slice(0, 5).map((r) => ({ chapter: r.chunk.chapter, score: +r.score.toFixed(3) })) };

  if (!picked.length) return { found: false, answer: NOT_FOUND, sources: [], excerpts: [], debug };

  const excerpts = picked.map((r) => r.chunk);
  const messages = [
    { role: 'system', content: SYSTEM_PROMPT },
    { role: 'user', content: buildUserMessage(question, excerpts) },
  ];
  debug.promptTokensEst = messages.reduce((s, m) => s + estTokens(m.content), 0);
  let raw = await chat(messages);
  debug.raw = raw;

  // A reply like "[1]" cites an excerpt but says nothing. Ask once more for words.
  if (raw && !NOT_FOUND_RE.test(raw) && !hasWords(raw)) {
    raw = await chat([
      ...messages,
      { role: 'assistant', content: raw },
      { role: 'user', content: 'Your reply had only citation numbers and no words. Write the answer in words, in full sentences, using only the excerpts, and end each sentence with its [number] citation. If the excerpts do not contain the answer, reply with exactly: NOT IN NOTES' },
    ]);
    debug.retryRaw = raw;
  }

  const shownExcerpts = picked.map((r, i) => ({ n: i + 1, chapter: r.chunk.chapter, text: r.chunk.text, score: +r.score.toFixed(3) }));

  // The model must either refuse or cite the excerpts it used. An answer with no valid
  // citation can't be traced back to the notes, so we don't show it. Nor do we show an
  // answer that is only citations with no words.
  const cited = [...new Set([...raw.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))]
    .filter((n) => n >= 1 && n <= excerpts.length);
  if (!raw || NOT_FOUND_RE.test(raw) || !cited.length || !hasWords(raw)) {
    return { found: false, answer: NOT_FOUND, sources: [], excerpts: shownExcerpts, debug };
  }

  const sources = [...new Map(cited.map((n) => [excerpts[n - 1].chapter, {
    chapter: excerpts[n - 1].chapter, chapterNum: excerpts[n - 1].chapterNum, file: excerpts[n - 1].file,
  }])).values()].sort((a, b) => a.chapterNum - b.chapterNum);

  // Show citations as chapter numbers the student understands: [2] -> [Ch 7]
  const pretty = raw.replace(/\[(\d+)\]/g, (m, n) => {
    const c = excerpts[Number(n) - 1];
    return c ? `[Ch ${c.chapterNum}]` : '';
  }).replace(/(\[Ch \d+\])(\s*\1)+/g, '$1');

  // Still show the answer, but list words the cited notes don't contain so the student can check.
  const unsupportedWords = findUnsupportedWords(raw, excerpts, cited);

  return { found: true, answer: pretty, sources, excerpts: shownExcerpts, cited, unsupportedWords, debug };
}

// ---------- HTTP ----------
const TYPES = { '.html': 'text/html; charset=utf-8', '.js': 'text/javascript; charset=utf-8', '.css': 'text/css; charset=utf-8' };

function send(res, status, body, type = 'application/json; charset=utf-8') {
  res.writeHead(status, { 'Content-Type': type });
  res.end(typeof body === 'string' || Buffer.isBuffer(body) ? body : JSON.stringify(body));
}

async function main() {
  const chunks = await buildIndex();

  http.createServer(async (req, res) => {
    try {
      const url = new URL(req.url, 'http://x');
      if (req.method === 'POST' && url.pathname === '/api/ask') {
        let body = '';
        for await (const part of req) { body += part; if (body.length > 10000) return send(res, 413, { error: 'Too long' }); }
        const question = String(JSON.parse(body || '{}').question || '').trim().slice(0, 1000);
        if (!question) return send(res, 400, { error: 'Please type a question.' });
        const result = await answer(question, chunks);
        console.log(`Q: ${question}\n  found=${result.found} top=${result.debug.topScores[0]?.score} raw=${JSON.stringify(result.debug.raw || '')}`);
        return send(res, 200, result);
      }
      if (req.method === 'GET') {
        const p = url.pathname === '/' ? '/index.html' : url.pathname;
        const file = path.join(PUBLIC_DIR, path.normalize(p).replace(/^(\.\.[/\\])+/, ''));
        if (file.startsWith(PUBLIC_DIR) && fs.existsSync(file) && fs.statSync(file).isFile()) {
          return send(res, 200, fs.readFileSync(file), TYPES[path.extname(file)] || 'application/octet-stream');
        }
      }
      send(res, 404, { error: 'Not found' });
    } catch (err) {
      console.error(err);
      send(res, 500, { error: 'Something went wrong talking to the model servers. Are they running?' });
    }
  }).listen(CFG.port, () => console.log(`Study Buddy running at http://localhost:${CFG.port}`));
}

if (require.main === module) {
  main().catch((err) => { console.error('Startup failed:', err.message); process.exit(1); });
}

module.exports = { loadChunks, selectExcerpts, estTokens, findUnsupportedWords, CFG };

public/index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Study Buddy</title>
  <style>
    :root { --bg:#f7f6f2; --card:#fff; --ink:#1f2328; --muted:#6a6f76; --line:#e2e0d8; --accent:#2f6f4f; --warn:#9a5b00; --warnbg:#fff6e5; }
    @media (prefers-color-scheme: dark) {
      :root { --bg:#16181b; --card:#1f2226; --ink:#e8e6e1; --muted:#9aa0a6; --line:#33373c; --accent:#7cc39c; --warn:#f0b35a; --warnbg:#2b2416; }
    }
    * { box-sizing: border-box; }
    body { margin:0; background:var(--bg); color:var(--ink); font:16px/1.55 system-ui, sans-serif; }
    main { max-width: 720px; margin: 0 auto; padding: 32px 16px 64px; }
    h1 { margin:0 0 4px; font-size: 1.6rem; }
    .sub { color:var(--muted); margin:0 0 24px; }
    form { display:flex; gap:8px; }
    input { flex:1; min-width:0; padding:12px 14px; font:inherit; color:inherit; background:var(--card); border:1px solid var(--line); border-radius:10px; }
    input:focus { outline:2px solid var(--accent); outline-offset:1px; }
    button { padding:12px 18px; font:inherit; font-weight:600; color:#fff; background:var(--accent); border:0; border-radius:10px; cursor:pointer; }
    @media (prefers-color-scheme: dark) { button { color:#10261a; } }
    button:disabled { opacity:.6; cursor:wait; }
    .card { margin-top:20px; padding:18px 20px; background:var(--card); border:1px solid var(--line); border-radius:12px; }
    .q { color:var(--muted); font-size:.9rem; margin-bottom:8px; }
    .answer { white-space:pre-wrap; }
    .card.miss { background:var(--warnbg); border-color:var(--warn); }
    .card.miss .answer { color:var(--warn); font-weight:600; }
    .sources { margin-top:12px; display:flex; flex-wrap:wrap; gap:6px; align-items:center; font-size:.9rem; }
    .chip { padding:2px 10px; border:1px solid var(--accent); color:var(--accent); border-radius:999px; }
    details { margin-top:12px; font-size:.9rem; color:var(--muted); }
    details p { margin:8px 0; padding-left:10px; border-left:3px solid var(--line); }
    details p.used { border-left-color:var(--accent); color:var(--ink); }
    .err { color:#c0392b; }
    .check { margin-top:10px; padding:8px 12px; font-size:.9rem; color:var(--warn); background:var(--warnbg); border-left:3px solid var(--warn); border-radius:6px; }
  </style>
</head>
<body>
<main>
  <h1>Study Buddy</h1>
  <p class="sub">Ask anything about your class notes. Answers come only from your notes, with the chapter shown.</p>
  <form id="f">
    <input id="q" autocomplete="off" placeholder="e.g. What is photosynthesis?" required>
    <button id="b">Ask</button>
  </form>
  <div id="out"></div>
</main>
<script src="app.js"></script>
</body>
</html>

public/app.js:

const form = document.getElementById('f');
const input = document.getElementById('q');
const button = document.getElementById('b');
const out = document.getElementById('out');

function el(tag, cls, text) {
  const e = document.createElement(tag);
  if (cls) e.className = cls;
  if (text !== undefined) e.textContent = text;
  return e;
}

form.addEventListener('submit', async (e) => {
  e.preventDefault();
  const question = input.value.trim();
  if (!question) return;

  const card = el('div', 'card');
  card.append(el('div', 'q', question), el('div', 'answer', 'Thinking…'));
  out.prepend(card);
  button.disabled = true;

  try {
    const res = await fetch('/api/ask', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ question }),
    });
    const data = await res.json();
    if (!res.ok) throw new Error(data.error || 'Request failed');
    render(card, question, data);
    input.value = '';
  } catch (err) {
    card.querySelector('.answer').replaceWith(el('div', 'answer err', err.message));
  } finally {
    button.disabled = false;
    input.focus();
  }
});

function render(card, question, data) {
  card.textContent = '';
  card.classList.toggle('miss', !data.found);
  card.append(el('div', 'q', question), el('div', 'answer', data.answer));

  if (data.found && data.unsupportedWords && data.unsupportedWords.length) {
    const w = el('div', 'check', 'Check this: some words are not in the cited note: ');
    w.append(el('strong', '', data.unsupportedWords.join(', ')));
    card.append(w);
  }

  if (data.found && data.sources.length) {
    const s = el('div', 'sources', 'Source:');
    for (const src of data.sources) s.append(el('span', 'chip', src.chapter));
    card.append(s);
  }

  // Let the student check the exact note text the answer was based on.
  if (data.excerpts && data.excerpts.length) {
    const d = el('details');
    d.append(el('summary', '', data.found ? 'Show the note text used' : 'Closest parts of your notes'));
    const used = new Set(data.cited || []);
    for (const x of data.excerpts) {
      const p = el('p', used.has(x.n) ? 'used' : '');
      p.append(el('strong', '', x.chapter + ': '), document.createTextNode(x.text));
      d.append(p);
    }
    card.append(d);
  }
}

Common mistakes (real)

Mistake What you see Why Fix
Ask the model with no notes "According to my class notes, the answer is 30 years." It never saw the notes Retrieve, then answer
Paste every chapter 3,692 of 4,096 tokens, 50 s per question The whole book per question Send only the closest chunks
Keyword search "eat" ×4 wins; the answer has 0 hits Words aren't meaning Embeddings + cosine similarity
MIN_SCORE = 0.6 (a guess) Scurvy answer: "not found" Not measured Pick it from real scores
Accepting any citation Answer: "[Ch 11]" No check for words Require words; retry once
Trusting the citation Mushrooms → Chapter 1 The model cited wrong Show the note text; check words
Trusting "only from notes" "Earth's magnetic field" Small models break rules Word check; a stronger model

Mini project

Build a study buddy for your own notes (or a book chapter you are allowed to use):

  1. Ask 5 questions your notes answer and 5 they don't. Write down the top score of each and choose your cut-off.
  2. Try chunks of one sentence, one paragraph and one whole chapter. Which gives the best answers? Which the fastest?
  3. Find one answer with a wrong source or an outside fact. Did the word check catch it?
  4. Add a "Was this right?" button that saves the question, answer and sources to a file, so you can review mistakes later.

Practice questions

  1. Why did the model say "30 years" when Asha's notes say 25?
Answer

It was never given her notes. It answered from its training and wrongly claimed the answer came from her notes (a hallucination).

  1. Pasting all the notes gave the right answer. Name two problems with doing it for every question.
Answer

It used 3,692 of 4,096 tokens and took 50 seconds for one line (slow and, on a paid API, costly per question), and the notes will soon outgrow the context window.

  1. What are the three steps of RAG in the open-book exam picture?
Answer

Retrieve (flip to the right page), generate (write the answer from that page), cite (write which page).

  1. Why did keyword search pick the herbivores paragraph?
Answer

It contains "eat" four times; keyword search counts shared words. The scurvy paragraph shares no words with the question, though it means the same thing.

  1. In one line, what is an embedding?
Answer

A list of numbers that places a text on a map of meaning, so texts with similar meanings are close together.

  1. Two vectors point in exactly the same direction. What is their cosine similarity?
Answer

1.

  1. The top score for an in-notes question is 0.586 and the cut-off is 0.6. What happens, and what should you do?
Answer

The chunk is thrown away and the app says "not found". Measure real scores for in-notes and off-topic questions and set the cut-off in the gap (here 0.52).

  1. Why does the system prompt ask for an exact phrase like NOT IN NOTES?
Answer

So the code can recognise a refusal reliably and show a clear "not found" message.

  1. An answer has a citation. Can you trust it?
Answer

Not fully: in our tests the model cited the wrong paragraph (mushrooms) and added outside facts with a citation (compass). Show the note text and check.

  1. Why are the embeddings saved in .cache/ and listed in .gitignore?
Answer

Saving them avoids re-embedding all 61 chunks on every restart; they are generated data that can be rebuilt, so they don't belong in Git.

Quiz

Which one does an embedding help with?

A) Making the answers shorter
B) Finding "lack of vitamin C" for "no oranges"
C) Keeping the API key secret
D) Making the model obey NOT IN NOTES
Answer

B. An embedding turns text into numbers so that similar meanings are close together. It finds the scurvy paragraph even though it shares no words with the question. It doesn't change the answer's length, protect keys, or make the model obey rules.

Interview questions

  1. Explain RAG to a non-programmer, and say where it can still go wrong.
  2. How would you choose a chunk size? What are the trade-offs?
  3. Keyword (lexical) search vs embedding (semantic) search: when is each better? (Exact codes, names and numbers often favour keywords; paraphrases favour embeddings. Many systems combine both.)
  4. How would you pick and test a similarity cut-off?
  5. How would you measure whether a RAG app's answers are grounded in the sources?
  6. Why might an app embed the chapter title together with each paragraph?
  7. When the notes change, what has to be recomputed, and how does the AI's cache key handle it? (It hashes the model name and the chunk text, so only changed chunks are re-embedded.)

Glossary

Term Meaning (plain words) Example
Hallucination A confident answer that is not true to the source "According to my class notes, ... 30 years"
Context window How many tokens a model can look at at once 4,096 tokens
Retrieval Finding the parts of your documents that match a question The 5 closest paragraphs
Generation The LLM writing an answer from the given text "25 years or more [Ch 8]"
RAG (retrieval augmented generation) Retrieve the relevant text, then let the model answer from it The study buddy
Chunk A small piece of a document One paragraph
Chunking Cutting documents into chunks 11 chapters → 61 chunks
Keyword (lexical) search Matching the exact words of the question Ctrl+F
Embedding A list of numbers that places a text on a map of meaning 384 numbers
Embedding model A model that turns text into embeddings (not text) bge-small-en-v1.5
Vector A list of numbers [0.9, 0.1, 0.0]
Cosine similarity How closely two vectors point the same way: 1 = same, 0 = unrelated 0.984
Similarity score How close a chunk is to the question 0.586
Cut-off (threshold) The minimum score a chunk needs to be used MIN_SCORE = 0.52
Top-k Keep only the k best results Up to 5 chunks
Index The stored chunks and their embeddings, ready to search .cache/embeddings.json
Cache Saved results so they are not recomputed Embeddings reused after a restart
Excerpt A chunk given to the model, numbered [1] (Chapter 7: The Water Cycle)
Citation A pointer to the source of a sentence [Ch 7]
Grounded answer An answer supported by the given sources The pinhole camera answer
System prompt The rules for the conversation "Answer ... using ONLY the numbered excerpts"
NOT IN NOTES The exact refusal phrase the code checks for "Who discovered the water cycle?"
False negative Saying "not found" when the answer is there Amla at 0.6
False alarm A warning when nothing is wrong "vapor" vs "vapour"
.env A file of secret settings, never committed LLM_API_KEY=<your key>
.gitignore Files Git must never commit .env, .cache/
Accept edits An agent permission mode: it may write files without asking each time --permission-mode acceptEdits

Summary

  • Without the notes, the model guesses and may even claim to quote them. With all the notes, every question reads the whole book and soon won't fit.
  • RAG: chunk the notes, embed each chunk, embed the question, retrieve the closest chunks by cosine similarity, and let the model answer only from them, with citations and an exact "NOT IN NOTES" refusal.
  • Keyword search misses answers that use different words; embeddings compare meanings.
  • The AI's real mistakes: a guessed cut-off that hid a real answer, and accepting an answer that was only a citation. Measure real scores; require words.
  • RAG still goes wrong: wrong sources, outside facts, and "not found" when it's there. Show the exact note text and check the cited words.

Next episode: app builders like v0, Lovable and Bolt: a whole app from one prompt, and who owns the code.