SyntaxPilot
Courses/Vibe Coding/Lesson 20
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Vibe Coding · Lesson 20 of 40

Project: an AI-powered app (call an LLM API)

LLM API requestmodelprompttemperature in plain wordsstreamingcosts per tokenkeeping the key on the servera summarizer apphandling API errors and limits

Vibe Coding: build real apps with AI · Episode 20. Every agent run below is real: Claude Code 2.1.322 (model claude-opus-5-5, from the run's init line) in print mode (claude -p --permission-mode acceptEdits), on the morning of 12 November 2026 (India time), in /home/asha/projects/summarizer (a Git repository). The agent was not allowed to run commands without approval, so it could not test its own code; we ran and tested everything.

The LLM API used in this episode. The model runs on the same computer: llama-cpp-python 0.3.36, which "offers an OpenAI API compatible web server", serving the open model Qwen2.5-1.5B-Instruct (Qwen team, Alibaba Cloud; Apache-2.0 licence; 1.54 billion parameters; the 4-bit Q4_K_M file from Hugging Face). It was started with a context size of 4,096 tokens and a made-up API key, demo-key-not-real:

pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --model_alias qwen2.5-1.5b-instruct --api_key demo-key-not-real --n_ctx 4096 --port 8080

Every request, response, error message, token count and timing in these notes is real output from that server. The same code works with a cloud provider that accepts the OpenAI-style format (for example the Claude API, see the end): you change the base URL, the model name and the key. Facts about Claude (prices, error codes, temperature rules, tokens) come from the Claude Platform docs, read on 12 November 2026. No real key is used or shown anywhere.

The problem: ten minutes before class

Asha's teacher's notes for Chapter 7, the water cycle, are 2,089 characters (372 words). Class is in ten minutes and she wants the 5 most important points: a summary.

The notes (real, written for this video):

Good morning, class. Before we start, a reminder: the science test is on Friday, and please bring your lab notebooks tomorrow. Also, the library will be closed on Wednesday afternoon. Okay, let's begin today's chapter.

Chapter 7: The Water Cycle. The water on Earth is always moving. ...
The first step is evaporation. ... The second step is condensation. ...
The third step is precipitation. ... The fourth step is collection. ...
Homework: draw the water cycle with all four steps and label each one. ...

The obvious way: "a summary is the first few sentences"

Run in the compiler (JavaScript):

const notes = `Good morning, class. Before we start, a reminder: the science
test is on Friday, and please bring your lab notebooks tomorrow. Also, the library
will be closed on Wednesday afternoon. Okay, let's begin today's chapter.
Chapter 7: The Water Cycle. The water on Earth is always moving.`;

// Try 1: "a summary is the first few sentences"
const sentences = notes.replace(/\n/g, " ").match(/[^.!?]+[.!?]/g).map((s) => s.trim());
console.log("Summary (first 3 sentences):");
sentences.slice(0, 3).forEach((s) => console.log(" -", s));

Real output:

Summary (first 3 sentences):
 - Good morning, class.
 - Before we start, a reminder: the science test is on Friday, and please bring your lab notebooks tomorrow.
 - Also, the library will be closed on Wednesday afternoon.

Why it fails: the code only sees sentences and characters. Deciding what matters depends on meaning. Smarter rules (skip greetings, count frequent words) break on the next teacher's notes.

The fix: a large language model (LLM), a program trained on huge amounts of text that predicts text, so it can answer, rewrite and summarize. A good one runs on a big computer somewhere else, so our app talks to it through an API.

What an LLM API is (a restaurant counter)

At a restaurant LLM API
The counter: one fixed place to order Endpoint: the exact URL you send requests to, e.g. /v1/chat/completions
Your order slip Request: a JSON message with the model, the messages and settings
Which chef cooks it (the senior chef costs more) Model: which LLM answers. Bigger is usually better, slower, more expensive
Your token number and the bill API key (who you are, who pays) and usage (how many tokens you used)

Your first API request (real)

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5-1.5b-instruct",
       "messages": [
         {"role": "system", "content": "You are a helpful study assistant."},
         {"role": "user", "content": "In one sentence: what is evaporation?"}],
       "temperature": 0, "max_tokens": 60}'
  • curl: a terminal tool that sends web requests.
  • Authorization: Bearer ...: the API key, read from a variable (never typed into a command you share).
  • model: which LLM answers.
  • messages: the conversation. The system message sets the rules (a job description); the user message is the actual question. The model's answers have the role assistant.
  • temperature: how random the answer is (below).
  • max_tokens: the longest answer allowed, in tokens.

Real response (shortened: id, created, index and logprobs removed):

{
  "object": "chat.completion",
  "model": "qwen2.5-1.5b-instruct",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Evaporation is the process by which a liquid changes to a vapor."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 29, "completion_tokens": 14, "total_tokens": 43 }
}
  • choices[0].message.content: the answer.
  • finish_reason: stop = the model finished on its own; length = it was cut off by max_tokens.
  • usage: tokens in (prompt_tokens) and out (completion_tokens). Cloud providers bill exactly these numbers.

Tokens

Claude's glossary: tokens are "the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes". In our stream (below), evaporation arrived as two tokens: Ev + aporation.

  • English is about 3.5 to 4.6 characters per token (Claude's docs: about 3.5 English characters per token on earlier models; Claude 4.7 and later use a newer tokenizer that makes about 30% more tokens for the same text).
  • Each model has its own tokenizer (the part that splits text into tokens), so counts differ between models and languages. Our server's tokenizer (POST /extras/tokenize) measured:
Text Characters Tokens Characters per token
Asha's English notes 2,089 452 4.62
The same chapter in Hindi 911 864 1.05

The API key (real 401s)

An API key is a long secret string that works like a password for the API: it says who is asking and who pays.

$ curl ... -H "Authorization: Bearer demo-key-wrong" ...
{"detail":"Invalid API key"}
HTTP 401

$ curl ... (no Authorization header) ...
{"detail":"Invalid API key"}
HTTP 401

401 means "we don't know who you are". Claude's API returns 401 authentication_error for a malformed, revoked or expired key. With a cloud provider, anyone holding your key can spend your money.

The obvious way to build the app: the key inside the page

A one-file version (we wrote it to show the trap). It works, but:

<script>
  const API_KEY = "demo-key-not-real";   // the key, inside the page
  document.getElementById("go").onclick = async () => {
    const res = await fetch("http://localhost:8080/v1/chat/completions", {
      method: "POST",
      headers: { "Content-Type": "application/json",
                 "Authorization": "Bearer " + API_KEY },
      ...

Right-click → View page source shows the key on line 9. A web page is a file sent to the visitor's computer (the client); whatever is in it, they can read, minified or not.

The fix: a small server in the middle. The page asks our server; our server adds the key (from .env) and asks the LLM. .env (a file of secret settings) goes in .gitignore, so it never reaches GitHub. Claude's error docs even describe a 401 for requests sent straight from a browser, with the advice: "send the request from your own back end".

Browser (no key)  →  our server: /api/summarize (key from .env)  →  LLM API

The AI builds it (agent run 1)

claude -p "Build me a summarizer web app in this folder. A page where I paste my class notes, click Summarize, and get the 5 most important points as bullets. It should call my LLM API: it is OpenAI-compatible, the base URL, model name and API key are in .env. Keep it simple: plain HTML and JavaScript." --permission-mode acceptEdits

What it did (real log, shortened):

Bash   sed 's/=.*/=<redacted>/' .env      (reads the names, hides the values)
Bash   curl .../v1/models                  → "requires approval"
Reply: "I'll put a tiny Node server in front of the page. That way
   the API key stays on the server and never reaches the browser."
Write  server.js, index.html, .gitignore   (.env and node_modules/)
Bash   node smoke.js                        → "requires approval"
Reply: "I built the summarizer, but I haven't run it ...
   Notes are limited to 50,000 characters."

Good habits to copy: it read .env with every value replaced by <redacted>, it kept the key on the server without being asked, and it added .env to .gitignore.

The heart of server.js (version 1):

const SYSTEM_PROMPT =
  "You summarize a student's class notes. Identify the 5 most important points " +
  "about the subject being taught. Ignore greetings and small talk. " +
  "Reply with exactly 5 lines. Each line starts with \"- \" followed by one short, " +
  "clear sentence. No title, no introduction, no extra text.";

const res = await fetch(`${BASE_URL}/chat/completions`, {
  method: "POST",
  headers: { "Content-Type": "application/json",
             Authorization: `Bearer ${API_KEY}` },      // from .env, on the server
  body: JSON.stringify({
    model: MODEL,
    temperature: 0.2,
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      { role: "user", content: `Class notes:\n\n${notes}` },
    ],
  }),
});

And the page only calls our server:

const res = await fetch("/api/summarize", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ notes }),
});

It works (real)

node server.js, open http://localhost:3000, paste the notes, click Summarize. One real result:

- The water cycle involves evaporation, condensation, precipitation and collection.
- Evaporation occurs when the sun heats water in oceans, rivers and lakes, turning it into vapor.
- Condensation happens as cooler air holds less moisture, causing vapor to turn back into liquid drops.
- Precipitation is the formation of these drops into clouds, leading to rain or other forms of precipitation.
- The cycle continues through collection, with rainwater flowing into water bodies and groundwater.

But two real problems: we timed 25 s and 16 s of "Summarizing..." with nothing on screen, and point 4 is wrong (see "The model's mistakes").

AI mistake 1: a limit four times too big

Asha pastes a whole unit: 20,908 characters (we pasted today's notes ten times). The page showed the raw error, spilling out of its box:

LLM API returned 400: {"error":{"message":"This model's maximum context length is 4096 tokens. However, you requested 4594 tokens (4594 in the messages, None in the completion). Please reduce the length of the messages or completion.","type":"invalid_request_error","param":"messages","code":"context_length_exceeded"}}

Why: the AI's code had const MAX_NOTES_CHARS = 50000; and passed every API error straight to the page. 50,000 characters is more than 11,000 tokens of English: it could never fit.

The context window

The context window is the model's desk: the instructions, the notes and the answer must all fit on it at once. Claude's glossary calls it the model's "working memory". Our server was started with 4,096 tokens (--n_ctx 4096); cloud models have far bigger windows, but every window has an edge.

The fix (agent run 2): length check, streaming, rate limit

claude -p "Three problems. 1) I pasted a long unit (20,908 characters) and the page showed a raw error: LLM API returned 400: {...context_length_exceeded...}. Your limit is 50,000 characters, but my model can only read 4096 tokens. Check the length before calling the API, show a short friendly message, and keep the technical details in the server log only. 2) I wait about 20 seconds looking at \"Summarizing...\" and it feels broken. Stream the answer so the points appear while they are being written. 3) Anyone on my Wi-Fi could call /api/summarize in a loop and use up my API credits. Add a simple rate limit: 5 summaries per minute per IP address, with a clear message. Keep everything else the same." --permission-mode acceptEdits

(In the real prompt the error was pasted in full.) It edited both files and wrote a fake LLM server to test with, but could not run it. Its new limit:

const CONTEXT_TOKENS = 4096;
const MAX_REPLY_TOKENS = 400;
const PROMPT_OVERHEAD_TOKENS = 150;
const CHARS_PER_TOKEN = 3.5;
const MAX_NOTES_CHARS = Math.floor(
  (CONTEXT_TOKENS - MAX_REPLY_TOKENS - PROMPT_OVERHEAD_TOKENS) * CHARS_PER_TOKEN);  // 12,411

Real retest with the same 20,908 characters: "These notes are too long for the AI model to read in one go (20,907 characters; the limit is about 12,411). Try summarizing them in smaller parts." No API call, no cost, details only in the server log.

Streaming

A normal response is tomorrow's newspaper: the full report, late. Streaming is live commentary: the model writes one token at a time anyway, so the API sends each piece as soon as it is made. Same total time, but the user sees progress at once.

Real stream ("stream": true, max_tokens 12; id, model, created, object, index and logprobs removed from each line):

data: {"choices": [{"delta": {"role": "assistant"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": "Ev"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": "aporation"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": " is"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": " the"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": " process"}, "finish_reason": null}]}
   ... " by", " which", " a", " liquid", " changes", " into"
data: {"choices": [{"delta": {"content": " a"}, "finish_reason": null}]}
data: {"choices": [{"delta": {}, "finish_reason": "length"}]}
data: [DONE]

Each data: line carries a delta, the next small piece; the code glues them together. finish_reason: "length" because we allowed only 12 tokens. data: [DONE] ends the stream. In the app, the first point appears while the second is still being written.

AI mistake 2: Hindi notes, "please try again" forever

Asha's friend pastes Hindi notes: 6,382 characters, under the 12,411 limit, so allowed.

Page:        "Sorry, the summary couldn't be created right now.
              Please try again in a moment."
Server log:  Error: The model returned an empty summary.
             at handleSummarize (server.js:214:37)

The same request sent straight to the LLM server with "stream": true came back HTTP 200 with an empty body: with streaming on, this server did not report the "too long" error at all, so the app guessed "try again", which can never work.

Why it got through (run in the compiler, real counts from /extras/tokenize):

const samples = [
  { name: "English notes", chars: 2089, tokens: 452 },
  { name: "Hindi notes",   chars: 911,  tokens: 864 },
];
const LIMIT = 3546;   // 4096 context - 400 for the reply - 150 for instructions

for (const s of samples) {
  const perToken = s.chars / s.tokens;
  const guess = Math.round(s.chars / 3.5);           // the AI's estimate
  console.log(s.name.padEnd(14), (perToken.toFixed(2) + " chars/token").padEnd(17),
              "guess:", String(guess).padStart(4), " real:", String(s.tokens).padStart(4));
}
console.log("Guess for 6,382 chars:", Math.round(6382 / 3.5), "tokens -> allowed");
console.log("Real count: 6050 tokens -> limit is", LIMIT, "-> too long!");
English notes  4.62 chars/token  guess:  597  real:  452
Hindi notes    1.05 chars/token  guess:  260  real:  864
Guess for 6,382 chars: 1823 tokens -> allowed
Real count: 6050 tokens -> limit is 3546 -> too long!

The AI's own comment said "~3.5 characters per token; English is usually ~4": true only for English.

The fix (agent run 3): count, don't guess

claude -p "A bug. My friend pasted Hindi notes: 6,383 characters, under your limit of 12,411. The page said \"Sorry, the summary couldn't be created right now. Please try again in a moment.\" and the server log says \"Error: The model returned an empty summary.\" Trying again never helps. I checked with my LLM server's POST /extras/tokenize endpoint (body {\"input\": \"...\"}, it returns {\"tokens\": [...]}): 911 characters of Hindi are 864 tokens, but 2,089 characters of English are 452 tokens. Your 3.5 characters per token only works for English. Before calling the model, count the real tokens with /extras/tokenize (same base URL host, same API key). If that endpoint fails, fall back to 1 character per token. When the notes are too long, show the \"too long\" message, not \"try again\". Change nothing else." --permission-mode acceptEdits

It added countTokens() (with a 10-second timeout and the 1-character-per-token fallback) and a token check. Our retest:

Server log: Rejected notes: 6382 chars = 6050 tokens > limit 3546 tokens (context 4096)
Page:       These notes are too long for the AI model to read in one go
            (6,382 characters; the limit is about 3,740). Try summarizing them in smaller parts.

Lessons: test with your real users' text (other languages, very long, empty); say "try again" only when trying again can work; count tokens with the model's own tokenizer. /extras/tokenize belongs to this local server; cloud providers have their own token counting (Claude's docs point to their token counting feature for exact counts).

Temperature

Claude's glossary: temperature "controls the randomness of a model's predictions". Low = the most likely words (Mummy's chai, same every day); high = sometimes less likely words (a cook who experiments).

Real runs, the same request six times ("Give a catchy 5-word title for a class chapter on the water cycle."):

temperature 0   → "Exploring the Water Cycle: A Journey Through the Atmosphere"
temperature 0   → "Exploring the Water Cycle: A Journey Through the Atmosphere"
temperature 0   → "Exploring the Water Cycle: A Journey Through the Atmosphere"
temperature 1.5 → "Rain Cycle Revealed: Understanding the Water Cycle"
temperature 1.5 → "Rivers, Lakes, and the Sky Cycle"
temperature 1.5 → "Exploring the Water Cycle: From Evaporation to Precipitation"
  • Low (0 to 0.3) for summaries, facts and code; higher for names, ideas and stories.
  • "5-word" titles with 9 words: models don't count reliably, so check the format in code (the AI's parseBullets keeps at most 5 bullets; in one real run the model wrote 6).
  • Claude's docs: "Even with temperature set to 0, the results will not be fully deterministic."
  • Claude's OpenAI SDK compatibility page: temperature is "Between 0 and 1"; on Claude 4.7 and later models "any value below 1 returns a 400 error, so omit it". Settings differ per model: read the docs before copying one.

The model's mistakes: check the answer against the source

Real summaries from version 1 (temperature 0.2):

  • "Precipitation is the formation of these drops into clouds" (wrong: forming clouds is condensation; precipitation is the drops falling).
  • "Evaporation occurs when water turns into vapor due to heat from the sun or plants." (wrong: plants release water vapour, they don't heat water).
  • "The monsoon brings most of India's rainfall during June to September, affecting local climates." ("affecting local climates" is not in the notes).

The code was correct; the answers were wrong. A small model mixes up facts even when they are in the text. For anything that matters, let the user check the source (Episode 21 builds answers with sources).

Cost per token

Run in the compiler. One real summary used 526 prompt tokens and 113 completion tokens; prices from the Claude pricing page on 12 November 2026 (US dollars per million tokens, MTok; Haiku 5.5 price for prompts up to 100,000 tokens):

const promptTokens = 526, outputTokens = 113;

const models = {
  "Claude Haiku 5.5": { input: 0.10, output: 0.50 },
  "Claude Sonnet 5.5": { input: 2, output: 10 },
};

for (const [name, price] of Object.entries(models)) {
  const one = (promptTokens * price.input + outputTokens * price.output) / 1e6;
  const month = one * 40 * 10 * 30;          // 40 students, 10 a day, 30 days
  const spam = one * 10 * 60 * 60;           // a loop: 10 calls a second, 1 hour
  console.log(name.padEnd(17), "one $" + one.toFixed(6),
              " month $" + month.toFixed(2), " loop $" + spam.toFixed(2));
}
Claude Haiku 5.5  one $0.000109  month $1.31  loop $3.93
Claude Sonnet 5.5 one $0.002182  month $26.18  loop $78.55

These token counts are our local model's; Claude counts the same text differently, so treat the result as an estimate. Prices change: check the pricing page on the day you build.

Rate limits

A rate limit caps how many requests someone can make in a time window. Real run: five quick requests, then Asha clicked Summarize: "You've reached the limit of 5 summaries per minute. Please wait 27 seconds and try again." (HTTP 429 with a Retry-After header). The AI's limiter keeps a list of request times per IP address and counts only requests that would really call the API.

Providers limit you too. Claude's error docs: 429 rate_limit_error (your organization hit a rate limit or spend cap), 529 overloaded_error (the API is temporarily overloaded for everyone), 500 api_error (retry with exponential backoff).

Switching to a cloud model

# .env on our laptop (local server, made-up key)
LLM_BASE_URL=http://localhost:8080/v1
LLM_MODEL=qwen2.5-1.5b-instruct
LLM_API_KEY=demo-key-not-real

# .env for the Claude API (Claude docs: OpenAI SDK compatibility)
LLM_BASE_URL=https://api.anthropic.com/v1
LLM_MODEL=claude-haiku-5-5
LLM_API_KEY=<your key from the Claude Console>

Claude's docs say this OpenAI-style layer "is primarily intended to test and compare model capabilities"; for production they recommend the native Claude API. Also note: the server-specific /extras/tokenize endpoint does not exist there, so the AI's fallback (1 character per token) would apply; use the provider's own token counting instead.

Project history:

eaa4e19 Count real tokens before calling the model (Claude Code, prompt 3)
c88d8be Length check, streaming, rate limit (Claude Code, prompt 2)
1b03e51 Summarizer: page + server that keeps the key (Claude Code, prompt 1)
47cfaec Class notes

git ls-files lists only .gitignore, index.html, notes.txt and server.js: .env was never committed.

The finished app (all files)

.gitignore:

.env
node_modules/

server.js (Node 18+, no dependencies; run node server.js):

// Minimal server: serves index.html and proxies summarize requests to the LLM.
// The API key stays here on the server and is never sent to the browser.
// Run: node server.js   (Node 20+, no dependencies)

const http = require("http");
const fs = require("fs");
const path = require("path");

// --- Load .env (simple KEY=VALUE parser) ---
const env = {};
const envPath = path.join(__dirname, ".env");
if (fs.existsSync(envPath)) {
  for (const line of fs.readFileSync(envPath, "utf8").split(/\r?\n/)) {
    const m = line.match(/^\s*([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)\s*$/);
    if (m) env[m[1]] = m[2].replace(/^(['"])(.*)\1$/, "$2");
  }
}
const BASE_URL = (process.env.LLM_BASE_URL || env.LLM_BASE_URL || "").replace(/\/+$/, "");
const MODEL = process.env.LLM_MODEL || env.LLM_MODEL;
const API_KEY = process.env.LLM_API_KEY || env.LLM_API_KEY;
const PORT = process.env.PORT || 3000;

if (!BASE_URL || !MODEL) {
  console.error("Missing LLM_BASE_URL or LLM_MODEL in .env");
  process.exit(1);
}

// --- Size limits ---
// The model can only read CONTEXT_TOKENS tokens in total (notes + instructions + the reply).
// We can't count tokens exactly without the model's tokenizer, so we estimate
// conservatively (~3.5 characters per token; English is usually ~4) and refuse
// notes that won't fit *before* spending an API call on them.
const CONTEXT_TOKENS = Number(process.env.LLM_CONTEXT_TOKENS || env.LLM_CONTEXT_TOKENS) || 4096;
const MAX_REPLY_TOKENS = 400;      // 5 short lines need far less; this caps the reply
const PROMPT_OVERHEAD_TOKENS = 150; // system prompt + chat formatting + safety margin
const CHARS_PER_TOKEN = 3.5;
const MAX_NOTES_CHARS = Math.min(
  50000,
  Math.floor((CONTEXT_TOKENS - MAX_REPLY_TOKENS - PROMPT_OVERHEAD_TOKENS) * CHARS_PER_TOKEN)
);
const MAX_BODY_CHARS = 100000; // raw request size guard (unchanged)
// The character estimate above only holds for English (Hindi is ~1 char per token),
// so before calling the model we also count the real tokens with the LLM server's
// tokenizer. If that fails, we assume the worst case of 1 character per token.
const MAX_NOTES_TOKENS = CONTEXT_TOKENS - MAX_REPLY_TOKENS - PROMPT_OVERHEAD_TOKENS;
const TOKENIZE_URL = `${new URL(BASE_URL).origin}/extras/tokenize`;

// Returns the number of tokens in `text`, or text.length if the tokenizer can't be used.
async function countTokens(text) {
  try {
    const headers = { "Content-Type": "application/json" };
    if (API_KEY) headers.Authorization = `Bearer ${API_KEY}`;
    const res = await fetch(TOKENIZE_URL, {
      method: "POST",
      headers,
      body: JSON.stringify({ input: text }),
      signal: AbortSignal.timeout(10000),
    });
    if (!res.ok) throw new Error(`tokenize returned ${res.status}: ${(await res.text()).slice(0, 200)}`);
    const data = await res.json();
    if (!Array.isArray(data?.tokens)) throw new Error("tokenize reply has no tokens array");
    return data.tokens.length;
  } catch (err) {
    console.warn(`Token count failed (${err.message}); assuming 1 character per token.`);
    return text.length;
  }
}

// --- Rate limit: RATE_LIMIT summaries per RATE_WINDOW_MS per IP address ---
const RATE_LIMIT = 5;
const RATE_WINDOW_MS = 60 * 1000;
const recentRequests = new Map(); // ip -> [timestamps]

// Returns 0 if the request is allowed (and records it), else seconds to wait.
function rateLimitWait(ip) {
  const now = Date.now();
  const times = (recentRequests.get(ip) || []).filter((t) => now - t < RATE_WINDOW_MS);
  if (times.length >= RATE_LIMIT) {
    recentRequests.set(ip, times);
    return Math.max(1, Math.ceil((times[0] + RATE_WINDOW_MS - now) / 1000));
  }
  times.push(now);
  recentRequests.set(ip, times);
  return 0;
}
// Forget idle IPs so the map doesn't grow forever.
setInterval(() => {
  const now = Date.now();
  for (const [ip, times] of recentRequests)
    if (!times.some((t) => now - t < RATE_WINDOW_MS)) recentRequests.delete(ip);
}, RATE_WINDOW_MS).unref();

// --- Friendly messages for the page (technical details go to the server log only) ---
// `limit` defaults to the English-based estimate; for a token-count rejection we pass
// the limit scaled to these notes' own characters-per-token, so the number is honest.
const MSG_TOO_LONG = (len, limit = MAX_NOTES_CHARS) =>
  `These notes are too long for the AI model to read in one go (${len.toLocaleString("en-US")} characters; ` +
  `the limit is about ${limit.toLocaleString("en-US")}). Try summarizing them in smaller parts.`;
const MSG_FAILED = "Sorry, the summary couldn't be created right now. Please try again in a moment.";

// An error whose message is safe to show on the page.
class UserError extends Error {
  constructor(userMessage, status, detail) {
    super(detail || userMessage);
    this.userMessage = userMessage;
    this.status = status;
  }
}

const SYSTEM_PROMPT =
  "You summarize a student's class notes. Identify the 5 most important points " +
  "about the subject being taught. Ignore greetings and small talk. " +
  "Reply with exactly 5 lines. Each line starts with \"- \" followed by one short, " +
  "clear sentence. No title, no introduction, no extra text.";

// Pull bullet lines out of the model's reply, tolerating "1.", "*", "•", etc.
function parseBullets(text) {
  return text
    .split(/\r?\n/)
    .map((l) => l.trim())
    .filter(Boolean)
    .map((l) => l.replace(/^(?:[-*•]|\d+[.)])\s*/, "").replace(/^\*\*(.+?)\*\*:?\s*/, "$1: ").trim())
    .filter((l) => l && !/^(here (are|is)|summary|key points)\b.*:$/i.test(l))
    .slice(0, 5);
}

// Start a streaming request to the LLM. Resolves once the API has accepted it
// (so errors like "too long" can still be sent as a normal error response).
async function startSummary(notes, signal) {
  const headers = { "Content-Type": "application/json" };
  if (API_KEY) headers.Authorization = `Bearer ${API_KEY}`;

  const res = await fetch(`${BASE_URL}/chat/completions`, {
    method: "POST",
    headers,
    body: JSON.stringify({
      model: MODEL,
      temperature: 0.2,
      max_tokens: MAX_REPLY_TOKENS,
      stream: true,
      messages: [
        { role: "system", content: SYSTEM_PROMPT },
        { role: "user", content: `Class notes:\n\n${notes}` },
      ],
    }),
    signal: AbortSignal.any([signal, AbortSignal.timeout(120000)]),
  });

  if (!res.ok) {
    const detail = `LLM API returned ${res.status}: ${(await res.text()).slice(0, 500)}`;
    // Our estimate let it through but the model still says it's too long.
    if (/context_length_exceeded|maximum context length/i.test(detail))
      throw new UserError(MSG_TOO_LONG(notes.length), 400, detail);
    throw new UserError(MSG_FAILED, 502, detail);
  }
  return res;
}

// Read the API's server-sent events and call onText(fullTextSoFar) as text arrives.
async function readStream(res, onText) {
  const decoder = new TextDecoder();
  let buf = "";
  let text = "";
  for await (const chunk of res.body) {
    buf += decoder.decode(chunk, { stream: true });
    let nl;
    while ((nl = buf.indexOf("\n")) >= 0) {
      const line = buf.slice(0, nl).trim();
      buf = buf.slice(nl + 1);
      if (!line.startsWith("data:")) continue;
      const payload = line.slice(5).trim();
      if (payload === "[DONE]") return text;
      let evt;
      try { evt = JSON.parse(payload); } catch { continue; }
      if (evt.error) throw new Error(`LLM stream error: ${JSON.stringify(evt.error).slice(0, 500)}`);
      const delta = evt?.choices?.[0]?.delta?.content;
      if (delta) { text += delta; onText(text); }
    }
  }
  return text;
}

function sendJson(res, status, obj, extraHeaders = {}) {
  res.writeHead(status, { "Content-Type": "application/json", ...extraHeaders });
  res.end(JSON.stringify(obj));
}

function clientIp(req) {
  return (req.socket.remoteAddress || "unknown").replace(/^::ffff:/, "");
}

async function handleSummarize(req, res, body) {
  let notes;
  try {
    notes = String(JSON.parse(body).notes || "").trim();
  } catch {
    return sendJson(res, 400, { error: "Something went wrong sending your notes. Please try again." });
  }
  if (!notes) return sendJson(res, 400, { error: "Please paste some notes first." });
  if (notes.length > MAX_NOTES_CHARS) {
    console.warn(`Rejected notes: ${notes.length} chars > limit ${MAX_NOTES_CHARS} (context ${CONTEXT_TOKENS} tokens)`);
    return sendJson(res, 400, { error: MSG_TOO_LONG(notes.length) });
  }
  const tokens = await countTokens(`Class notes:\n\n${notes}`);
  if (tokens > MAX_NOTES_TOKENS) {
    const limitChars = Math.floor((notes.length * MAX_NOTES_TOKENS) / tokens);
    console.warn(`Rejected notes: ${notes.length} chars = ${tokens} tokens > limit ${MAX_NOTES_TOKENS} tokens (context ${CONTEXT_TOKENS})`);
    return sendJson(res, 400, { error: MSG_TOO_LONG(notes.length, limitChars) });
  }

  // Only requests that would actually call the API count toward the limit.
  const ip = clientIp(req);
  const wait = rateLimitWait(ip);
  if (wait) {
    console.warn(`Rate limit hit for ${ip}`);
    return sendJson(
      res, 429,
      { error: `You've reached the limit of ${RATE_LIMIT} summaries per minute. Please wait ${wait} second${wait === 1 ? "" : "s"} and try again.` },
      { "Retry-After": String(wait) }
    );
  }

  // Stop the LLM call (and stop spending credits) if the browser goes away.
  const abort = new AbortController();
  res.on("close", () => { if (!res.writableFinished) abort.abort(); });

  let streaming = false;
  try {
    const upstream = await startSummary(notes, abort.signal);

    // From here on we stream newline-delimited JSON events to the page.
    res.writeHead(200, {
      "Content-Type": "application/x-ndjson; charset=utf-8",
      "Cache-Control": "no-cache",
      "X-Accel-Buffering": "no",
    });
    streaming = true;
    const send = (obj) => res.write(JSON.stringify(obj) + "\n");

    let lastSent = "";
    const text = await readStream(upstream, (soFar) => {
      const bullets = parseBullets(soFar);
      const key = JSON.stringify(bullets);
      if (key !== lastSent) { lastSent = key; send({ type: "bullets", bullets }); }
    });

    const bullets = parseBullets(text);
    if (bullets.length === 0) throw new Error("The model returned an empty summary.");
    send({ type: "done", bullets });
    res.end();
  } catch (err) {
    if (abort.signal.aborted) return console.warn(`Client ${ip} disconnected; LLM request cancelled.`);
    console.error(err);
    const message = err instanceof UserError ? err.userMessage : MSG_FAILED;
    if (streaming) {
      res.write(JSON.stringify({ type: "error", error: message }) + "\n");
      res.end();
    } else {
      sendJson(res, err instanceof UserError ? err.status : 502, { error: message });
    }
  }
}

const server = http.createServer(async (req, res) => {
  if (req.method === "GET" && (req.url === "/" || req.url === "/index.html")) {
    res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
    fs.createReadStream(path.join(__dirname, "index.html")).pipe(res);
    return;
  }

  if (req.method === "POST" && req.url === "/api/summarize") {
    let body = "";
    let tooBig = false;
    req.on("data", (chunk) => {
      body += chunk;
      if (body.length > MAX_BODY_CHARS) { tooBig = true; req.destroy(); }
    });
    req.on("end", () => {
      if (tooBig) return;
      handleSummarize(req, res, body);
    });
    return;
  }

  res.writeHead(404);
  res.end("Not found");
});

server.listen(PORT, () => {
  console.log(`Summarizer running at http://localhost:${PORT}  (model: ${MODEL})`);
  console.log(`Notes limit: ~${MAX_NOTES_CHARS} characters (${CONTEXT_TOKENS}-token model). Rate limit: ${RATE_LIMIT}/min per IP.`);
});

index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Notes Summarizer</title>
  <style>
    :root { --bg: #f6f7f9; --card: #fff; --text: #1d2330; --muted: #667085; --accent: #2f6fed; --border: #d9dde5; --err: #c0392b; }
    @media (prefers-color-scheme: dark) {
      :root { --bg: #14171c; --card: #1d2128; --text: #e6e8ec; --muted: #9aa3b2; --accent: #5b8cff; --border: #333a45; --err: #ff7b6b; }
    }
    * { box-sizing: border-box; }
    body { margin: 0; background: var(--bg); color: var(--text); font: 16px/1.5 system-ui, -apple-system, Segoe UI, Roboto, sans-serif; }
    main { max-width: 760px; margin: 0 auto; padding: 32px 16px; }
    h1 { margin: 0 0 4px; font-size: 1.6rem; }
    p.sub { margin: 0 0 20px; color: var(--muted); }
    textarea { width: 100%; min-height: 260px; padding: 12px; border: 1px solid var(--border); border-radius: 8px; background: var(--card); color: var(--text); font: inherit; resize: vertical; }
    .row { display: flex; align-items: center; gap: 12px; margin: 12px 0 24px; }
    button { padding: 10px 22px; border: 0; border-radius: 8px; background: var(--accent); color: #fff; font: inherit; font-weight: 600; cursor: pointer; }
    button:disabled { opacity: .6; cursor: wait; }
    #count { color: var(--muted); font-size: .9rem; }
    #result { background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 16px 20px; }
    #result h2 { margin: 0 0 8px; font-size: 1.1rem; }
    #result ul { margin: 0; padding-left: 20px; }
    #result li { margin: 6px 0; }
    .error { color: var(--err); }
    .status { margin: 0; color: var(--muted); }
    [hidden] { display: none !important; }
  </style>
</head>
<body>
  <main>
    <h1>Notes Summarizer</h1>
    <p class="sub">Paste your class notes and get the 5 most important points.</p>

    <textarea id="notes" placeholder="Paste your class notes here..."></textarea>
    <div class="row">
      <button id="go">Summarize</button>
      <span id="count">0 characters</span>
    </div>

    <section id="result" hidden>
      <h2>Key points</h2>
      <p id="status" class="status" hidden></p>
      <ul id="bullets"></ul>
      <p id="error" class="error" hidden></p>
    </section>
  </main>

  <script>
    const notesEl = document.getElementById("notes");
    const btn = document.getElementById("go");
    const countEl = document.getElementById("count");
    const resultEl = document.getElementById("result");
    const bulletsEl = document.getElementById("bullets");
    const errorEl = document.getElementById("error");
    const statusEl = document.getElementById("status");

    notesEl.addEventListener("input", () => {
      countEl.textContent = notesEl.value.length + " characters";
    });

    function renderBullets(points) {
      bulletsEl.innerHTML = "";
      for (const point of points) {
        const li = document.createElement("li");
        li.textContent = point; // textContent, so model output can't inject HTML
        bulletsEl.appendChild(li);
      }
    }

    function showError(msg) {
      bulletsEl.innerHTML = "";
      errorEl.textContent = msg;
      errorEl.hidden = false;
      statusEl.hidden = true;
      resultEl.hidden = false;
    }

    btn.addEventListener("click", async () => {
      const notes = notesEl.value.trim();
      if (!notes) return showError("Please paste some notes first.");

      btn.disabled = true;
      btn.textContent = "Summarizing…";
      errorEl.hidden = true;

      try {
        const res = await fetch("/api/summarize", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ notes }),
        });
        if (!res.ok) {
          const data = await res.json().catch(() => ({}));
          throw new Error(data.error || "Something went wrong. Please try again.");
        }

        // Show the box right away, then fill in points as the server streams them.
        bulletsEl.innerHTML = "";
        statusEl.textContent = "Reading your notes…";
        statusEl.hidden = false;
        resultEl.hidden = false;

        const reader = res.body.getReader();
        const decoder = new TextDecoder();
        let buf = "";
        let finished = false;
        while (true) {
          const { value, done } = await reader.read();
          if (done) break;
          buf += decoder.decode(value, { stream: true });
          let nl;
          while ((nl = buf.indexOf("\n")) >= 0) {
            const line = buf.slice(0, nl).trim();
            buf = buf.slice(nl + 1);
            if (!line) continue;
            const msg = JSON.parse(line);
            if (msg.type === "error") throw new Error(msg.error);
            if (msg.type === "bullets" || msg.type === "done") {
              statusEl.hidden = true;
              renderBullets(msg.bullets);
            }
            if (msg.type === "done") finished = true;
          }
        }
        if (!finished) throw new Error("The summary was interrupted. Please try again.");
      } catch (err) {
        showError(err.message === "Failed to fetch"
          ? "Couldn't reach the summarizer. Is the server running?"
          : err.message);
      } finally {
        btn.disabled = false;
        btn.textContent = "Summarize";
      }
    });
  </script>
</body>
</html>

Common mistakes (real)

Mistake What you see Why Fix
const API_KEY = "..." in the page The key in View page source The browser gets the whole page Key in .env on the server; page calls your server
Wrong or missing key {"detail":"Invalid API key"} HTTP 401 The API doesn't know who you are Check .env, restart the server
MAX_NOTES_CHARS = 50000 This model's maximum context length is 4096 tokens Limit not checked against the model Check length (in tokens) before calling
Raw API error sent to the page JSON spilling out of the box Technical details aren't for users Friendly message; details in the server log
chars / 3.5 for every language Hindi: "try again" forever 3.5-4.6 chars/token is English only Count with the model's tokenizer
No streaming 16-25 s of "Summarizing..." The whole answer is sent at the end "stream": true, show pieces as they arrive
No rate limit A loop could cost $78.55 an hour (Sonnet 5.5) Every call is billed Limit per IP; 429 + Retry-After
Trusting the answer "Precipitation ... into clouds" The model mixed up facts Low temperature, check against the source
Copying a temperature setting A 400 error on Claude 4.7+ below 1 Settings differ per model Read the model's docs

Mini project

Build a "Quiz me" app with an AI: paste notes, get 3 multiple-choice questions with answers. Rules: the key stays on the server in .env; the page only calls your server.

  1. Ask for the questions as JSON. Run it five times: does the model always return valid JSON? Add a check in code and a friendly message when it doesn't.
  2. Try temperature 0 and 1.2: which gives better questions? Which gives more wrong answers?
  3. Paste notes in another language and count the tokens. Is your length check still right?
  4. Work out the cost of one quiz with your provider's current prices and the usage numbers.

Practice questions

  1. Why did "first 3 sentences" fail as a summary?
Answer

It only looks at positions and characters. The first sentences were greetings and reminders; deciding what matters needs meaning.

  1. Name the four main parts of a chat completions request.
Answer

The endpoint URL, the Authorization header with the API key, the model, and the messages (plus optional settings such as temperature and max_tokens).

  1. What is the difference between a system message and a user message?
Answer

The system message sets the rules for the whole conversation (a job description). The user message is the actual question or input.

  1. A response has "finish_reason": "length". What happened?
Answer

The answer was cut off because it reached max_tokens. Raise max_tokens or ask for a shorter answer.

  1. Why is minifying the JavaScript not enough to hide a key in a web page?
Answer

Minifying only makes code harder to read; the whole file is still sent to every visitor, so the key is still there (View page source, DevTools Network tab).

  1. A model has a 4,096-token context window. You reserve 400 tokens for the answer and 150 for the instructions. How many tokens of notes fit?
Answer

4,096 - 400 - 150 = 3,546 tokens.

  1. Why did 6,382 characters of Hindi fail while 12,000 characters of English would pass?
Answer

Hindi was about 1.05 characters per token on this model (6,050 tokens), English about 4.6 (about 2,600 tokens). The limit is in tokens, not characters.

  1. What does streaming change, and what does it not change?
Answer

It shows the answer piece by piece as it's written (first words much sooner). It does not make the whole answer faster or cheaper.

  1. One request uses 1,000 input tokens and 200 output tokens on Claude Sonnet 5.5 ($2 / $10 per MTok). What does it cost?
Answer

(1,000 × 2 + 200 × 10) / 1,000,000 = (2,000 + 2,000) / 1,000,000 = $0.004.

  1. The same summary request gives slightly different answers each time. Is something broken?
Answer

No. With temperature above 0 the model picks among likely words, and Claude's docs say even temperature 0 is not fully deterministic. Use low temperature for facts and check the content.

Quiz

Where should the API key live?

A) In a const at the top of index.html
B) In the page's JavaScript, minified
C) In .env on your server, which calls the LLM
D) In the README, so teammates can find it
Answer

C. Everything in a web page is sent to the visitor, minified or not. The key belongs in .env on the server (listed in .gitignore); the page calls your server, and your server calls the LLM API.

Interview questions

  1. Walk through what happens between a user clicking "Summarize" and the bullets appearing, in this app.
  2. Why put an LLM call behind your own back end? Name three reasons (key safety, rate limiting, input checks, logging, switching providers).
  3. What is a context window, and how would you handle a document that is bigger than it? (Split it into parts, summarize each, then summarize the summaries; or retrieve only the relevant parts, as in Episode 21.)
  4. How would you estimate the monthly cost of an LLM feature before launching it?
  5. How do server-sent events work, and how do errors show up in a stream?
  6. What would you log for an LLM request, and what would you never log? (Never the API key; be careful with users' text.)
  7. How do you make an LLM feature's output safe to insert into a web page? (Treat it as untrusted text: textContent, not innerHTML, as the AI's code does.)

Glossary

Term Meaning (plain words) Example
Summary A short version that keeps the most important points 5 bullets from 2,089 characters
LLM (large language model) A program trained on huge amounts of text that predicts text, so it can answer, rewrite and summarize Qwen2.5-1.5B-Instruct, Claude
Parameters The numbers a model learned in training; more usually means more capable and slower 1.54 billion
Open model A model whose files you can download and run yourself, under a licence Qwen2.5, Apache-2.0
API A fixed way for one program to ask another for something The LLM API
LLM API request A message to the API asking the model for an answer POST /v1/chat/completions
Endpoint The exact URL you send a request to http://localhost:8080/v1/chat/completions
Base URL The start of every endpoint address for one API http://localhost:8080/v1
OpenAI-compatible Uses the same request and response format as OpenAI's API, so the same code works with many servers llama-cpp-python's server, Claude's compatibility layer
curl A terminal tool that sends web requests curl http://localhost:8080/v1/models
Header Extra information sent with a request Content-Type: application/json
JSON A text format for data: "key": value {"model": "..."}
Model (in a request) Which LLM should answer "model": "qwen2.5-1.5b-instruct"
Prompt The text you give the model: instructions plus input "Summarize these notes in 5 bullets"
Messages The conversation sent to the model, each with a role system, user, assistant
System message The rules for the conversation "You summarize a student's class notes..."
User message The actual question or input "Class notes: ..."
Response What the API sends back choices[0].message.content
finish_reason Why the answer ended stop, length
max_tokens The longest answer allowed, in tokens "max_tokens": 60
Token The smallest piece of text a model reads and writes Ev + aporation
Tokenizer The part of a model that splits text into tokens; each model has its own /extras/tokenize
Usage How many tokens a request used, in and out "prompt_tokens": 29, "completion_tokens": 14
Context window How many tokens a model can look at at once: instructions + input + answer 4,096 tokens on our server
API key A secret string that identifies you to an API and decides who pays Authorization: Bearer <key>
401 HTTP status: not authenticated (wrong or missing key) {"detail":"Invalid API key"}
400 HTTP status: the request is wrong (e.g. too long) context_length_exceeded
429 HTTP status: too many requests; slow down Our rate limit, Claude's rate_limit_error
Client The program the user runs, here the browser The summarizer page
Server / back end The program on your computer or a host that the page talks to server.js
.env A file of secret settings read by the server LLM_API_KEY=...
.gitignore A list of files Git must never commit .env
View page source The browser view that shows a page's full file Line 9: the key
Streaming Sending the answer in small pieces as it is written "stream": true
Server-sent events (SSE) The data: ... line format used to stream data: [DONE]
Delta The next small piece of text in a stream {"content": " is"}
Temperature How random the model's word choices are 0 = same answer, 1.5 = varied
Deterministic Always giving the same output for the same input Temperature 0 is close, not guaranteed
Hallucination The model stating something false or not in the source "Precipitation ... into clouds"
MTok One million tokens; the unit in price lists $0.10 / MTok
Cost per token Price × tokens, input and output priced separately $0.000109 per summary on Haiku 5.5
Rate limit A cap on requests per time window 5 summaries per minute per IP
IP address The network address of the device making the request 127.0.0.1
Retry-After A response header saying how many seconds to wait Retry-After: 27
Accept edits An agent permission mode: it may write files without asking each time --permission-mode acceptEdits

Summary

  • Rules count characters; an LLM understands meaning. Your app reaches it through an API: an endpoint, a model, messages (system + user) and settings, answered with content, a finish_reason and usage in tokens.
  • An API key is a password and a wallet. Never put it in a web page: keep it in .env on your server, list .env in .gitignore, and let the page call your server.
  • The context window is in tokens, and tokens per character depend on the language: the AI's 3.5 characters per token was right for English and badly wrong for Hindi. Count with the model's tokenizer.
  • Streaming shows the answer as it's written; temperature sets randomness (low for facts); cost = tokens × price; a rate limit protects your bill.
  • The AI's real mistakes: a 50,000-character limit for a 4,096-token model, raw errors shown to users, and an English-only token guess. The model's real mistakes: facts mixed up in the summary. Test with real users' text and check answers against the source.

Next episode: an AI study buddy that answers only from your own notes, and shows where each answer came from.