These are the notes for AI episode 13. Last episode you turned text into token IDs. But an ID is only a label: it says nothing about meaning. Here you watch IDs and one-hot vectors fail, then invent embeddings: lists of numbers where similar meanings sit close together. You measure closeness with cosine similarity, do king − man + woman, learn embeddings from a word's neighbours (the idea behind word2vec), meet the embedding layer inside every language model, and use embeddings for semantic search. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).
Labels aren't meaning: roll numbers
| In a classroom | In a language model |
|---|---|
| Roll numbers, given in alphabetical order | Token IDs, given by the tokenizer |
| Roll 21 and 22 sit together on the list but may have nothing in common | Close IDs say nothing about meaning |
| A student's profile: likes cricket, good at maths, plays guitar | A list of features that describes what something is |
Problem 1: IDs as meaning
GPT-2's real token IDs (from episode 12's tokenizer) for " cat", " dog", " car" and " kitten":
# GPT-2's real token IDs (from episode 12's tokenizer)
ids = {"cat": 3797, "dog": 3290, "car": 1097, "kitten": 42143}
def gap(a, b):
return abs(ids[a] - ids[b])
print("cat <-> dog: ", gap("cat", "dog"))
print("cat <-> car: ", gap("cat", "car"))
print("cat <-> kitten:", gap("cat", "kitten"))
Output:
cat <-> dog: 507
cat <-> car: 2700
cat <-> kitten: 38346
By ID, a cat is closer to a car than to a kitten. IDs reflect how the tokenizer was built, not meaning; maths on them would teach a network nonsense.
Problem 2: one-hot vectors
A vector is just a list of numbers. A one-hot vector has a 0 for every word in the vocabulary and a single 1 for this word. math.dist gives the straight-line distance between two vectors.
import math
words = ["cat", "dog", "car", "kitten"]
def one_hot(word): # a list of 0s with a single 1
return [1 if w == word else 0 for w in words]
for w in words:
print(f"{w:7}", one_hot(w))
print("cat <-> kitten:", round(math.dist(one_hot("cat"), one_hot("kitten")), 3))
print("cat <-> car: ", round(math.dist(one_hot("cat"), one_hot("car")), 3))
Output:
cat [1, 0, 0, 0]
dog [0, 1, 0, 0]
car [0, 0, 1, 0]
kitten [0, 0, 0, 1]
cat <-> kitten: 1.414
cat <-> car: 1.414
Every word is exactly 1.414 (√2) from every other word: fair, but no meaning. And with GPT-2's vocabulary, each one-hot vector would be 50,257 numbers long, all zeros but one.
The fix: describe each word with a few meaningful numbers
- Pick features: is it an animal? a vehicle? small? a pet?
cat = [0.9, 0.0, 0.6, 0.9]: a vector of four meaningful numbers.- The dimension is how many numbers the vector has (here, 4).
- Similar meaning → similar numbers → the vectors sit close together.
Measuring closeness: cosine similarity
Two friends both pointing north-east agree, even if one has a longer arm. One pointing north and the other east (a right angle) don't agree at all. Cosine similarity compares the direction of two vectors, not their length:
- Dot product: multiply matching numbers and add them up.
- Length (magnitude):
math.hypot(*v), the square root of the sum of squares. - cosine = dot product ÷ (length of a × length of b): 1 = same direction, 0 = unrelated (right angle), −1 = opposite.
import math
# animal vehicle small pet
vec = {"cat": [0.9, 0.0, 0.6, 0.9],
"kitten": [0.9, 0.0, 0.8, 0.9],
"dog": [0.9, 0.0, 0.3, 0.7],
"car": [0.0, 0.9, 0.2, 0.0],
"bus": [0.0, 1.0, 0.0, 0.0]}
def cosine(a, b): # 1 = same direction, 0 = unrelated
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.hypot(*a) * math.hypot(*b))
for other in ["kitten", "dog", "car", "bus"]:
print(f"cat vs {other:6}: {cosine(vec['cat'], vec[other]):.2f}")
Output:
cat vs kitten: 0.99
cat vs dog : 0.98
cat vs car : 0.09
cat vs bus : 0.00
The numbers finally agree with meaning. A vector of numbers that captures what a token means is an embedding.
Word arithmetic: king − man + woman
import math
# royal male female person
vec = {"king": [0.9, 0.9, 0.0, 1.0],
"queen": [0.9, 0.0, 0.9, 1.0],
"man": [0.0, 0.9, 0.0, 1.0],
"woman": [0.0, 0.0, 0.9, 1.0],
"prince": [0.7, 0.9, 0.0, 1.0],
"boy": [0.0, 0.9, 0.0, 0.8]}
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.hypot(*a) * math.hypot(*b))
target = [k - m + w for k, m, w in zip(vec["king"], vec["man"], vec["woman"])]
print("king - man + woman =", [round(x, 2) for x in target])
best = max((w for w in vec if w not in ("king", "man", "woman")),
key=lambda w: cosine(vec[w], target))
for w in ["queen", "prince", "boy"]:
print(f"{w:6} {cosine(vec[w], target):.2f}")
print("closest word:", best)
Output:
king - man + woman = [0.9, 0.0, 0.9, 1.0]
queen 1.00
prince 0.66
boy 0.41
closest word: queen
In this hand-made toy, the answer lands exactly on queen. In 2013, Tomas Mikolov and colleagues at Google reported the same thing in learned word vectors that no human designed: vector("King") − vector("Man") + vector("Woman") is closest to vector("Queen") (Efficient Estimation of Word Representations in Vector Space). Directions in embedding space carry meaning (royal, male → female).
The next problem: who writes all these numbers?
- GPT-2's vocabulary has 50,257 tokens; every one needs a vector.
- Real embeddings have hundreds or thousands of numbers each: GPT-3 uses 12,288 per token (from its paper).
- Which features would you even choose? Nobody can list them. The model must learn them.
Known by its neighbours
Read "I drank a hot cup of kahwa". Never heard of kahwa? Drank, hot, cup: it's a drink, something like chai. Words that appear in similar contexts (the words around them) tend to have similar meanings. This is the distributional idea (often called the distributional hypothesis).
import math
from collections import Counter, defaultdict
sentences = ["raju drinks hot chai", "neha drinks hot coffee", "raju drinks cold coffee",
"neha drinks cold chai", "raju rides a bike", "neha rides a scooter",
"raju parks the bike", "neha washes the scooter", "chai is hot", "coffee is hot"]
context = defaultdict(Counter) # which words appear near each word?
for s in sentences:
words = s.split()
for i, w in enumerate(words):
for j in range(max(0, i - 2), min(len(words), i + 3)):
if j != i:
context[w][words[j]] += 1
def cosine(a, b):
dot = sum(a[k] * b[k] for k in a)
size = lambda v: math.sqrt(sum(x * x for x in v.values()))
return dot / (size(a) * size(b))
print("chai's neighbours:", dict(context["chai"]))
for a, b in [("chai", "coffee"), ("bike", "scooter"), ("chai", "bike")]:
print(f"{a} vs {b}: {cosine(context[a], context[b]):.2f}")
Output:
chai's neighbours: {'drinks': 2, 'hot': 2, 'cold': 1, 'is': 1}
chai vs coffee: 1.00
bike vs scooter: 0.75
chai vs bike: 0.00
Each word's neighbour counts are its vector (a co-occurrence vector). The program never knew what chai or coffee are; it found them similar (1.00) from the company they keep, while chai and bike share no neighbours (0.00).
word2vec: learned embeddings at scale
- 2013, word2vec (Mikolov, Chen, Corrado, Dean, Google): a small network trained to predict a word from its neighbours (CBOW) or the neighbours from a word (skip-gram).
- Their paper reports learning high-quality word vectors from a 1.6 billion word data set in less than a day.
- The vectors are the network's weights, learned by backpropagation (episode 9): embeddings are learned parameters.
The embedding layer: a learned lookup table
The first layer of every language model is an embedding layer: a table with one row per token in the vocabulary, each row that token's vector. It starts random and is learned during training. Embedding text is just a lookup.
import random
random.seed(1)
vocab = ["river", "bank", "loan", "the", "near", "a"]
dim = 4 # numbers per token
table = [[round(random.uniform(-1, 1), 2) for _ in range(dim)]
for _ in vocab] # starts random, learned by training
def embed(sentence): # an embedding layer = a lookup
return [table[vocab.index(w)] for w in sentence.split()]
for s in ["near the river bank", "a bank loan"]:
i = s.split().index("bank")
print(f"{s:20} bank -> {embed(s)[i]}")
print(f"GPT-3's table: 50,257 tokens x 12,288 numbers = {50257 * 12288:,}")
Output:
near the river bank bank -> [-0.01, -0.1, 0.3, 0.58]
a bank loan bank -> [-0.01, -0.1, 0.3, 0.58]
GPT-3's table: 50,257 tokens x 12,288 numbers = 617,558,016
- GPT-3's table: 50,257 tokens × 12,288 numbers ≈ 617.6 million parameters, just for the first layer (GPT-3 uses GPT-2's tokenizer, whose vocabulary is 50,257).
- The catch: "bank" gets the same vector in "near the river bank" and "a bank loan". A lookup gives each token one meaning, whatever its neighbours say. Episode 14 (attention) fixes this with contextual embeddings.
Using embeddings: semantic search
import math
# tiny hand-made embeddings: cheap phone food travel
emb = {"cheap": [0.9, 0.0, 0.0, 0.0], "budget": [0.8, 0.0, 0.0, 0.1],
"phone": [0.0, 0.9, 0.0, 0.0], "mobile": [0.0, 0.8, 0.0, 0.0],
"biryani": [0.0, 0.0, 0.9, 0.0], "train": [0.0, 0.0, 0.0, 0.9],
"under": [0.1, 0.0, 0.0, 0.0], "10000": [0.3, 0.1, 0.0, 0.0],
"best": [0.0, 0.0, 0.1, 0.0], "in": [0.0, 0.0, 0.0, 0.0],
"hyderabad": [0.0, 0.0, 0.3, 0.3],
"tickets": [0.1, 0.0, 0.0, 0.8], "to": [0.0, 0.0, 0.0, 0.1], "goa": [0.0, 0.0, 0.1, 0.8]}
docs = ["budget mobile under 10000", "best biryani in hyderabad", "train tickets to goa"]
query = "cheap phone"
def keyword_match(q, d):
return len(set(q.split()) & set(d.split()))
def meaning(text): # average the word vectors
vs = [emb[w] for w in text.split()]
return [sum(col) / len(vs) for col in zip(*vs)]
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.hypot(*a) * math.hypot(*b))
for d in docs:
k, m = keyword_match(query, d), cosine(meaning(query), meaning(d))
print(f"{d:28} keywords: {k} meaning: {m:.2f}")
Output:
budget mobile under 10000 keywords: 0 meaning: 0.99
best biryani in hyderabad keywords: 0 meaning: 0.00
train tickets to goa keywords: 0 meaning: 0.03
Keyword search finds nothing (0 shared words). Comparing meanings (here, the average of the word vectors) finds "budget mobile under 10000" at 0.99. Finding things by meaning instead of exact words is semantic search; it powers modern search, recommendations, and AI assistants that read your documents (RAG, episode 24). Systems that store millions of embeddings and find the nearest ones fast are called vector databases.
Embeddings everywhere
- Sentences and documents can get one vector each (like our averaged
meaning(), but learned much better): that's how a question finds the right paragraph. - Photos: models trained on pictures with captions place a photo of a dog on a beach near the words "dog on a beach", so you can search a gallery without tags.
- Songs, products, videos get embeddings from what people watch and buy together: "people who liked this also liked…" is often "find the nearest vectors".
- Vector databases store millions of vectors and find the nearest ones fast.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Token ID | a token's label number (episode 12) | " cat" = 3797 in GPT-2 |
| Vector | a list of numbers | [0.9, 0.0, 0.6, 0.9] |
| Dimension | how many numbers a vector has | 4 in our demo; 12,288 in GPT-3 |
| One-hot vector | all zeros except a single 1 | cat = [1, 0, 0, 0] |
| Feature | one measurable property | "is it a pet?" |
| Distance | how far apart two vectors are | math.dist(a, b) |
| Dot product | multiply matching numbers, add them up | sum(x * y for x, y in zip(a, b)) |
| Length (magnitude) | the size of a vector | math.hypot(*a) |
| Cosine similarity | how alike two vectors' directions are, −1 to 1 | cat vs kitten = 0.99 |
| Embedding | a learned vector that captures a token's meaning | the row for "cat" |
| Embedding space | the space all the vectors live in | similar words cluster together |
| Word analogy | vector arithmetic that follows meaning | king − man + woman ≈ queen |
| Context | the words around a word | "drinks", "hot" around chai |
| Distributional idea | words in similar contexts have similar meanings | kahwa ≈ chai |
| Co-occurrence | how often words appear near each other | chai's neighbours |
| word2vec | 2013 Google method for learning word vectors | CBOW, skip-gram |
| Embedding layer | the lookup table at the start of a model | GPT-3: 50,257 × 12,288 |
| Lookup | fetching a row by its index | table[vocab.index(w)] |
| Semantic search | finding by meaning, not exact words | "cheap phone" → "budget mobile" |
| Keyword search | finding by shared words | 0 matches above |
| Vector database | stores embeddings, finds the nearest ones fast | used in RAG apps |
| Contextual embedding | a vector that changes with the surrounding words | episode 14: attention |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Comparing vectors of different lengths | math.dist([1, 0, 0], [1, 0]) |
ValueError: both points must have the same number of dimensions |
| An all-zero vector in cosine | "chai": [0.0, 0.0, 0.0, 0.0], then cosine(...) |
ZeroDivisionError: float division by zero |
| Looking up a word that isn't in the table | emb["priya"] |
KeyError: 'priya' |
| Treating token IDs as amounts | abs(ids["cat"] - ids["kitten"]) |
no error; 38346, meaningless |
Note on zip |
zip([1, 2, 3], [1, 2]) in a dot product |
no error; silently ignores the extra number |
Fixes: one system, one dimension; check for a zero length before dividing; embed tokens (episode 12), not whole words; never do maths on IDs; check len(a) == len(b).
Mini project: a tiny "find similar" app
Use the FEAT vectors and write most_similar(word) that returns the other words sorted by cosine similarity, most similar first. Then add five words of your own (e.g. scooter, train, puppy, lion, truck) with your own feature values and check the rankings make sense.
Practice
- Add
"puppy": [0.9, 0.0, 0.8, 0.8]to the features demo. Is it closest to kitten, dog or cat? - Add three sentences about lassi to the learning demo (
"raju drinks cold lassi","neha drinks sweet lassi","lassi is cold") and compare lassi with chai, coffee and bike. - In the search demo, add
"travel": [0.0, 0.0, 0.0, 1.0]and search for"goa travel". - How many parameters is an embedding table of 100,000 tokens × 1,024 numbers? How many MB at 16-bit?
Solutions
1.
puppy vs kitten: 1.00
puppy vs dog : 0.94
puppy vs cat : 0.99
puppy vs car : 0.12
With our hand-picked features, puppy lands next to kitten (both small young pets). Real learned embeddings would also pull puppy toward dog, because puppy and dog share contexts ("bark", "walk").
2.
chai's neighbours: {'drinks': 2, 'hot': 2, 'cold': 1, 'is': 1}
lassi vs chai: 0.70
lassi vs coffee: 0.70
lassi vs bike: 0.00
Three sentences are enough to place lassi with the drinks, far from bike.
3.
budget mobile under 10000 keywords: 0 meaning: 0.07
best biryani in hyderabad keywords: 0 meaning: 0.28
train tickets to goa keywords: 1 meaning: 1.00
4. 100,000 × 1,024 = 102,400,000 parameters; at 2 bytes each, about 205 MB.
Quick quiz 1
Cat vs bus has cosine similarity 0.00. What does 0.00 mean? A: exactly the same meaning · B: opposite meaning · C: unrelated directions · D: one word is missing
Answer
C: unrelated. The vectors are at a right angle. 1 means the same direction; −1 means opposite.
Quick quiz 2
A vocabulary of 32,000 tokens and 4,096 numbers per token. How many parameters in the embedding table? A: 36,096 · B: 4,096 · C: about 13 million · D: about 131 million
Answer
D: about 131 million. 32,000 × 4,096 = 131,072,000.
Interview questions
- What is an embedding? A dense vector of learned numbers representing a token (or a sentence, image…) so that similar meanings are close together.
- Why not use one-hot vectors? They're huge (vocabulary-sized), sparse, and every pair is equally distant, so they carry no similarity.
- What is cosine similarity and why use it over plain distance? The cosine of the angle between two vectors (dot product ÷ product of lengths). It compares direction and ignores length, which often reflects frequency rather than meaning.
- How are embeddings learned? As weights of a network trained on a prediction task (word2vec: predict a word from its neighbours or vice versa; LLMs: predict the next token), using backpropagation.
- What is an embedding layer? A vocabulary × dimension lookup table at the input of a model; its parameter count is vocabulary size × dimension.
- What's the limitation of static word embeddings? One vector per word regardless of context ("bank"). Transformers produce contextual embeddings with attention.
Summary
- Token IDs are labels: cat is "closer" to car than to kitten.
- One-hot vectors: every word 1.414 from every other; no meaning.
- Embeddings: vectors of meaningful numbers; similar meanings sit close.
- Cosine similarity compares directions: cat vs kitten 0.99, cat vs bus 0.00.
- king − man + woman ≈ queen: directions carry meaning (word2vec, 2013).
- Embeddings are learned from neighbours; inside a model they're an embedding layer (GPT-3: ≈ 617.6 million numbers).
- Semantic search finds meaning, not keywords. But "bank" gets one vector everywhere.
- Next episode: attention and the Transformer.