Machine Learning Full Course · Episode 76. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and Hugging Face transformers 5.22.2 on a 2-core CPU, and the output shown is the real output. Models come from the Hugging Face Hub: distilbert/distilbert-base-uncased-finetuned-sst-2-english, google-bert/bert-base-uncased, openai-community/gpt2, google/vit-base-patch16-224. Data: stanfordnlp/sst2 (67,349 training and 872 validation sentences, counted) and the image huggingface/cats-image (two tabby cats on a pink sofa), both from the Hub.
The big idea: a transformer trained from scratch on 2,000 labelled reviews memorises them (train 94%, validation about 60%) because 2,000 sentences cannot teach a language. Pretraining teaches a model language from huge unlabelled text once; fine-tuning adapts it to your task with a little labelled data. This reuse is transfer learning. The Hugging Face pipeline API runs a pretrained model in one line (91.1% on the same 872 reviews). Inside it: an AutoTokenizer (text → sub-word ids + attention mask), an AutoModel (ids → context-aware vectors) and post-processing. BERT was pretrained by masked language modelling (fill the blanks), GPT by causal language modelling (next token). We fine-tuned BERT with Trainer on the same 2,000 reviews: 86.9%. The Vision Transformer (ViT) reads an image as 196 patches of 16 × 16 pixels.
Sources: Hugging Face, Pipeline tutorial ("a simple but powerful inference API that is readily available for a variety of machine learning tasks with any model from the Hugging Face Hub"; a pipeline runs preprocess → forward → postprocess; "Each task is configured to use a default pretrained model and preprocessor"). Model card, DistilBERT base uncased finetuned SST-2 (fine-tuned on SST-2; 91.3 accuracy on the dev set; the bert-base-uncased version reaches 92.7). Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT (masked LM: 15% of WordPiece tokens chosen, of which 80% become [MASK], 10% a random token, 10% unchanged; pretraining data BooksCorpus, 800M words, and English Wikipedia, 2,500M words; a 30,000-token WordPiece vocabulary; the final [CLS] vector is used for classification; BERT-Base 12 layers, 768 hidden, 12 heads, 110M parameters; fine-tuning results reproducible in at most 1 hour on a single Cloud TPU). Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al., An Image is Worth 16x16 Words (split an image into patches and feed the sequence of linear patch embeddings to a standard Transformer; a learnable [class] token like BERT's; ViT-Base: 12 layers, hidden size 768, 86M parameters; pretrained on ImageNet-21k, 21k classes and 14M images, or JFT, 303M images; transformers lack some inductive biases of CNNs, but "large scale training trumps inductive bias").
Words you need first
- Transformer, self-attention, causal mask (Episodes 74-75).
- Token: one unit of text the model reads: a word, part of a word, or a symbol.
- Embedding (Episode 72): a vector of numbers that represents a token.
- Overfitting (Episode 20): great on training data, poor on new data.
- Logits and softmax (Episode 23): raw scores and their conversion into probabilities.
- AdamW, learning rate schedule (Episodes 61-62).
- Self-supervised learning (Episode 1): the labels come from the data itself.
The story: 2,000 reviews are not enough
Your food-delivery app gets thousands of reviews a day; you want sentiment analysis: label each one positive or negative. You have 2,000 labelled sentences (from SST-2, the Stanford Sentiment Treebank, a public set of movie-review sentences). The obvious attempt is to train our own transformer, like Episode 75:
import torch, torch.nn as nn
from datasets import load_dataset
torch.manual_seed(76)
sst = load_dataset("stanfordnlp/sst2")
train = sst["train"].shuffle(seed=76).select(range(2000)) # 2,000 labelled reviews
val = sst["validation"] # 872 reviews it never sees
words = {w for s in train["sentence"] for w in s.split()}
vocab = {w: i + 2 for i, w in enumerate(sorted(words))} # 0 = pad, 1 = unknown
def encode(sents, T=40):
ids = [[vocab.get(w, 1) for w in s.split()][:T] for s in sents]
return torch.tensor([x + [0] * (T - len(x)) for x in ids])
Xtr, ytr = encode(train["sentence"]), torch.tensor(train["label"])
Xva, yva = encode(val["sentence"]), torch.tensor(val["label"])
class TinyTransformer(nn.Module): # built from scratch, like episode 75
def __init__(s, V, D=64):
super().__init__(); s.emb, s.pos = nn.Embedding(V, D), nn.Embedding(40, D)
layer = nn.TransformerEncoderLayer(D, 4, 4 * D, batch_first=True, norm_first=True)
s.enc, s.head = nn.TransformerEncoder(layer, 2), nn.Linear(D, 2)
def forward(s, x):
h = s.enc(s.emb(x) + s.pos(torch.arange(x.shape[1])), src_key_padding_mask=x == 0)
return s.head(h.masked_fill((x == 0)[..., None], 0).sum(1) / (x != 0).sum(1, keepdim=True))
m = TinyTransformer(len(vocab) + 2); opt = torch.optim.AdamW(m.parameters(), 1e-3)
print(f"{len(train)} reviews, {len(vocab):,} different words, {sum(p.numel() for p in m.parameters()):,} weights")
for epoch in range(1, 11):
for i in range(0, 2000, 32):
loss = nn.functional.cross_entropy(m(Xtr[i:i + 32]), ytr[i:i + 32])
opt.zero_grad(); loss.backward(); opt.step()
if epoch % 2 == 0:
with torch.no_grad():
tr = (m(Xtr).argmax(1) == ytr).float().mean(); va = (m(Xva).argmax(1) == yva).float().mean()
print(f"epoch {epoch:2}: train accuracy {tr:.1%} validation accuracy {va:.1%}")
unk = sum(w not in vocab for s in val["sentence"] for w in s.split()) / sum(len(s.split()) for s in val["sentence"])
print(f"validation words never seen in training: {unk:.0%}")
2000 reviews, 4,655 different words, 400,706 weights
epoch 2: train accuracy 66.5% validation accuracy 54.8%
epoch 4: train accuracy 75.2% validation accuracy 54.7%
epoch 6: train accuracy 93.0% validation accuracy 59.3%
epoch 8: train accuracy 94.3% validation accuracy 55.5%
epoch 10: train accuracy 90.8% validation accuracy 59.9%
validation words never seen in training: 17%
Why it failed: it memorised the training set (94%) but stays near 55-60% on new reviews (a coin flip is 50%), and 17% of the words in new reviews never appeared in training. From 2,000 sentences it would have to learn English and sentiment.
Real-life example: hire a graduate, don't raise a baby
| Hiring for a job | Building a model |
|---|---|
| Raise a baby and teach it to read before any work | Training from scratch: learn English from 2,000 reviews |
| Hire a graduate who already reads; someone else paid for school | Pretraining: a lab trains on billions of words |
| One week of job training on your company's rules | Fine-tuning: a little more training on your labelled data |
Pretraining, fine-tuning and Hugging Face
- Pretraining: learning general knowledge (here, language) from a huge amount of unlabelled data. Expensive, done once. BERT read BooksCorpus (about 800M words) and English Wikipedia (about 2,500M words).
- Fine-tuning: start from the pretrained weights and train a little more on your labelled data. BERT's paper: at most one hour on one Cloud TPU (Google's AI chip) per task.
- Transfer learning: reusing what a model learned on one task for another.
- Hugging Face Hub: a free online library of pretrained models and datasets. transformers: the Python library that downloads and runs them.
The pipeline API
from transformers import pipeline
clf = pipeline("sentiment-analysis",
model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
reviews = ["The biryani was amazing, I will order again!",
"Delivery was late and the food was cold.",
"Not bad at all.",
"The movie was not good."]
for r, out in zip(reviews, clf(reviews)):
print(f"{out['label']:8} {out['score']:.3f} {r}")
POSITIVE 1.000 The biryani was amazing, I will order again!
NEGATIVE 1.000 Delivery was late and the food was cold.
POSITIVE 0.999 Not bad at all.
NEGATIVE 1.000 The movie was not good.
The score is the model's confidence (0 to 1). It handles negation: "Not bad at all" is positive, "not good" negative. The same 872 validation reviews the scratch model failed on:
from datasets import load_dataset
from transformers import pipeline
val = load_dataset("stanfordnlp/sst2")["validation"] # the same 872 test reviews
clf = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
preds = clf(list(val["sentence"]), batch_size=32)
right = sum((p["label"] == "POSITIVE") == bool(y) for p, y in zip(preds, val["label"]))
print(f"pretrained + fine-tuned by someone else: {right}/{len(val)} = {right / len(val):.1%}")
pretrained + fine-tuned by someone else: 794/872 = 91.1%
Our 91.1% matches the model card's 91.3 closely. (This model was fine-tuned on SST-2's 67,349 training sentences.)
What the pipeline does
- Preprocess: the tokenizer turns text into numbers.
- Forward: the model turns the numbers into scores (logits).
- Postprocess: softmax and the label names turn scores into
POSITIVE 0.999.
AutoTokenizer.from_pretrained(name) loads the tokenizer the model was trained with; AutoModel.from_pretrained(name) reads the model's configuration file and builds the right model class. "Auto" means: give a name, and it picks the class for you.
Tokenizers: sub-words, ids, masks and padding
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
print("vocabulary size:", tok.vocab_size)
print(tok.tokenize("Unbelievably tasty biryani!"))
enc = tok(["Chai is great", "The samosa was unbelievably good"],
padding=True, return_tensors="pt")
print("input_ids:\n", enc["input_ids"])
print("attention_mask:\n", enc["attention_mask"])
print(tok.convert_ids_to_tokens(enc["input_ids"][0]))
vocabulary size: 30522
['un', '##bel', '##ie', '##va', '##bly', 'ta', '##sty', 'bi', '##rya', '##ni', '!']
input_ids:
tensor([[ 101, 15775, 2072, 2003, 2307, 102, 0, 0, 0, 0,
0, 0],
[ 101, 1996, 3520, 8820, 2001, 4895, 8671, 2666, 3567, 6321,
2204, 102]])
attention_mask:
tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
['[CLS]', 'cha', '##i', 'is', 'great', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']
- Sub-word tokenization (WordPiece for BERT): rare words are split into known pieces;
##means "continues the previous word". No word is ever completely unknown. - input_ids: the token numbers. [CLS] (101) starts every input; [SEP] (102) separates or ends; [PAD] (0) fills shorter rows.
- Padding: adding [PAD] tokens so every row in a batch has the same length (a tensor must be a rectangle). Truncation: cutting texts that are too long.
- attention_mask: 1 = a real token, 0 = padding to ignore.
Mistake: batching without padding
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
batch = ["Chai is great", "The samosa was unbelievably good"]
try:
enc = tok(batch, return_tensors="pt") # two different lengths, no padding
except ValueError as e:
print("ValueError:", e)
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
Fix: tok(batch, padding=True, truncation=True, return_tensors="pt").
Models: contextual vectors
import torch
from transformers import AutoTokenizer, AutoModel
name = "google-bert/bert-base-uncased"
tok, model = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name)
enc = tok("I deposited cash at the bank", return_tensors="pt")
with torch.no_grad():
out = model(**enc)
print("weights:", f"{sum(p.numel() for p in model.parameters()):,}")
print("tokens:", tok.convert_ids_to_tokens(enc["input_ids"][0]))
print("last_hidden_state:", tuple(out.last_hidden_state.shape))
def bank(sentence): # the vector BERT gives the word "bank"
e = tok(sentence, return_tensors="pt")
i = tok.convert_ids_to_tokens(e["input_ids"][0]).index("bank")
with torch.no_grad(): return model(**e).last_hidden_state[0, i]
cos = torch.nn.functional.cosine_similarity
money, river = bank("I deposited cash at the bank"), bank("We sat on the river bank")
print(f"'bank' money vs river, after BERT: {cos(money, river, dim=0):.3f}")
weights: 109,482,240
tokens: ['[CLS]', 'i', 'deposited', 'cash', 'at', 'the', 'bank', '[SEP]']
last_hidden_state: (1, 8, 768)
'bank' money vs river, after BERT: 0.692
BERT-base has 109,482,240 weights (the paper rounds to 110M). The last hidden state has shape (sentences, tokens, 768). The vector for "bank" next to "cash" and next to "river" has cosine 0.692, not 1: these are contextual embeddings, different for the same word in different sentences (Episode 72's word2vec gave one vector per word).
Real-life example: fill in the blanks vs finish the sentence
| School exercise | Pretraining task |
|---|---|
| A "fill in the blanks" worksheet: read both sides of the gap | Masked language modelling (BERT) |
| A phone keyboard suggesting your next word: only what came before | Causal language modelling (GPT) |
Masked language modelling (BERT)
Choose 15% of the tokens; the model must predict them from both sides. Of the chosen tokens, 80% are replaced by [MASK], 10% by a random token, 10% left unchanged (so the model cannot rely on seeing [MASK]). The hidden word is the label, so no human labelling is needed: self-supervised. The final vector of [CLS] becomes a summary of the sentence, used for classification.
from transformers import pipeline
fill = pipeline("fill-mask", model="google-bert/bert-base-uncased")
for guess in fill("The chai was too [MASK] to drink.", top_k=5):
print(f"{guess['token_str']:10} {guess['score']:.3f}")
hot 0.147
cold 0.093
good 0.091
thick 0.088
much 0.082
Causal language modelling (GPT-2)
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "openai-community/gpt2"
tok, model = AutoTokenizer.from_pretrained(name), AutoModelForCausalLM.from_pretrained(name)
print("weights:", f"{sum(p.numel() for p in model.parameters()):,}")
ids = tok("The best thing about Indian street food is", return_tensors="pt")
with torch.no_grad():
nxt = model(**ids).logits[0, -1].softmax(-1).topk(5)
print("next-token guesses:", [(tok.decode(i), round(p.item(), 3)) for p, i in zip(*nxt)])
out = model.generate(**ids, max_new_tokens=25, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0]))
weights: 124,439,808
next-token guesses: [(' that', 0.527), (' the', 0.139), (' its', 0.059), (' it', 0.051), (' how', 0.026)]
The best thing about Indian street food is that it's not just a food thing. It's a lifestyle thing. It's a lifestyle that's not just a food
GPT-2 (OpenAI) in its smallest size has 124,439,808 weights. do_sample=False is greedy decoding (always the most likely token), which tends to repeat ("a lifestyle thing ... a lifestyle"). Sampling (Episode 75) avoids that.
Mistake: the wrong tokenizer (no error!)
from transformers import AutoTokenizer
gpt_tok = AutoTokenizer.from_pretrained("openai-community/gpt2") # the WRONG tokenizer
bert_tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
ids = gpt_tok("Hello, wonderful weather today")["input_ids"]
print("GPT-2 token ids:", ids)
print("what BERT reads:", bert_tok.convert_ids_to_tokens(ids))
print("right tokenizer:", bert_tok.tokenize("Hello, wonderful weather today"))
GPT-2 token ids: [15496, 11, 7932, 6193, 1909]
what BERT reads: ['##burgh', '[unused10]', 'scattered', '1890', '王']
right tokenizer: ['hello', ',', 'wonderful', 'weather', 'today']
Every GPT-2 id here also exists in BERT's vocabulary, so nothing crashes; BERT simply reads nonsense. Always load the tokenizer and the model from the same name.
Fine-tuning BERT for classification with Trainer
AutoModelForSequenceClassification: pretrained BERT plus a new classification head (a linear layer from the [CLS] vector's 768 numbers to 2 classes). The head starts random.- Small learning rate (2e-5): adjust what BERT knows instead of erasing it.
Trainer: Hugging Face's ready-made training loop: batching, the AdamW optimiser, a learning-rate schedule (linear decay by default, visible in the log), logging and evaluation.TrainingArgumentsholds its settings.
import numpy as np, time
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
Trainer, TrainingArguments, set_seed)
set_seed(76)
name = "google-bert/bert-base-uncased" # pretrained, NOT fine-tuned
sst = load_dataset("stanfordnlp/sst2")
train = sst["train"].shuffle(seed=76).select(range(2000)) # the same 2,000 reviews
val = sst["validation"]
tok = AutoTokenizer.from_pretrained(name)
def prep(batch): return tok(batch["sentence"], truncation=True, padding="max_length", max_length=64)
train, val = train.map(prep, batched=True), val.map(prep, batched=True)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2) # + a new head
def accuracy(p): return {"accuracy": float((np.argmax(p.predictions, -1) == p.label_ids).mean())}
args = TrainingArguments("bert-sst2", num_train_epochs=1, per_device_train_batch_size=16,
learning_rate=2e-5, eval_strategy="no", report_to="none",
logging_steps=25, save_strategy="no")
trainer = Trainer(model=model, args=args, train_dataset=train, eval_dataset=val,
compute_metrics=accuracy)
print("before fine-tuning:", trainer.evaluate()["eval_accuracy"])
t0 = time.time(); trainer.train()
print(f"fine-tuned in {time.time() - t0:.0f} s")
print("after fine-tuning :", trainer.evaluate()["eval_accuracy"])
before fine-tuning: 0.48853211009174313
{'loss': '0.6852', 'grad_norm': '4.34', 'learning_rate': '1.616e-05', 'epoch': '0.2'}
{'loss': '0.546', 'grad_norm': '6.715', 'learning_rate': '1.216e-05', 'epoch': '0.4'}
{'loss': '0.3925', 'grad_norm': '22.21', 'learning_rate': '8.16e-06', 'epoch': '0.6'}
{'loss': '0.3986', 'grad_norm': '7.713', 'learning_rate': '4.16e-06', 'epoch': '0.8'}
{'loss': '0.3275', 'grad_norm': '17.67', 'learning_rate': '1.6e-07', 'epoch': '1'}
fine-tuned in 1103 s
after fine-tuning : 0.8692660550458715
(Trainer also prints metric dictionaries for each evaluate() and a training summary; they are left out above. It took 1,103 seconds on 2 CPU cores that were busy with other work.)
| Same 2,000 training reviews, same 872 validation reviews | Validation accuracy |
|---|---|
| Tiny transformer from scratch, 10 epochs | about 55-60% |
| BERT before fine-tuning (random head) | 48.9% |
| BERT fine-tuned by us, 1 epoch | 86.9% |
| DistilBERT fine-tuned by others on 67,349 reviews | 91.1% |
Real-life example: a photo as a mosaic
| A tiled mosaic floor | A Vision Transformer |
|---|---|
| The picture is made of small square tiles | The image is cut into 16 × 16 patches |
| Read the tiles row by row, like words | Each patch → one vector → a transformer |
The Vision Transformer (ViT)
From Dosovitskiy et al., "An Image is Worth 16x16 Words": split a 224 × 224 image into 16 × 16 patches (a 14 × 14 grid = 196 patches); flatten each patch (16 × 16 × 3 colours = 768 numbers) and pass it through a linear layer (the patch embedding); put a learnable [class] token in front, as in BERT, and add position embeddings. ViT has fewer built-in assumptions about images than a CNN (inductive biases), so it needs a lot of pretraining data (ImageNet-21k, 14M images; JFT, 303M).
import torch
img = torch.rand(3, 224, 224) # a colour photo: 3 channels, 224 x 224 pixels
P = 16 # patch size
patches = img.unfold(1, P, P).unfold(2, P, P) # cut into a 14 x 14 grid
print("grid of patches:", tuple(patches.shape[1:3]))
patches = patches.permute(1, 2, 0, 3, 4).reshape(-1, 3 * P * P)
print("patches as tokens:", tuple(patches.shape), "-> 196 'words' of 768 numbers")
embed = torch.nn.Linear(3 * P * P, 768) # patch embedding
cls = torch.zeros(1, 768) # the [CLS] token, like BERT
tokens = torch.cat([cls, embed(patches)])
print("sequence for the transformer:", tuple(tokens.shape))
grid of patches: (14, 14)
patches as tokens: (196, 768) -> 196 'words' of 768 numbers
sequence for the transformer: (197, 768)
from transformers import pipeline
vit = pipeline("image-classification", model="google/vit-base-patch16-224")
print("weights:", f"{sum(p.numel() for p in vit.model.parameters()):,}")
for guess in vit("cats.jpg", top_k=3):
print(f"{guess['score']:.3f} {guess['label']}")
weights: 86,567,656
0.934 Egyptian cat
0.047 tabby, tabby cat
0.013 tiger cat
86.6 million weights, close to the paper's 86M for ViT-Base. The labels are ImageNet's class names; all three top guesses are cats.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Training a big model from scratch on 2,000 rows | train ~94%, validation ~60% | can't learn a language from that | fine-tune a pretrained model |
tok(batch, return_tensors="pt") |
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' ... |
rows of different lengths | padding=True, truncation=True |
| GPT-2 tokenizer with BERT | no error; BERT reads '##burgh', '[unused10]', 'scattered', '1890', '王' |
different vocabularies | load both from the same name |
| Large learning rate when fine-tuning | accuracy collapses; pretrained knowledge erased | steps too large | small rates such as 2e-5 to 5e-5 |
| Greedy generation | repetition: "a lifestyle thing ... a lifestyle" | always the top token | sampling, temperature (Episode 75) |
| Trusting the score as truth | "Portion size was okay" → POSITIVE 1.000 | confidence is not correctness | check uncertain and borderline cases |
Mini project: label today's reviews
from collections import Counter
from transformers import pipeline
clf = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
reviews = ["Food arrived hot and on time", "The paneer was rubbery", "Rider was very polite",
"Waited 90 minutes, never ordering again", "Portion size was okay", "Best dosa in town!"]
results = clf(reviews)
for r, out in zip(reviews, results):
flag = " <- check" if out["score"] < 0.9 else ""
print(f"{out['label']:8} {out['score']:.3f} {r}{flag}")
print(Counter(o["label"] for o in results))
POSITIVE 0.999 Food arrived hot and on time
NEGATIVE 0.999 The paneer was rubbery
POSITIVE 0.998 Rider was very polite
NEGATIVE 0.987 Waited 90 minutes, never ordering again
POSITIVE 1.000 Portion size was okay
POSITIVE 0.999 Best dosa in town!
Counter({'POSITIVE': 4, 'NEGATIVE': 2})
Notice "Portion size was okay" got POSITIVE with 1.000: the model is overconfident on a neutral review, because SST-2 here has only two classes. A real system would add a neutral class or human review.
Practice
1. How does BERT's tokenizer split Indian words?
Solution
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
for word in ["chai", "samosa", "paneer", "Bengaluru", "machine", "learning", "unhappiness"]:
print(f"{word:12} -> {tok.tokenize(word)}")
chai -> ['cha', '##i']
samosa -> ['sam', '##osa']
paneer -> ['pan', '##eer']
Bengaluru -> ['bengal', '##uru']
machine -> ['machine']
learning -> ['learning']
unhappiness -> ['un', '##ha', '##pp', '##iness']
Common English words are single tokens; Indian words become two pieces, because the vocabulary was built from English books and Wikipedia.
2. Try fill-mask on your own sentences.
Solution
from transformers import pipeline
fill = pipeline("fill-mask", model="google-bert/bert-base-uncased")
for s in ["I drink [MASK] every morning.", "The train was [MASK] by two hours.", "She is a [MASK] at the hospital."]:
print(s, "->", [g["token_str"] for g in fill(s, top_k=3)])
I drink [MASK] every morning. -> ['it', 'coffee', 'this']
The train was [MASK] by two hours. -> ['delayed', 'extended', 'postponed']
She is a [MASK] at the hospital. -> ['nurse', 'patient', 'doctor']
"She is a [MASK] at the hospital" → nurse first: a pretrained model also learns the biases of its training text. Check for them before using a model on people.
3. Fine-tune with 500 and with 5,000 reviews and compare. Hint
Change select(range(2000)) to 500 or 5000 in the fine-tuning script. Expect accuracy to rise with more data, and the time per epoch to grow in proportion. On a CPU, try distilbert/distilbert-base-uncased (about 66M weights) to save time.
4. Classify your own photo with ViT. Hint
vit("my_photo.jpg", top_k=3). The model knows ImageNet's 1,000 classes; a photo of something outside them (a samosa, say) gets the nearest class it knows.
Quick quiz
The model sees "I paid the [MASK] at the canteen" and must guess the hidden word using words on both sides. Which pretraining task is this?
Answer
Masked language modelling, the way BERT was pretrained. No human labels: the hidden word is the answer.
Interview questions
- Pretraining vs fine-tuning? Pretraining learns general knowledge from huge unlabelled data once; fine-tuning adapts the weights to a specific task with a small labelled set.
- Why does fine-tuning beat training from scratch on small data? The model already knows language; it only learns the task, so it overfits far less.
- What do
input_idsandattention_maskmean? Token numbers, and which positions are real (1) vs padding (0). - Why sub-word tokenization? A fixed vocabulary that can still represent any word, with no unknown words.
- MLM vs causal LM? MLM predicts hidden tokens from both sides (BERT, good for understanding); causal LM predicts the next token from the left (GPT, good for generation).
- What does
AutoModelForSequenceClassificationadd? A new, randomly initialised linear head on the [CLS] representation. - Why a small learning rate for fine-tuning? Large steps destroy the pretrained weights (catastrophic forgetting).
- How does ViT turn an image into tokens? Cut into 16 × 16 patches, flatten, linear projection, add a [class] token and position embeddings.
Glossary
| Term | Meaning, with an example |
|---|---|
| Sentiment analysis | Labelling text as positive or negative: "late and cold" → NEGATIVE |
| SST-2 | Stanford Sentiment Treebank: movie-review sentences labelled positive/negative |
| Pretraining | Learning general knowledge from huge unlabelled data, once (BERT: books + Wikipedia) |
| Fine-tuning | Training a pretrained model a little more on your labelled data |
| Transfer learning | Reusing what a model learned on one task for another |
| Hugging Face Hub | A free online library of pretrained models and datasets |
| transformers (library) | Hugging Face's Python library to download and run models |
| Pipeline API | pipeline(task, model): preprocess → model → postprocess in one call |
| Preprocess / forward / postprocess | Tokenize; run the model; turn scores into labels |
| AutoTokenizer | Loads the right tokenizer for a model name |
| AutoModel | Builds the right model class from a model name's config |
| Config | A model's settings file (layers, sizes, vocabulary size) |
| Tokenizer | Turns text into token ids and back |
| Sub-word / WordPiece | Splitting rare words into known pieces: biryani → bi, ##rya, ##ni |
| Vocabulary | All tokens a tokenizer knows: 30,522 for BERT-base-uncased |
| input_ids | The token numbers: [101, 15775, 2072, ...] |
| [CLS], [SEP], [PAD], [MASK] | Start/summary token, separator, padding, hidden-word token |
| Padding / truncation | Fill short rows with [PAD] / cut long ones, so a batch is a rectangle |
| attention_mask | 1 for real tokens, 0 for padding |
| Last hidden state | The model's output vectors: shape (batch, tokens, 768) |
| Contextual embedding | A token vector that depends on the sentence: bank (money) vs bank (river), cosine 0.692 |
| Masked language modelling (MLM) | Hide 15% of tokens, predict them from both sides |
| Causal language modelling | Predict the next token from the previous ones |
| Greedy decoding | Always pick the most likely next token; can repeat |
| BERT | Bidirectional Encoder Representations from Transformers; encoder-only, MLM |
| GPT-2 | OpenAI's decoder-only language model; smallest has 124M weights |
| DistilBERT | A smaller, faster version of BERT |
| Classification head | A new linear layer on top of a pretrained model: 768 → 2 |
| AutoModelForSequenceClassification | BERT + a classification head on [CLS] |
| Trainer / TrainingArguments | Hugging Face's ready-made training loop and its settings |
| Vision Transformer (ViT) | A transformer that reads an image as a sequence of patches |
| Patch | A small square of the image: 16 × 16 pixels × 3 colours = 768 numbers |
| Patch embedding | A linear layer turning a flattened patch into a token vector |
| [class] token | A learnable token in front of the patches whose output classifies the image |
| Inductive bias | Built-in assumptions of a model, e.g. a CNN assumes nearby pixels matter together |
Summary
- From scratch on 2,000 reviews: 94% train, about 60% validation. Pretraining + fine-tuning (transfer learning) fixes it.
pipeline()= tokenizer → model → postprocess; a ready fine-tuned DistilBERT scored 91.1% on the 872 reviews.- Tokenizers give sub-word
input_ids, special tokens and anattention_mask; batch withpadding=True. - BERT learned by masked language modelling, GPT-2 by causal language modelling; BERT's vectors are contextual.
- Fine-tuning BERT with Trainer on the same 2,000 reviews: 48.9% → 86.9%. Use the same name for tokenizer and model, and a small learning rate.
- ViT: 196 patches + [class] = 197 tokens; it called the cats an "Egyptian cat" with 0.934.
Next: Episode 77, generative models: autoencoders, VAEs, GANs and diffusion models.