Machine Learning Full Course · Episode 28. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. All data (SMS messages, lengths, digit counts) is made up for teaching.
The big idea: to classify a message, ask "how likely is this message if it were spam, and how likely if it were ham?" and combine that with how common each class is (Bayes' theorem). Estimating the probability of a whole message is impossible, so naive Bayes makes one bold shortcut: it treats every word as independent once the class is known and simply multiplies per-word probabilities. Text becomes numbers with bag-of-words (word counts), unseen words are handled with Laplace smoothing, and the products are computed as sums of logs to avoid underflow. It comes in three flavours (multinomial, Bernoulli, Gaussian), trains by counting, and is a generative model: it can even write new spam.
Words you need first
- Classification: predicting a category (spam / ham). Text classification: classification where the input is text.
- Spam: unwanted bulk or scam messages. Ham: the normal messages you want (the opposite of spam).
- Probability, conditional probability P(A | B): how likely A is, given that B is known to be true (Episode 9).
- Feature: an input column; label: the answer to predict (Episode 1).
- Hyperparameter: a setting we choose, not learned by the model (Episode 8).
- Normal (Gaussian) distribution, Bernoulli distribution: the bell curve, and the yes/no distribution (Episode 10).
The problem: keyword rules can't keep up
Your phone receives scam SMS ("You WON Rs 50,000! Claim your prize now") and normal messages ("Mom: dinner at 8, bring curd"). The obvious filter is a list of keywords.
inbox = ["You WON Rs 50,000! Claim your prize now", "Mom: dinner at 8, bring curd",
"Congrats! U are selected for a free gift card", "We won the match yaar!",
"Your OTP for login is 4821", "Urgent: KYC pending, click link to claim reward"]
truth = ["spam", "ham", "spam", "ham", "ham", "spam"]
def rule(msg): # hand-written rules
return "spam" if any(w in msg.lower() for w in ["won", "prize", "lottery"]) else "ham"
for m, t in zip(inbox, truth):
p = rule(m)
print("ok " if p == t else "BAD", p.ljust(4), "|", m)
Output:
ok spam | You WON Rs 50,000! Claim your prize now
ok ham | Mom: dinner at 8, bring curd
BAD ham | Congrats! U are selected for a free gift card
BAD spam | We won the match yaar!
ok ham | Your OTP for login is 4821
BAD ham | Urgent: KYC pending, click link to claim reward
Three of six are wrong:
- "selected for a free gift card": no keyword, so the scam gets through.
- "We won the match yaar!": the innocent word won blocks a friend's message.
- "KYC pending, click link to claim reward": a scam with none of the keywords.
Why rules fail: scammers change the words, and a single word decides everything. Instead we want to learn from labelled examples which words point to spam and which to ham.
Bayes' theorem for spam
From Episode 9:
P(spam | message) = P(message | spam) × P(spam) / P(message)
- Prior P(spam): how common spam is before reading anything (e.g. 30% of messages).
- Likelihood P(message | spam): how likely this exact message would be if it were spam.
- Posterior P(spam | message): the answer after reading the message.
- Evidence P(message): the same for every class, so to compare classes we can ignore it.
The obvious attempt: count whole messages
from collections import Counter
train = [("free prize claim now", "spam"), ("dinner at 8 tonight", "ham"),
("claim free gift now", "spam"), ("call me after class", "ham")]
seen = Counter(msg for msg, _ in train)
seen_spam = Counter(msg for msg, lab in train if lab == "spam")
new = "claim your free prize"
print("times seen:", seen[new], " as spam:", seen_spam[new])
print("P(spam | message) =", seen_spam[new] / seen[new])
Output:
times seen: 0 as spam: 0
Traceback (most recent call last):
File "main.py", line 8, in <module>
print("P(spam | message) =", seen_spam[new] / seen[new])
~~~~~~~~~~~~~~~^~~~~~~~~~~
ZeroDivisionError: division by zero
The new message was never seen before, word for word, so its count is 0 and the division fails with ZeroDivisionError: division by zero. Almost every new message is new text, so we must count something smaller: words.
Bag-of-words and CountVectorizer
Real-life example: at the sabzi market, whatever order you pick vegetables in, they end up in one bag; the bag is described by counts ("2 tomatoes, 1 onion, 0 potatoes"). The vendor's list of everything he sells is the vocabulary.
- Bag-of-words: represent a text by how many times each word appears, throwing away word order.
- Token: one unit of text after splitting (here: a word). Tokenisation: splitting text into tokens.
- Vocabulary: every distinct token seen in the training texts. Each vocabulary word is one feature (column).
- Count vector: one number per vocabulary word for one message.
CountVectorizer(scikit-learn):fitlearns the vocabulary,transformturns texts into count vectors. It lowercases text, and its default token pattern keeps words of 2 or more letters/digits.
from sklearn.feature_extraction.text import CountVectorizer
msgs = ["free prize claim free", "dinner at home", "claim your prize now"]
vec = CountVectorizer()
X = vec.fit_transform(msgs) # learn the vocabulary, then count
print(vec.get_feature_names_out())
print(X.toarray())
print(vec.transform(["FREE free pizza at home"]).toarray())
Output:
['at' 'claim' 'dinner' 'free' 'home' 'now' 'prize' 'your']
[[0 1 0 2 0 0 1 0]
[1 0 1 0 1 0 0 0]
[0 1 0 0 0 1 1 1]]
[[1 0 0 2 1 0 0 0]]
- The vocabulary is sorted alphabetically:
freeis column 4, so "free prize claim free" has a 2 there. - "FREE free pizza at home" counts
freetwice (lowercased) and simply dropspizza: words not in the vocabulary are ignored. - The result is a sparse matrix (mostly zeros);
.toarray()shows it as a normal array. - Fit the vectorizer on training data only (Episode 16), so the test set cannot influence the vocabulary.
- Weakness: order is lost, "not good" = "good not". (n-grams,
CountVectorizer(ngram_range=(1, 2)), add word pairs as extra features.)
The naive assumption
Even with word counts, P(count vector | spam) for a 10,000-word vocabulary has astronomically many possible values. The shortcut is to look at one word at a time.
Real-life example: a detective at a railway station guessing who is going to a wedding judges each clue on its own (new clothes, a gift box, a sweets box) and multiplies the evidence, pretending the clues are unrelated, although new clothes and a gift usually go together. That is naive, yet it works.
- Independent events: knowing one tells you nothing about the other: P(A and B) = P(A) × P(B).
- Conditional independence: independent once the class is known: P(w₁, w₂ | spam) = P(w₁ | spam) × P(w₂ | spam).
- Naive Bayes assumption: all features are conditionally independent given the class. It's almost never exactly true for words, which is why it's called naive.
With it, for each class c:
score(c) = P(c) × P(w₁ | c) × P(w₂ | c) × … × P(wₙ | c) (∝ P(c | message))
P(word | c) = (times the word appears in class-c messages) / (total words in class-c messages)
Predict the class with the bigger score.
Naive Bayes from scratch
from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
("call me after class", "ham"), ("free for dinner tonight", "ham")]
classes = ["spam", "ham"]
words = {c: Counter() for c in classes} # word counts per class
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def score(msg, c, alpha):
p = sum(1 for _, lab in train if lab == c) / len(train) # prior P(c)
total = sum(words[c].values())
for w in msg.split(): # naive: multiply per word
p *= (words[c][w] + alpha) / (total + alpha * len(vocab))
return p
for msg in ["claim prize now", "free dinner tonight", "claim prize tonight"]:
for alpha in [0, 1]:
s = {c: score(msg, c, alpha) for c in classes}
win = "tie" if s["spam"] == s["ham"] else max(s, key=s.get)
print(f"{msg:<19} a={alpha} spam {s['spam']:.6f} ham {s['ham']:.6f} {win}")
Output:
claim prize now a=0 spam 0.003472 ham 0.000000 spam
claim prize now a=1 spam 0.000820 ham 0.000023 spam
free dinner tonight a=0 spam 0.000000 ham 0.001157 ham
free dinner tonight a=1 spam 0.000068 ham 0.000410 ham
claim prize tonight a=0 spam 0.000000 ham 0.000000 tie
claim prize tonight a=1 spam 0.000205 ham 0.000068 spam
Countercounts the words of each class;scoremultiplies the prior by each word's probability.- With
alpha=0(plain counting): "claim prize now" → spam, "free dinner tonight" → ham. - But "claim prize tonight" gives spam 0 and ham 0: a tie. Why: tonight never appeared in the 3 spam messages (12 words), so P(tonight | spam) = 0/12 = 0, and anything × 0 = 0. For ham, claim and prize never appeared. One unseen word vetoes all other evidence.
Laplace smoothing
Real-life example: a new dhaba on a food app with no reviews isn't a zero-star dhaba; we just don't know yet.
P(word | c) = (count + α) / (total words in c + α × vocabulary size)
- Laplace smoothing: α = 1, as if every word had been seen one extra time in every class.
- Additive (Lidstone) smoothing: any α > 0.
- The
α × vocabulary sizein the denominator keeps the probabilities of all words summing to 1. - α is a hyperparameter:
MultinomialNB(alpha=1.0)is the default.
In the output above, with alpha=1, "claim prize tonight" becomes spam 0.000205 vs ham 0.000068: spam wins by about 3×, and the other two messages keep their answers.
Work in logs: underflow
import math
probs = [0.01] * 400 # 400 words, each with probability 0.01
p = 1.0
for q in probs: p *= q
print("multiply:", p)
print("add logs:", sum(math.log(q) for q in probs))
print("smallest float:", 5e-324, " half of it:", 5e-324 / 2)
Output:
multiply: 0.0
add logs: -1842.0680743952364
smallest float: 5e-324 half of it: 0.0
- Underflow: a result smaller than the smallest positive float (about 5 × 10⁻³²⁴) is rounded to 0.0. 0.01⁴⁰⁰ = 10⁻⁸⁰⁰ underflows.
- Fix: log(a × b × c) = log a + log b + log c. Add log-probabilities instead of multiplying probabilities. The class with the bigger log score is the same class with the bigger score, because log is increasing.
- scikit-learn stores the learned log-probabilities in
feature_log_prob_(the trailing underscore means "learned during fit"). This sum is the log-likelihood from Episode 12.
Naive Bayes in scikit-learn: MultinomialNB
2,000 made-up SMS messages (30% spam) built from spammy, normal and common words with some overlap.
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28) # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n): # each word: 50% main pool, 15% other, 35% common
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400) # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
rule = [any(w in t.split() for w in ["won", "prize", "lottery"]) for t in t_te]
print(f"keyword rule accuracy {np.mean(np.array(rule) == y_te):.3f}")
model = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0)).fit(t_tr, y_tr)
print(f"naive Bayes accuracy {model.score(t_te, y_te):.3f}")
print(model.predict_proba(["claim your free cash now", "bring the cash home tonight"]).round(3))
vec, nb = model[0], model[1]
top = np.argsort(nb.feature_log_prob_[1] - nb.feature_log_prob_[0])[-5:]
print("most spammy words:", vec.get_feature_names_out()[top])
Output:
keyword rule accuracy 0.728
naive Bayes accuracy 0.886
[[0.046 0.954]
[0.963 0.037]]
most spammy words: ['bonus' 'offer' 'cash' 'prize' 'click']
- Keyword rule: 0.728. Naive Bayes: 0.886, learned by counting words.
predict_probacolumns are inclasses_order:[ham (0), spam (1)]. "claim your free cash now" → 95.4% spam; "bring the cash home tonight" → 96.3% ham, even though it contains cash.- The most spammy words (highest log P(word | spam) − log P(word | ham)) were found from data, not written by hand.
- Multinomial naive Bayes models a message as words drawn one after another from the class's word distribution, so counts matter.
- Training = counting in one pass: very fast, even for millions of messages.
- Its probabilities are often too extreme (overconfident) because correlated words are counted as independent evidence; rankings are good, exact percentages less so. Calibrate (Episode 25) if you need trustworthy probabilities.
Bernoulli naive Bayes
Real-life example: an attendance register records only present/absent, and absent is information too; a tally counts how many times.
- Bernoulli naive Bayes: every vocabulary word is a yes/no feature (present or absent). It uses P(word present | class) and also P(word absent | class) for every missing word.
- Use
CountVectorizer(binary=True)(or letBernoulliNB(binarize=0.0)do it).
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28) # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n): # each word: 50% main pool, 15% other, 35% common
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400) # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB
from sklearn.pipeline import make_pipeline
multi = make_pipeline(CountVectorizer(), MultinomialNB()).fit(t_tr, y_tr)
bern = make_pipeline(CountVectorizer(binary=True), BernoulliNB()).fit(t_tr, y_tr)
print(f"Multinomial (counts) {multi.score(t_te, y_te):.3f}")
print(f"Bernoulli (present/absent) {bern.score(t_te, y_te):.3f}")
msg = ["free free free free call mom"]
print("multinomial P(spam):", multi.predict_proba(msg)[0, 1].round(3))
print("bernoulli P(spam):", bern.predict_proba(msg)[0, 1].round(3))
Output:
Multinomial (counts) 0.886
Bernoulli (present/absent) 0.876
multinomial P(spam): 0.779
bernoulli P(spam): 0.079
- Overall accuracy is close (0.886 vs 0.876).
- "free free free free call mom": multinomial says 77.9% spam because free counts four times; Bernoulli says 7.9%: it sees free once, sees the ham words call and mom, and treats every missing spammy word (no claim, no prize, …) as evidence for ham.
- Bernoulli often suits short texts; multinomial suits longer texts where counts carry information.
Gaussian naive Bayes
Real-life example: heights of class 3 and class 10 students each follow their own bell curve; for a given height, ask which class's curve fits better.
- Gaussian naive Bayes: for numeric features, it assumes each feature follows a normal distribution within each class, and learns a mean and a variance per feature per class (
theta_,var_).
import numpy as np
from sklearn.naive_bayes import GaussianNB
rng = np.random.default_rng(28) # made-up numbers, not words
spam = np.c_[rng.normal(120, 25, 300), rng.normal(9, 3, 300)] # length, digits
ham = np.c_[rng.normal(45, 20, 700), rng.normal(1, 1.5, 700)]
X = np.clip(np.r_[spam, ham], 0, None).round(); y = np.array([1] * 300 + [0] * 700)
gnb = GaussianNB().fit(X, y)
for c, name in [(0, "ham "), (1, "spam")]: # [length, digits]
print(name, "mean", gnb.theta_[c].round(1), " variance", gnb.var_[c].round(1))
for m in [[100, 6], [60, 1], [80, 4]]:
print(m, "-> P(spam) =", f"{gnb.predict_proba([m])[0, 1]:.3f}")
Output:
ham mean [45.9 1.2] variance [385. 1.5]
spam mean [119.6 8.8] variance [612.5 8.7]
[100, 6] -> P(spam) = 1.000
[60, 1] -> P(spam) = 0.000
[80, 4] -> P(spam) = 0.394
- Spam (made up here) is long and has many digits; ham is short with few digits.
- [100 characters, 6 digits] → almost certainly spam; [60, 1] → ham; [80, 4] sits between: 39.4%.
The three flavours
| Flavour | Features | Learns per class | Typical use |
|---|---|---|---|
MultinomialNB |
counts (word counts) | P(word) from counts + α | longer texts, documents |
BernoulliNB |
0/1 (present/absent) | P(word present) | short texts, binary features |
GaussianNB |
real numbers | mean and variance of each feature | numeric data |
(scikit-learn also has ComplementNB, a multinomial variant that often works better on imbalanced text, and CategoricalNB for categorical features.)
Generative vs discriminative models
Real-life example: a cook knows how biryani and pulao are made, so she can recognise either and also make a new plate; a food critic only learns to tell them apart.
- Generative model: learns how each class generates data, P(x | class), plus P(class), then classifies with Bayes' theorem. Naive Bayes, Gaussian mixture models, and today's generative AI models are generative.
- Discriminative model: learns P(class | x), or just the decision boundary, directly. Logistic regression, kNN, decision trees, SVMs.
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28) # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n): # each word: 50% main pool, 15% other, 35% common
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400) # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
vec = CountVectorizer().fit(t_tr); A_tr, A_te = vec.transform(t_tr), vec.transform(t_te)
for n in [20, 60, 200, 1500]: # train on only the first n messages
nb = MultinomialNB().fit(A_tr[:n], y_tr[:n])
lr = LogisticRegression(max_iter=1000).fit(A_tr[:n], y_tr[:n])
print(f"{n:>4} messages: naive Bayes {nb.score(A_te, y_te):.3f} logistic {lr.score(A_te, y_te):.3f}")
p = np.exp(nb.feature_log_prob_[1]) # P(word | spam), learned from 1,500 messages
for _ in range(3): # a generative model can WRITE new spam
print("fake spam:", " ".join(rng.choice(vec.get_feature_names_out(), 6, p=p)))
Output:
20 messages: naive Bayes 0.740 logistic 0.702
60 messages: naive Bayes 0.850 logistic 0.790
200 messages: naive Bayes 0.856 logistic 0.856
1500 messages: naive Bayes 0.886 logistic 0.878
fake spam: win urgent for at kyc won
fake spam: gift me prize urgent gift the
fake spam: link click bonus win the reward
- With only 20-60 training messages, naive Bayes beats logistic regression (0.740 vs 0.702, 0.850 vs 0.790): its strong assumption lets it learn quickly from little data.
- With 1,500 messages they are close (0.886 vs 0.878). With lots of data, discriminative models often catch up or win.
- Because it learned P(word | spam), naive Bayes can sample new spam-like messages. They're word salad (order was never learned), but clearly spammy. Logistic regression cannot do this.
Naive Bayes at a glance
| Question | Answer |
|---|---|
| Training | Count (or compute means/variances) per class: one pass, very fast |
| Prediction | log prior + Σ log P(feature | class), pick the max |
| Key assumption | Features are conditionally independent given the class |
| Needs scaling? | No (Gaussian NB models each feature's own scale) |
| Strengths | Fast, tiny memory, strong with little data, great text baseline, handles many features |
| Weaknesses | Overconfident probabilities, can't model feature interactions, word order lost with bag-of-words |
| Main hyperparameter | alpha (smoothing) for Multinomial/Bernoulli; var_smoothing for Gaussian |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Hand-written keyword rules | 3 of 6 wrong; accuracy 0.728 | Scammers change words; one word decides | Learn from labelled data |
| Counting whole messages | ZeroDivisionError: division by zero |
New text was never seen | Bag-of-words, per-word probabilities |
No smoothing (alpha=0) |
spam 0.000000 ham 0.000000 -> tie |
One unseen word makes the product 0 | Laplace smoothing, alpha=1 |
| Multiplying hundreds of probabilities | multiply: 0.0 |
Float underflow | Add log-probabilities |
Fitting CountVectorizer on all data |
Optimistic test score | Test vocabulary leaks into training | Put it in a pipeline, fit on train only |
Trusting predict_proba as exact |
0.99 on unsure messages | Correlated words counted as independent | Calibrate (Episode 25) |
GaussianNB on word counts |
Poor accuracy | Counts aren't bell-shaped | MultinomialNB / BernoulliNB |
Negative values into MultinomialNB |
ValueError: Negative values in data passed to MultinomialNB (input X). |
Counts can't be negative | Don't standardise counts; use non-negative features |
The last error, reproduced:
import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.preprocessing import StandardScaler
counts = np.array([[2, 0, 1], [0, 3, 0], [1, 1, 4]])
X = StandardScaler().fit_transform(counts) # standardised counts: some are negative
MultinomialNB().fit(X, [1, 0, 1])
Output:
Traceback (most recent call last):
... (library frames) ...
ValueError: Negative values in data passed to MultinomialNB (input X).
Mini project: naive Bayes in NumPy
Write a MyMultinomialNB class that works in log space and check it against scikit-learn on the 2,000 messages.
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28) # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n): # each word: 50% main pool, 15% other, 35% common
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400) # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
vec = CountVectorizer().fit(t_tr)
A, B = vec.transform(t_tr).toarray(), vec.transform(t_te).toarray()
class MyMultinomialNB: # naive Bayes in NumPy, in log space
def __init__(self, alpha=1.0): self.alpha = alpha
def fit(self, X, y):
self.classes_ = np.unique(y)
self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
counts = np.array([X[y == c].sum(0) for c in self.classes_]) + self.alpha
self.log_prob_ = np.log(counts / counts.sum(1, keepdims=True)) # log P(word | class)
return self
def predict(self, X):
scores = X @ self.log_prob_.T + self.log_prior_ # count × log P, summed, + log prior
return self.classes_[scores.argmax(1)]
mine = MyMultinomialNB().fit(A, y_tr).predict(B)
sk = MultinomialNB().fit(A, y_tr).predict(B)
print(f"my accuracy {np.mean(mine == y_te):.3f} sklearn {np.mean(sk == y_te):.3f} agree on {np.mean(mine == sk):.1%}")
Output:
my accuracy 0.886 sklearn 0.886 agree on 100.0%
The matrix product X @ log_prob_.T computes, for every message and class, Σ count(word) × log P(word | class) in one step. Our 15-line class agrees with scikit-learn on 100% of the test messages.
Practice
1. Make the from-scratch version use logs (and α = 1).
Solution
import math
from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
("call me after class", "ham"), ("free for dinner tonight", "ham")]
classes = ["spam", "ham"]
words = {c: Counter() for c in classes}
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def log_score(msg, c, alpha=1):
s = math.log(sum(1 for _, lab in train if lab == c) / len(train)) # log prior
total = sum(words[c].values())
for w in msg.split():
s += math.log((words[c][w] + alpha) / (total + alpha * len(vocab))) # add, don't multiply
return s
for msg in ["claim prize now", "free dinner tonight", "claim prize tonight"]:
s = {c: log_score(msg, c) for c in classes}
print(f"{msg:<20} log spam {s['spam']:.3f} log ham {s['ham']:.3f} -> {max(s, key=s.get)}")
Output:
claim prize now log spam -7.106 log ham -10.690 -> spam
free dinner tonight log spam -9.591 log ham -7.799 -> ham
claim prize tonight log spam -8.493 log ham -9.591 -> spam
Same winners as the multiplied version with α = 1, but the scores are now ordinary negative numbers that can't underflow.
2. Try alpha = 0.01, 0.1, 1, 10, 100. What changes?
Solution
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28) # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n): # each word: 50% main pool, 15% other, 35% common
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400) # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
for a in [0.01, 0.1, 1, 10, 100]:
m = make_pipeline(CountVectorizer(), MultinomialNB(alpha=a)).fit(t_tr, y_tr)
p = m.predict_proba(["claim your free cash now"])[0, 1]
print(f"alpha {a:<5} accuracy {m.score(t_te, y_te):.3f} P(spam | 'claim your free cash now') {p:.3f}")
Output:
alpha 0.01 accuracy 0.888 P(spam | 'claim your free cash now') 0.956
alpha 0.1 accuracy 0.888 P(spam | 'claim your free cash now') 0.956
alpha 1 accuracy 0.886 P(spam | 'claim your free cash now') 0.954
alpha 10 accuracy 0.884 P(spam | 'claim your free cash now') 0.939
alpha 100 accuracy 0.874 P(spam | 'claim your free cash now') 0.761
Accuracy barely moves until α gets large; a big α pulls every word's probability towards the same value, so the evidence gets weaker (P(spam) for the spammy message falls to 0.761 at α = 100).
3. Add the spam message "lottery won claim tonight" to the 6 training messages. Classify "lottery tonight", "dinner tonight" and "won lottery".
Solution
from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
("call me after class", "ham"), ("free for dinner tonight", "ham"),
("lottery won claim tonight", "spam")] # the new message
classes = ["spam", "ham"]
words = {c: Counter() for c in classes}
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def score(msg, c, alpha=1):
p = sum(1 for _, lab in train if lab == c) / len(train)
total = sum(words[c].values())
for w in msg.split(): p *= (words[c][w] + alpha) / (total + alpha * len(vocab))
return p
for msg in ["lottery tonight", "dinner tonight", "won lottery"]:
s = {c: score(msg, c) for c in classes}
print(f"{msg:<16} spam {s['spam']:.5f} ham {s['ham']:.5f} -> {max(s, key=s.get)}")
Output:
lottery tonight spam 0.00198 ham 0.00143 -> spam
dinner tonight spam 0.00099 ham 0.00429 -> ham
won lottery spam 0.00198 ham 0.00048 -> spam
tonight now appears in spam once, so "lottery tonight" becomes spam, while "dinner tonight" stays ham.
4. BernoulliNB vs MultinomialNB on long messages (30-59 words).
Solution
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n, lo, hi):
pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
return [" ".join(word() for _ in range(rng.integers(lo, hi))) for _ in range(n)]
for lo, hi in [(4, 10), (30, 60)]: # short vs long messages
texts = sms(SPAM, HAM, 600, lo, hi) + sms(HAM, SPAM, 1400, lo, hi); y = np.array([1] * 600 + [0] * 1400)
a, b, ya, yb = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
mu = make_pipeline(CountVectorizer(), MultinomialNB()).fit(a, ya).score(b, yb)
be = make_pipeline(CountVectorizer(binary=True), BernoulliNB()).fit(a, ya).score(b, yb)
print(f"{lo}-{hi - 1} words: multinomial {mu:.3f} bernoulli {be:.3f}")
Output:
4-9 words: multinomial 0.886 bernoulli 0.876
30-59 words: multinomial 0.998 bernoulli 1.000
On these made-up long messages both become nearly perfect, because every message has many words of evidence. On real long documents with repeated words, multinomial is usually the safer default; Bernoulli is often good for short texts.
5. By hand: P(spam) = 0.3, P(ham) = 0.7. P(free | spam) = 0.2, P(free | ham) = 0.02. A one-word message "free": which class wins?
Solution
spam: 0.3 × 0.2 = 0.06; ham: 0.7 × 0.02 = 0.014. Spam wins. Normalised: P(spam | free) = 0.06 / (0.06 + 0.014) ≈ 0.81.
Quick quiz
- With α = 0, P(win | spam) = 0.10, P(cash | spam) = 0.08, P(tonight | spam) = 0. The spam score of "win cash tonight" is? 0: anything times 0 is 0.
- Which flavour for "number of times each word appears"? MultinomialNB.
- Which flavour uses the absence of a word as evidence? BernoulliNB.
- Why add logs instead of multiplying? Multiplying many small probabilities underflows to 0.0.
- Is naive Bayes generative or discriminative? Generative: it learns P(x | class).
Interview questions
- Why is naive Bayes called naive? It assumes all features are conditionally independent given the class, which is rarely true (e.g. "free" and "prize" co-occur).
- Why does it still work well? Classification only needs the correct class to have the highest score; even with wrong probabilities, the ranking is often right. It also has few parameters, so it learns well from little data.
- What is Laplace smoothing and why is it needed? Adding α (1) to every count so that a word never seen in a class doesn't get probability 0 and veto all other evidence.
- Why use log-probabilities? To avoid floating-point underflow when multiplying many small numbers; log turns products into sums and keeps the same winner.
- Multinomial vs Bernoulli vs Gaussian NB? Counts; binary presence/absence (absence counts as evidence); continuous features with a normal distribution per class.
- Generative vs discriminative? Generative models learn P(x | y) and P(y) (naive Bayes); discriminative models learn P(y | x) or the boundary directly (logistic regression). Generative models learn fast with little data and can sample; discriminative usually win with lots of data.
- Are naive Bayes probabilities well calibrated? Usually not: they're pushed towards 0 and 1. Use calibration (Platt scaling or isotonic regression) if probabilities matter.
- Does naive Bayes need feature scaling? No.
- What does bag-of-words lose? Word order and context ("not good" = "good not").
- How would you handle a word never seen in training at prediction time? CountVectorizer ignores it (it's not in the vocabulary); smoothing handles words seen in one class but not the other.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Text classification | Sorting texts into categories | SMS → spam or ham |
| Spam / ham | Unwanted bulk or scam message / normal wanted message | "You WON Rs 50,000" / "Mom: dinner at 8" |
| Bayes' theorem | P(A | B) = P(B | A) P(A) / P(B): flips a conditional probability | P(spam | message) from P(message | spam) |
| Prior | Probability of a class before seeing the data | P(spam) = 0.3 |
| Likelihood | Probability of the data if a class were true | P(message | spam) |
| Posterior | Probability of a class after seeing the data | P(spam | message) |
| Bag-of-words | Text as word counts, order thrown away | "free prize free" → free: 2, prize: 1 |
| Token | One unit of text after splitting | "claim", "prize" |
| Vocabulary | All distinct tokens seen in training | 8 words in the BoW example |
| Count vector | One count per vocabulary word for a text | [0 1 0 2 0 0 1 0] |
| CountVectorizer | scikit-learn tool that builds the vocabulary and count vectors | CountVectorizer().fit_transform(msgs) |
| Sparse matrix | A matrix stored without its many zeros | output of fit_transform |
| Independent | Knowing one event says nothing about the other | two coin tosses |
| Conditional independence | Independent once the class is known | P(free, prize | spam) = P(free | spam) P(prize | spam) |
| Naive Bayes | Bayes' theorem + the conditional-independence assumption | the spam filter |
| Laplace smoothing | Add 1 to every count so nothing has probability 0 | (0 + 1) / (12 + 1 × 16) for tonight in spam |
| Additive smoothing | Same with any α > 0 | MultinomialNB(alpha=0.1) |
| Underflow | A number too small for a float becomes 0.0 | 0.01⁴⁰⁰ → 0.0 |
| Log-probability | The logarithm of a probability; sums replace products | Σ log 0.01 = −1842.07 |
| Multinomial naive Bayes | Naive Bayes on counts | MultinomialNB on word counts |
| Bernoulli naive Bayes | Naive Bayes on present/absent features; absence is evidence | BernoulliNB + binary=True |
| Gaussian naive Bayes | Naive Bayes with a bell curve per numeric feature per class | length and digits of a message |
| Generative model | Learns how each class generates data; can sample new data | naive Bayes writing fake spam |
| Discriminative model | Learns the boundary / P(class | x) directly | logistic regression |
| Overconfident | Probabilities closer to 0 or 1 than the truth | predict_proba 0.99 on a borderline message |
Summary
- Keyword rules fail; learn from labelled messages with Bayes' theorem.
- Turn text into numbers with bag-of-words (
CountVectorizer). - The naive assumption (words conditionally independent given the class) turns an impossible estimate into simple per-word counts.
- Laplace smoothing stops unseen words from zeroing everything; logs stop underflow.
- Multinomial for counts, Bernoulli for present/absent, Gaussian for numbers.
- Naive Bayes is generative: fast to train, strong with little data, can sample; discriminative models often win with lots of data.
- Next episode: decision trees from scratch.