SyntaxPilot
Courses/Machine Learning/Lesson 28
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 28 of 90

Naive Bayes

generative vs discriminative modelsnaive conditional independence assumptionGaussian naive Bayesmultinomial naive BayesBernoulli naive BayesLaplace (additive) smoothingbag-of-wordsCountVectorizertext classification

Machine Learning Full Course · Episode 28. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. All data (SMS messages, lengths, digit counts) is made up for teaching.

The big idea: to classify a message, ask "how likely is this message if it were spam, and how likely if it were ham?" and combine that with how common each class is (Bayes' theorem). Estimating the probability of a whole message is impossible, so naive Bayes makes one bold shortcut: it treats every word as independent once the class is known and simply multiplies per-word probabilities. Text becomes numbers with bag-of-words (word counts), unseen words are handled with Laplace smoothing, and the products are computed as sums of logs to avoid underflow. It comes in three flavours (multinomial, Bernoulli, Gaussian), trains by counting, and is a generative model: it can even write new spam.

Words you need first

  • Classification: predicting a category (spam / ham). Text classification: classification where the input is text.
  • Spam: unwanted bulk or scam messages. Ham: the normal messages you want (the opposite of spam).
  • Probability, conditional probability P(A | B): how likely A is, given that B is known to be true (Episode 9).
  • Feature: an input column; label: the answer to predict (Episode 1).
  • Hyperparameter: a setting we choose, not learned by the model (Episode 8).
  • Normal (Gaussian) distribution, Bernoulli distribution: the bell curve, and the yes/no distribution (Episode 10).

The problem: keyword rules can't keep up

Your phone receives scam SMS ("You WON Rs 50,000! Claim your prize now") and normal messages ("Mom: dinner at 8, bring curd"). The obvious filter is a list of keywords.

inbox = ["You WON Rs 50,000! Claim your prize now", "Mom: dinner at 8, bring curd",
         "Congrats! U are selected for a free gift card", "We won the match yaar!",
         "Your OTP for login is 4821", "Urgent: KYC pending, click link to claim reward"]
truth = ["spam", "ham", "spam", "ham", "ham", "spam"]
def rule(msg):                              # hand-written rules
    return "spam" if any(w in msg.lower() for w in ["won", "prize", "lottery"]) else "ham"
for m, t in zip(inbox, truth):
    p = rule(m)
    print("ok " if p == t else "BAD", p.ljust(4), "|", m)

Output:

ok  spam | You WON Rs 50,000! Claim your prize now
ok  ham  | Mom: dinner at 8, bring curd
BAD ham  | Congrats! U are selected for a free gift card
BAD spam | We won the match yaar!
ok  ham  | Your OTP for login is 4821
BAD ham  | Urgent: KYC pending, click link to claim reward

Three of six are wrong:

  • "selected for a free gift card": no keyword, so the scam gets through.
  • "We won the match yaar!": the innocent word won blocks a friend's message.
  • "KYC pending, click link to claim reward": a scam with none of the keywords.

Why rules fail: scammers change the words, and a single word decides everything. Instead we want to learn from labelled examples which words point to spam and which to ham.

Bayes' theorem for spam

From Episode 9:

P(spam | message) = P(message | spam) × P(spam) / P(message)
  • Prior P(spam): how common spam is before reading anything (e.g. 30% of messages).
  • Likelihood P(message | spam): how likely this exact message would be if it were spam.
  • Posterior P(spam | message): the answer after reading the message.
  • Evidence P(message): the same for every class, so to compare classes we can ignore it.

The obvious attempt: count whole messages

from collections import Counter
train = [("free prize claim now", "spam"), ("dinner at 8 tonight", "ham"),
         ("claim free gift now", "spam"), ("call me after class", "ham")]
seen = Counter(msg for msg, _ in train)
seen_spam = Counter(msg for msg, lab in train if lab == "spam")
new = "claim your free prize"
print("times seen:", seen[new], " as spam:", seen_spam[new])
print("P(spam | message) =", seen_spam[new] / seen[new])

Output:

times seen: 0  as spam: 0
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    print("P(spam | message) =", seen_spam[new] / seen[new])
                                 ~~~~~~~~~~~~~~~^~~~~~~~~~~
ZeroDivisionError: division by zero

The new message was never seen before, word for word, so its count is 0 and the division fails with ZeroDivisionError: division by zero. Almost every new message is new text, so we must count something smaller: words.

Bag-of-words and CountVectorizer

Real-life example: at the sabzi market, whatever order you pick vegetables in, they end up in one bag; the bag is described by counts ("2 tomatoes, 1 onion, 0 potatoes"). The vendor's list of everything he sells is the vocabulary.

  • Bag-of-words: represent a text by how many times each word appears, throwing away word order.
  • Token: one unit of text after splitting (here: a word). Tokenisation: splitting text into tokens.
  • Vocabulary: every distinct token seen in the training texts. Each vocabulary word is one feature (column).
  • Count vector: one number per vocabulary word for one message.
  • CountVectorizer (scikit-learn): fit learns the vocabulary, transform turns texts into count vectors. It lowercases text, and its default token pattern keeps words of 2 or more letters/digits.
from sklearn.feature_extraction.text import CountVectorizer
msgs = ["free prize claim free", "dinner at home", "claim your prize now"]
vec = CountVectorizer()
X = vec.fit_transform(msgs)                 # learn the vocabulary, then count
print(vec.get_feature_names_out())
print(X.toarray())
print(vec.transform(["FREE free pizza at home"]).toarray())

Output:

['at' 'claim' 'dinner' 'free' 'home' 'now' 'prize' 'your']
[[0 1 0 2 0 0 1 0]
 [1 0 1 0 1 0 0 0]
 [0 1 0 0 0 1 1 1]]
[[1 0 0 2 1 0 0 0]]
  • The vocabulary is sorted alphabetically: free is column 4, so "free prize claim free" has a 2 there.
  • "FREE free pizza at home" counts free twice (lowercased) and simply drops pizza: words not in the vocabulary are ignored.
  • The result is a sparse matrix (mostly zeros); .toarray() shows it as a normal array.
  • Fit the vectorizer on training data only (Episode 16), so the test set cannot influence the vocabulary.
  • Weakness: order is lost, "not good" = "good not". (n-grams, CountVectorizer(ngram_range=(1, 2)), add word pairs as extra features.)

The naive assumption

Even with word counts, P(count vector | spam) for a 10,000-word vocabulary has astronomically many possible values. The shortcut is to look at one word at a time.

Real-life example: a detective at a railway station guessing who is going to a wedding judges each clue on its own (new clothes, a gift box, a sweets box) and multiplies the evidence, pretending the clues are unrelated, although new clothes and a gift usually go together. That is naive, yet it works.

  • Independent events: knowing one tells you nothing about the other: P(A and B) = P(A) × P(B).
  • Conditional independence: independent once the class is known: P(w₁, w₂ | spam) = P(w₁ | spam) × P(w₂ | spam).
  • Naive Bayes assumption: all features are conditionally independent given the class. It's almost never exactly true for words, which is why it's called naive.

With it, for each class c:

score(c) = P(c) × P(w₁ | c) × P(w₂ | c) × … × P(wₙ | c)          (∝ P(c | message))
P(word | c) = (times the word appears in class-c messages) / (total words in class-c messages)

Predict the class with the bigger score.

Naive Bayes from scratch

from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
         ("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
         ("call me after class", "ham"), ("free for dinner tonight", "ham")]
classes = ["spam", "ham"]
words = {c: Counter() for c in classes}               # word counts per class
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def score(msg, c, alpha):
    p = sum(1 for _, lab in train if lab == c) / len(train)     # prior P(c)
    total = sum(words[c].values())
    for w in msg.split():                                       # naive: multiply per word
        p *= (words[c][w] + alpha) / (total + alpha * len(vocab))
    return p
for msg in ["claim prize now", "free dinner tonight", "claim prize tonight"]:
    for alpha in [0, 1]:
        s = {c: score(msg, c, alpha) for c in classes}
        win = "tie" if s["spam"] == s["ham"] else max(s, key=s.get)
        print(f"{msg:<19} a={alpha} spam {s['spam']:.6f} ham {s['ham']:.6f} {win}")

Output:

claim prize now     a=0 spam 0.003472 ham 0.000000 spam
claim prize now     a=1 spam 0.000820 ham 0.000023 spam
free dinner tonight a=0 spam 0.000000 ham 0.001157 ham
free dinner tonight a=1 spam 0.000068 ham 0.000410 ham
claim prize tonight a=0 spam 0.000000 ham 0.000000 tie
claim prize tonight a=1 spam 0.000205 ham 0.000068 spam
  • Counter counts the words of each class; score multiplies the prior by each word's probability.
  • With alpha=0 (plain counting): "claim prize now" → spam, "free dinner tonight" → ham.
  • But "claim prize tonight" gives spam 0 and ham 0: a tie. Why: tonight never appeared in the 3 spam messages (12 words), so P(tonight | spam) = 0/12 = 0, and anything × 0 = 0. For ham, claim and prize never appeared. One unseen word vetoes all other evidence.

Laplace smoothing

Real-life example: a new dhaba on a food app with no reviews isn't a zero-star dhaba; we just don't know yet.

P(word | c) = (count + α) / (total words in c + α × vocabulary size)
  • Laplace smoothing: α = 1, as if every word had been seen one extra time in every class.
  • Additive (Lidstone) smoothing: any α > 0.
  • The α × vocabulary size in the denominator keeps the probabilities of all words summing to 1.
  • α is a hyperparameter: MultinomialNB(alpha=1.0) is the default.

In the output above, with alpha=1, "claim prize tonight" becomes spam 0.000205 vs ham 0.000068: spam wins by about 3×, and the other two messages keep their answers.

Work in logs: underflow

import math
probs = [0.01] * 400                 # 400 words, each with probability 0.01
p = 1.0
for q in probs: p *= q
print("multiply:", p)
print("add logs:", sum(math.log(q) for q in probs))
print("smallest float:", 5e-324, " half of it:", 5e-324 / 2)

Output:

multiply: 0.0
add logs: -1842.0680743952364
smallest float: 5e-324  half of it: 0.0
  • Underflow: a result smaller than the smallest positive float (about 5 × 10⁻³²⁴) is rounded to 0.0. 0.01⁴⁰⁰ = 10⁻⁸⁰⁰ underflows.
  • Fix: log(a × b × c) = log a + log b + log c. Add log-probabilities instead of multiplying probabilities. The class with the bigger log score is the same class with the bigger score, because log is increasing.
  • scikit-learn stores the learned log-probabilities in feature_log_prob_ (the trailing underscore means "learned during fit"). This sum is the log-likelihood from Episode 12.

Naive Bayes in scikit-learn: MultinomialNB

2,000 made-up SMS messages (30% spam) built from spammy, normal and common words with some overlap.

import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)                                   # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n):                       # each word: 50% main pool, 15% other, 35% common
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400)  # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
rule = [any(w in t.split() for w in ["won", "prize", "lottery"]) for t in t_te]
print(f"keyword rule accuracy  {np.mean(np.array(rule) == y_te):.3f}")
model = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0)).fit(t_tr, y_tr)
print(f"naive Bayes accuracy   {model.score(t_te, y_te):.3f}")
print(model.predict_proba(["claim your free cash now", "bring the cash home tonight"]).round(3))
vec, nb = model[0], model[1]
top = np.argsort(nb.feature_log_prob_[1] - nb.feature_log_prob_[0])[-5:]
print("most spammy words:", vec.get_feature_names_out()[top])

Output:

keyword rule accuracy  0.728
naive Bayes accuracy   0.886
[[0.046 0.954]
 [0.963 0.037]]
most spammy words: ['bonus' 'offer' 'cash' 'prize' 'click']
  • Keyword rule: 0.728. Naive Bayes: 0.886, learned by counting words.
  • predict_proba columns are in classes_ order: [ham (0), spam (1)]. "claim your free cash now" → 95.4% spam; "bring the cash home tonight" → 96.3% ham, even though it contains cash.
  • The most spammy words (highest log P(word | spam) − log P(word | ham)) were found from data, not written by hand.
  • Multinomial naive Bayes models a message as words drawn one after another from the class's word distribution, so counts matter.
  • Training = counting in one pass: very fast, even for millions of messages.
  • Its probabilities are often too extreme (overconfident) because correlated words are counted as independent evidence; rankings are good, exact percentages less so. Calibrate (Episode 25) if you need trustworthy probabilities.

Bernoulli naive Bayes

Real-life example: an attendance register records only present/absent, and absent is information too; a tally counts how many times.

  • Bernoulli naive Bayes: every vocabulary word is a yes/no feature (present or absent). It uses P(word present | class) and also P(word absent | class) for every missing word.
  • Use CountVectorizer(binary=True) (or let BernoulliNB(binarize=0.0) do it).
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)                                   # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n):                       # each word: 50% main pool, 15% other, 35% common
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400)  # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB
from sklearn.pipeline import make_pipeline
multi = make_pipeline(CountVectorizer(), MultinomialNB()).fit(t_tr, y_tr)
bern = make_pipeline(CountVectorizer(binary=True), BernoulliNB()).fit(t_tr, y_tr)
print(f"Multinomial (counts)       {multi.score(t_te, y_te):.3f}")
print(f"Bernoulli (present/absent) {bern.score(t_te, y_te):.3f}")
msg = ["free free free free call mom"]
print("multinomial P(spam):", multi.predict_proba(msg)[0, 1].round(3))
print("bernoulli   P(spam):", bern.predict_proba(msg)[0, 1].round(3))

Output:

Multinomial (counts)       0.886
Bernoulli (present/absent) 0.876
multinomial P(spam): 0.779
bernoulli   P(spam): 0.079
  • Overall accuracy is close (0.886 vs 0.876).
  • "free free free free call mom": multinomial says 77.9% spam because free counts four times; Bernoulli says 7.9%: it sees free once, sees the ham words call and mom, and treats every missing spammy word (no claim, no prize, …) as evidence for ham.
  • Bernoulli often suits short texts; multinomial suits longer texts where counts carry information.

Gaussian naive Bayes

Real-life example: heights of class 3 and class 10 students each follow their own bell curve; for a given height, ask which class's curve fits better.

  • Gaussian naive Bayes: for numeric features, it assumes each feature follows a normal distribution within each class, and learns a mean and a variance per feature per class (theta_, var_).
import numpy as np
from sklearn.naive_bayes import GaussianNB
rng = np.random.default_rng(28)                   # made-up numbers, not words
spam = np.c_[rng.normal(120, 25, 300), rng.normal(9, 3, 300)]     # length, digits
ham = np.c_[rng.normal(45, 20, 700), rng.normal(1, 1.5, 700)]
X = np.clip(np.r_[spam, ham], 0, None).round(); y = np.array([1] * 300 + [0] * 700)
gnb = GaussianNB().fit(X, y)
for c, name in [(0, "ham "), (1, "spam")]:                        # [length, digits]
    print(name, "mean", gnb.theta_[c].round(1), " variance", gnb.var_[c].round(1))
for m in [[100, 6], [60, 1], [80, 4]]:
    print(m, "-> P(spam) =", f"{gnb.predict_proba([m])[0, 1]:.3f}")

Output:

ham  mean [45.9  1.2]  variance [385.    1.5]
spam mean [119.6   8.8]  variance [612.5   8.7]
[100, 6] -> P(spam) = 1.000
[60, 1] -> P(spam) = 0.000
[80, 4] -> P(spam) = 0.394
  • Spam (made up here) is long and has many digits; ham is short with few digits.
  • [100 characters, 6 digits] → almost certainly spam; [60, 1] → ham; [80, 4] sits between: 39.4%.

The three flavours

Flavour Features Learns per class Typical use
MultinomialNB counts (word counts) P(word) from counts + α longer texts, documents
BernoulliNB 0/1 (present/absent) P(word present) short texts, binary features
GaussianNB real numbers mean and variance of each feature numeric data

(scikit-learn also has ComplementNB, a multinomial variant that often works better on imbalanced text, and CategoricalNB for categorical features.)

Generative vs discriminative models

Real-life example: a cook knows how biryani and pulao are made, so she can recognise either and also make a new plate; a food critic only learns to tell them apart.

  • Generative model: learns how each class generates data, P(x | class), plus P(class), then classifies with Bayes' theorem. Naive Bayes, Gaussian mixture models, and today's generative AI models are generative.
  • Discriminative model: learns P(class | x), or just the decision boundary, directly. Logistic regression, kNN, decision trees, SVMs.
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)                                   # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n):                       # each word: 50% main pool, 15% other, 35% common
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400)  # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
vec = CountVectorizer().fit(t_tr); A_tr, A_te = vec.transform(t_tr), vec.transform(t_te)
for n in [20, 60, 200, 1500]:                     # train on only the first n messages
    nb = MultinomialNB().fit(A_tr[:n], y_tr[:n])
    lr = LogisticRegression(max_iter=1000).fit(A_tr[:n], y_tr[:n])
    print(f"{n:>4} messages: naive Bayes {nb.score(A_te, y_te):.3f}  logistic {lr.score(A_te, y_te):.3f}")
p = np.exp(nb.feature_log_prob_[1])               # P(word | spam), learned from 1,500 messages
for _ in range(3):                                # a generative model can WRITE new spam
    print("fake spam:", " ".join(rng.choice(vec.get_feature_names_out(), 6, p=p)))

Output:

  20 messages: naive Bayes 0.740  logistic 0.702
  60 messages: naive Bayes 0.850  logistic 0.790
 200 messages: naive Bayes 0.856  logistic 0.856
1500 messages: naive Bayes 0.886  logistic 0.878
fake spam: win urgent for at kyc won
fake spam: gift me prize urgent gift the
fake spam: link click bonus win the reward
  • With only 20-60 training messages, naive Bayes beats logistic regression (0.740 vs 0.702, 0.850 vs 0.790): its strong assumption lets it learn quickly from little data.
  • With 1,500 messages they are close (0.886 vs 0.878). With lots of data, discriminative models often catch up or win.
  • Because it learned P(word | spam), naive Bayes can sample new spam-like messages. They're word salad (order was never learned), but clearly spammy. Logistic regression cannot do this.

Naive Bayes at a glance

Question Answer
Training Count (or compute means/variances) per class: one pass, very fast
Prediction log prior + Σ log P(feature | class), pick the max
Key assumption Features are conditionally independent given the class
Needs scaling? No (Gaussian NB models each feature's own scale)
Strengths Fast, tiny memory, strong with little data, great text baseline, handles many features
Weaknesses Overconfident probabilities, can't model feature interactions, word order lost with bag-of-words
Main hyperparameter alpha (smoothing) for Multinomial/Bernoulli; var_smoothing for Gaussian

Common mistakes

Mistake What you see Why Fix
Hand-written keyword rules 3 of 6 wrong; accuracy 0.728 Scammers change words; one word decides Learn from labelled data
Counting whole messages ZeroDivisionError: division by zero New text was never seen Bag-of-words, per-word probabilities
No smoothing (alpha=0) spam 0.000000 ham 0.000000 -> tie One unseen word makes the product 0 Laplace smoothing, alpha=1
Multiplying hundreds of probabilities multiply: 0.0 Float underflow Add log-probabilities
Fitting CountVectorizer on all data Optimistic test score Test vocabulary leaks into training Put it in a pipeline, fit on train only
Trusting predict_proba as exact 0.99 on unsure messages Correlated words counted as independent Calibrate (Episode 25)
GaussianNB on word counts Poor accuracy Counts aren't bell-shaped MultinomialNB / BernoulliNB
Negative values into MultinomialNB ValueError: Negative values in data passed to MultinomialNB (input X). Counts can't be negative Don't standardise counts; use non-negative features

The last error, reproduced:

import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.preprocessing import StandardScaler
counts = np.array([[2, 0, 1], [0, 3, 0], [1, 1, 4]])
X = StandardScaler().fit_transform(counts)            # standardised counts: some are negative
MultinomialNB().fit(X, [1, 0, 1])

Output:

Traceback (most recent call last):
  ... (library frames) ...
ValueError: Negative values in data passed to MultinomialNB (input X).

Mini project: naive Bayes in NumPy

Write a MyMultinomialNB class that works in log space and check it against scikit-learn on the 2,000 messages.

import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)                                   # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n):                       # each word: 50% main pool, 15% other, 35% common
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400)  # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
vec = CountVectorizer().fit(t_tr)
A, B = vec.transform(t_tr).toarray(), vec.transform(t_te).toarray()
class MyMultinomialNB:                                   # naive Bayes in NumPy, in log space
    def __init__(self, alpha=1.0): self.alpha = alpha
    def fit(self, X, y):
        self.classes_ = np.unique(y)
        self.log_prior_ = np.log([np.mean(y == c) for c in self.classes_])
        counts = np.array([X[y == c].sum(0) for c in self.classes_]) + self.alpha
        self.log_prob_ = np.log(counts / counts.sum(1, keepdims=True))   # log P(word | class)
        return self
    def predict(self, X):
        scores = X @ self.log_prob_.T + self.log_prior_   # count × log P, summed, + log prior
        return self.classes_[scores.argmax(1)]
mine = MyMultinomialNB().fit(A, y_tr).predict(B)
sk = MultinomialNB().fit(A, y_tr).predict(B)
print(f"my accuracy {np.mean(mine == y_te):.3f}  sklearn {np.mean(sk == y_te):.3f}  agree on {np.mean(mine == sk):.1%}")

Output:

my accuracy 0.886  sklearn 0.886  agree on 100.0%

The matrix product X @ log_prob_.T computes, for every message and class, Σ count(word) × log P(word | class) in one step. Our 15-line class agrees with scikit-learn on 100% of the test messages.

Practice

1. Make the from-scratch version use logs (and α = 1).

Solution
import math
from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
         ("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
         ("call me after class", "ham"), ("free for dinner tonight", "ham")]
classes = ["spam", "ham"]
words = {c: Counter() for c in classes}
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def log_score(msg, c, alpha=1):
    s = math.log(sum(1 for _, lab in train if lab == c) / len(train))      # log prior
    total = sum(words[c].values())
    for w in msg.split():
        s += math.log((words[c][w] + alpha) / (total + alpha * len(vocab)))  # add, don't multiply
    return s
for msg in ["claim prize now", "free dinner tonight", "claim prize tonight"]:
    s = {c: log_score(msg, c) for c in classes}
    print(f"{msg:<20} log spam {s['spam']:.3f}  log ham {s['ham']:.3f}  -> {max(s, key=s.get)}")

Output:

claim prize now      log spam -7.106  log ham -10.690  -> spam
free dinner tonight  log spam -9.591  log ham -7.799  -> ham
claim prize tonight  log spam -8.493  log ham -9.591  -> spam

Same winners as the multiplied version with α = 1, but the scores are now ordinary negative numbers that can't underflow.

2. Try alpha = 0.01, 0.1, 1, 10, 100. What changes?

Solution
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)                                   # 2,000 made-up SMS messages
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n):                       # each word: 50% main pool, 15% other, 35% common
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(4, 10))) for _ in range(n)]
texts = sms(SPAM, HAM, 600) + sms(HAM, SPAM, 1400); y = np.array([1] * 600 + [0] * 1400)  # 1 = spam
t_tr, t_te, y_tr, y_te = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
for a in [0.01, 0.1, 1, 10, 100]:
    m = make_pipeline(CountVectorizer(), MultinomialNB(alpha=a)).fit(t_tr, y_tr)
    p = m.predict_proba(["claim your free cash now"])[0, 1]
    print(f"alpha {a:<5} accuracy {m.score(t_te, y_te):.3f}  P(spam | 'claim your free cash now') {p:.3f}")

Output:

alpha 0.01  accuracy 0.888  P(spam | 'claim your free cash now') 0.956
alpha 0.1   accuracy 0.888  P(spam | 'claim your free cash now') 0.956
alpha 1     accuracy 0.886  P(spam | 'claim your free cash now') 0.954
alpha 10    accuracy 0.884  P(spam | 'claim your free cash now') 0.939
alpha 100   accuracy 0.874  P(spam | 'claim your free cash now') 0.761

Accuracy barely moves until α gets large; a big α pulls every word's probability towards the same value, so the evidence gets weaker (P(spam) for the spammy message falls to 0.761 at α = 100).

3. Add the spam message "lottery won claim tonight" to the 6 training messages. Classify "lottery tonight", "dinner tonight" and "won lottery".

Solution
from collections import Counter
train = [("free prize claim now", "spam"), ("claim free gift now", "spam"),
         ("win cash prize now", "spam"), ("dinner at 8 tonight", "ham"),
         ("call me after class", "ham"), ("free for dinner tonight", "ham"),
         ("lottery won claim tonight", "spam")]                        # the new message
classes = ["spam", "ham"]
words = {c: Counter() for c in classes}
for msg, c in train: words[c].update(msg.split())
vocab = {w for c in classes for w in words[c]}
def score(msg, c, alpha=1):
    p = sum(1 for _, lab in train if lab == c) / len(train)
    total = sum(words[c].values())
    for w in msg.split(): p *= (words[c][w] + alpha) / (total + alpha * len(vocab))
    return p
for msg in ["lottery tonight", "dinner tonight", "won lottery"]:
    s = {c: score(msg, c) for c in classes}
    print(f"{msg:<16} spam {s['spam']:.5f}  ham {s['ham']:.5f}  -> {max(s, key=s.get)}")

Output:

lottery tonight  spam 0.00198  ham 0.00143  -> spam
dinner tonight   spam 0.00099  ham 0.00429  -> ham
won lottery      spam 0.00198  ham 0.00048  -> spam

tonight now appears in spam once, so "lottery tonight" becomes spam, while "dinner tonight" stays ham.

4. BernoulliNB vs MultinomialNB on long messages (30-59 words).

Solution
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(28)
SPAM = "free prize claim win cash offer urgent click link reward gift won kyc bonus".split()
HAM = "dinner home call class tonight meet bring mom office train late tomorrow bus".split()
COMMON = "now today your the at for me you is to".split()
def sms(main, other, n, lo, hi):
    pools = [main, other, COMMON]; word = lambda: rng.choice(pools[rng.choice(3, p=[.5, .15, .35])])
    return [" ".join(word() for _ in range(rng.integers(lo, hi))) for _ in range(n)]
for lo, hi in [(4, 10), (30, 60)]:                     # short vs long messages
    texts = sms(SPAM, HAM, 600, lo, hi) + sms(HAM, SPAM, 1400, lo, hi); y = np.array([1] * 600 + [0] * 1400)
    a, b, ya, yb = train_test_split(texts, y, test_size=0.25, random_state=0, stratify=y)
    mu = make_pipeline(CountVectorizer(), MultinomialNB()).fit(a, ya).score(b, yb)
    be = make_pipeline(CountVectorizer(binary=True), BernoulliNB()).fit(a, ya).score(b, yb)
    print(f"{lo}-{hi - 1} words: multinomial {mu:.3f}  bernoulli {be:.3f}")

Output:

4-9 words: multinomial 0.886  bernoulli 0.876
30-59 words: multinomial 0.998  bernoulli 1.000

On these made-up long messages both become nearly perfect, because every message has many words of evidence. On real long documents with repeated words, multinomial is usually the safer default; Bernoulli is often good for short texts.

5. By hand: P(spam) = 0.3, P(ham) = 0.7. P(free | spam) = 0.2, P(free | ham) = 0.02. A one-word message "free": which class wins?

Solution

spam: 0.3 × 0.2 = 0.06; ham: 0.7 × 0.02 = 0.014. Spam wins. Normalised: P(spam | free) = 0.06 / (0.06 + 0.014) ≈ 0.81.

Quick quiz

  1. With α = 0, P(win | spam) = 0.10, P(cash | spam) = 0.08, P(tonight | spam) = 0. The spam score of "win cash tonight" is? 0: anything times 0 is 0.
  2. Which flavour for "number of times each word appears"? MultinomialNB.
  3. Which flavour uses the absence of a word as evidence? BernoulliNB.
  4. Why add logs instead of multiplying? Multiplying many small probabilities underflows to 0.0.
  5. Is naive Bayes generative or discriminative? Generative: it learns P(x | class).

Interview questions

  1. Why is naive Bayes called naive? It assumes all features are conditionally independent given the class, which is rarely true (e.g. "free" and "prize" co-occur).
  2. Why does it still work well? Classification only needs the correct class to have the highest score; even with wrong probabilities, the ranking is often right. It also has few parameters, so it learns well from little data.
  3. What is Laplace smoothing and why is it needed? Adding α (1) to every count so that a word never seen in a class doesn't get probability 0 and veto all other evidence.
  4. Why use log-probabilities? To avoid floating-point underflow when multiplying many small numbers; log turns products into sums and keeps the same winner.
  5. Multinomial vs Bernoulli vs Gaussian NB? Counts; binary presence/absence (absence counts as evidence); continuous features with a normal distribution per class.
  6. Generative vs discriminative? Generative models learn P(x | y) and P(y) (naive Bayes); discriminative models learn P(y | x) or the boundary directly (logistic regression). Generative models learn fast with little data and can sample; discriminative usually win with lots of data.
  7. Are naive Bayes probabilities well calibrated? Usually not: they're pushed towards 0 and 1. Use calibration (Platt scaling or isotonic regression) if probabilities matter.
  8. Does naive Bayes need feature scaling? No.
  9. What does bag-of-words lose? Word order and context ("not good" = "good not").
  10. How would you handle a word never seen in training at prediction time? CountVectorizer ignores it (it's not in the vocabulary); smoothing handles words seen in one class but not the other.

Glossary

Term Meaning (plain words) Example from this episode
Text classification Sorting texts into categories SMS → spam or ham
Spam / ham Unwanted bulk or scam message / normal wanted message "You WON Rs 50,000" / "Mom: dinner at 8"
Bayes' theorem P(A | B) = P(B | A) P(A) / P(B): flips a conditional probability P(spam | message) from P(message | spam)
Prior Probability of a class before seeing the data P(spam) = 0.3
Likelihood Probability of the data if a class were true P(message | spam)
Posterior Probability of a class after seeing the data P(spam | message)
Bag-of-words Text as word counts, order thrown away "free prize free" → free: 2, prize: 1
Token One unit of text after splitting "claim", "prize"
Vocabulary All distinct tokens seen in training 8 words in the BoW example
Count vector One count per vocabulary word for a text [0 1 0 2 0 0 1 0]
CountVectorizer scikit-learn tool that builds the vocabulary and count vectors CountVectorizer().fit_transform(msgs)
Sparse matrix A matrix stored without its many zeros output of fit_transform
Independent Knowing one event says nothing about the other two coin tosses
Conditional independence Independent once the class is known P(free, prize | spam) = P(free | spam) P(prize | spam)
Naive Bayes Bayes' theorem + the conditional-independence assumption the spam filter
Laplace smoothing Add 1 to every count so nothing has probability 0 (0 + 1) / (12 + 1 × 16) for tonight in spam
Additive smoothing Same with any α > 0 MultinomialNB(alpha=0.1)
Underflow A number too small for a float becomes 0.0 0.01⁴⁰⁰ → 0.0
Log-probability The logarithm of a probability; sums replace products Σ log 0.01 = −1842.07
Multinomial naive Bayes Naive Bayes on counts MultinomialNB on word counts
Bernoulli naive Bayes Naive Bayes on present/absent features; absence is evidence BernoulliNB + binary=True
Gaussian naive Bayes Naive Bayes with a bell curve per numeric feature per class length and digits of a message
Generative model Learns how each class generates data; can sample new data naive Bayes writing fake spam
Discriminative model Learns the boundary / P(class | x) directly logistic regression
Overconfident Probabilities closer to 0 or 1 than the truth predict_proba 0.99 on a borderline message

Summary

  • Keyword rules fail; learn from labelled messages with Bayes' theorem.
  • Turn text into numbers with bag-of-words (CountVectorizer).
  • The naive assumption (words conditionally independent given the class) turns an impossible estimate into simple per-word counts.
  • Laplace smoothing stops unseen words from zeroing everything; logs stop underflow.
  • Multinomial for counts, Bernoulli for present/absent, Gaussian for numbers.
  • Naive Bayes is generative: fast to train, strong with little data, can sample; discriminative models often win with lots of data.
  • Next episode: decision trees from scratch.