SyntaxPilot
Courses/Machine Learning/Lesson 71
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 71 of 90

NLP basics

natural language processing (NLP)corpustext normalisationword tokenisationstop wordsstemminglemmatisationn-gramsregex cleaningsentiment analysisTF-IDF plus logistic regression text classifier

Machine Learning Full Course · Episode 71. Every code sample below was run with Python 3.13, scikit-learn and NLTK 3.10.4 (NLTK data packages movie_reviews, stopwords, wordnet and punkt_tab, installed with nltk.download(...)), and the output shown is the real output.

The big idea: to a computer, text is just characters, so a hand-made list of "happy words" fails: it misses capital letters, repeated letters, word forms and negation, and reached only 65.3% on 2,000 real movie reviews. Natural language processing turns text into numbers in steps: normalise and clean it (regex), cut it into tokens, optionally drop stop words and reduce words to stems or lemmas, add n-grams to keep some order, weigh the words with TF-IDF, and let logistic regression learn a weight for every word. That pipeline reached 88.4% on reviews it had never seen.

Sources: the corpus is the polarity dataset v2.0 of Bo Pang and Lillian Lee (1,000 positive and 1,000 negative movie reviews; first used in "A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts", Proceedings of the ACL, 2004, according to the README shipped with NLTK's movie_reviews corpus). Martin F. Porter, "An algorithm for suffix stripping", Program 14(3):130-137, 1980 (tartarus.org: the stemmer removes "the commoner morphological and inflexional endings from words in English").

Words you need first

  • One-hot vector and bag of words (Episodes 16, 28): a column per word; a review becomes its word counts, order forgotten.
  • TF-IDF (Episode 39): term frequency × inverse document frequency: a word's count in this text, down-weighted if it appears in many texts.
  • Logistic regression (Episodes 22-23): a linear model with a sigmoid; one weight per feature; C is the inverse regularisation strength.
  • Pipeline and leakage (Episodes 15, 17): chain preprocessing and the model so preprocessing is fitted on training data only.
  • Stratified split (Episode 15): keeps the same share of each class in train and test.

The story: a rule that reads reviews

Imagine a food delivery app with thousands of reviews a day. Natural language processing (NLP) is getting computers to work with human language. Sentiment analysis is deciding whether a text is positive or negative. A corpus is the collection of texts we learn from.

from nltk.corpus import movie_reviews as mr          # 2,000 real movie reviews, labelled
def rule(text):                                     # obvious attempt: look for happy words
    happy = sum(text.count(w) for w in ["good", "great", "loved", "best"])
    sad = sum(text.count(w) for w in ["bad", "worst", "boring", "awful"])
    return "pos" if happy > sad else "neg"
for r in ["The biryani was good", "Not good at all, cold food", "GREAT taste!!!",
          "Bad packing but the best paneer I've had", "goooood"]:
    print(f"{r!r:45} -> {rule(r)}")
ids = mr.fileids()
right = sum(rule(mr.raw(f)) == mr.categories(f)[0] for f in ids)
print(f"rule on {len(ids)} real reviews: {right / len(ids):.1%} right")

Output:

'The biryani was good'                        -> pos
'Not good at all, cold food'                  -> pos
'GREAT taste!!!'                              -> neg
"Bad packing but the best paneer I've had"    -> pos
'goooood'                                     -> neg
rule on 2000 real reviews: 65.3% right
  • "Not good at all" is positive (the rule saw good and missed not).
  • "GREAT taste!!!" is negative ("great" in lower case is not in "GREAT"), and "goooood" too.
  • "Bad packing but the best paneer" is right only by luck: the rule never saw Bad with a capital B.
  • 65.3% on 2,000 real reviews.

Why: text is just characters (GREAT ≠ great ≠ goooood), one word has many forms (love, loved, loving), the rule ignores order, and eight hand-picked words miss most of the signal.

Real-life example: preparing vegetables

Kitchen NLP step
Wash off mud and stickers Normalisation + regex cleaning: lower case, remove links, squash repeated letters
Chop into pieces Tokenisation: text → a list of tokens
Throw away the peels (not the useful bits!) Stop-word removal: drop very common words
Diced, sliced or grated: still potato Stemming / lemmatisation: loved, loving → love

Normalisation, regex cleaning and tokenisation

  • Text normalisation: making different ways of writing the same thing identical: lower case, every number → num, "goooood" → "good".
  • Regular expression (regex): a small pattern language for finding text. \d+ = one or more digits; https?://\S+ = http or https, ://, then non-space characters; (.)\1{2,} = a character followed by two or more copies of itself. re.sub(pattern, replacement, text) replaces every match.
  • Token: one unit of text, usually a word or a punctuation mark. Word tokenisation: turning text into a list of tokens.
import re
from nltk.tokenize import word_tokenize
review = "LOVED it!!! Delivery in 20 mins :) but the dal wasn't hot... 5/5 https://t.co/xyz"
print("split()  :", review.split())
text = review.lower()                                 # normalise case
text = re.sub(r"https?://\S+", " ", text)             # regex: remove links
text = re.sub(r"\d+", " num ", text)                  # numbers -> one token
text = re.sub(r"(.)\1{2,}", r"\1\1", text)            # "!!!" -> "!!", "goooood" -> "good"
print("cleaned  :", text)
print("regex    :", re.findall(r"[a-z']+", text))   # words only
print("tokenize :", word_tokenize(text))              # NLTK's word tokeniser
print("goooood  ->", re.sub(r"(.)\1{2,}", r"\1\1", "goooood"))

Output:

split()  : ['LOVED', 'it!!!', 'Delivery', 'in', '20', 'mins', ':)', 'but', 'the', 'dal', "wasn't", 'hot...', '5/5', 'https://t.co/xyz']
cleaned  : loved it!! delivery in  num  mins :) but the dal wasn't hot..  num / num  
regex    : ['loved', 'it', 'delivery', 'in', 'num', 'mins', 'but', 'the', 'dal', "wasn't", 'hot', 'num', 'num']
tokenize : ['loved', 'it', '!', '!', 'delivery', 'in', 'num', 'mins', ':', ')', 'but', 'the', 'dal', 'was', "n't", 'hot', '..', 'num', '/', 'num']
goooood  -> good

split() keeps punctuation glued on ("it!!!", "hot...") and the whole link as one word. NLTK's word_tokenize separates punctuation and splits "wasn't" into was + n't, so the negation becomes its own token.

Stop words, stemming and lemmatisation

  • Stop words: the most common words (the, is, and), often removed to save space. NLTK's English list has 198 words; scikit-learn's built-in stop_words="english" list has 318. Both contain "not" and "no".
  • Stemming: chopping endings with fixed rules (Porter, 1980). Fast; the stem may not be a real word: studies → studi.
  • Lemmatisation: looking up the dictionary form, the lemma, in a dictionary (WordNet). It needs the part of speech (noun, verb, adjective); NLTK's default is noun.
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
stop = set(stopwords.words("english"))                # NLTK's English stop-word list
print("stop words:", len(stop), "| e.g.", sorted(stop)[:5])
words = "the food was not good and the delivery was not fast".split()
print("kept:", [w for w in words if w not in stop])   # "not" is a stop word!
print("'not' in list?", "not" in stop, "| 'no' in list?", "no" in stop)
stem, lem = PorterStemmer(), WordNetLemmatizer()
for w in ["studies", "studying", "better", "running", "ran", "cookies", "university"]:
    v, n = lem.lemmatize(w, "v"), lem.lemmatize(w)          # as a verb, as a noun
    print(f"{w:10} stem {stem.stem(w):7} verb {v:10} noun {n}")
print("better as an adjective:", lem.lemmatize("better", "a"))

Output:

stop words: 198 | e.g. ['a', 'about', 'above', 'after', 'again']
kept: ['food', 'good', 'delivery', 'fast']
'not' in list? True | 'no' in list? True
studies    stem studi   verb study      noun study
studying   stem studi   verb study      noun studying
better     stem better  verb better     noun better
running    stem run     verb run        noun running
ran        stem ran     verb run        noun ran
cookies    stem cooki   verb cookies    noun cooky
university stem univers verb university noun university
better as an adjective: good

"the food was not good and the delivery was not fast" became ['food', 'good', 'delivery', 'fast']: a complaint turned into praise. For sentiment, never drop negations blindly. Note the lemmatiser quirks: as a noun, cookies → cooky and ran stays ran; as a verb, ran → run; as an adjective, better → good.

N-grams

An n-gram is n tokens in a row: "good" is a 1-gram (unigram), "not good" a 2-gram (bigram), "was not good" a 3-gram (trigram). A sentence of n words has n − 1 bigrams. Adding bigrams puts some order back into a bag of words, at the cost of many more columns.

from nltk.util import ngrams
from sklearn.feature_extraction.text import CountVectorizer
a, b = "good not bad", "bad not good"
for n in (1, 2):
    print(f"{n}-grams of {a!r}:", [" ".join(g) for g in ngrams(a.split(), n)])
    print(f"{n}-grams of {b!r}:", [" ".join(g) for g in ngrams(b.split(), n)])
for rng in [(1, 1), (1, 2)]:
    cv = CountVectorizer(ngram_range=rng).fit([a, b])
    X = cv.transform([a, b]).toarray()
    print(f"ngram_range={rng}: columns {list(cv.get_feature_names_out())}")
    print("   rows:", X.tolist(), "| same row?", (X[0] == X[1]).all())

Output:

1-grams of 'good not bad': ['good', 'not', 'bad']
1-grams of 'bad not good': ['bad', 'not', 'good']
2-grams of 'good not bad': ['good not', 'not bad']
2-grams of 'bad not good': ['bad not', 'not good']
ngram_range=(1, 1): columns ['bad', 'good', 'not']
   rows: [[1, 1, 1], [1, 1, 1]] | same row? True
ngram_range=(1, 2): columns ['bad', 'bad not', 'good', 'good not', 'not', 'not bad', 'not good']
   rows: [[1, 0, 1, 1, 1, 1, 0], [1, 1, 1, 0, 1, 0, 1]] | same row? False

A TF-IDF + logistic regression sentiment classifier

  • TfidfVectorizer(sublinear_tf=True): TF-IDF with 1 + log(count) instead of the raw count, so a word repeated 20 times is not 20 times louder.
  • LogisticRegression(C=10): one weight per column; positive weights push towards "pos".
  • make_pipeline: the vectorizer learns its vocabulary from the training reviews only.
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]
y = [mr.categories(f)[0] for f in mr.fileids()]          # "pos" or "neg"
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
for name, kw in [("words", {}), ("words, stop words removed", {"stop_words": "english"}),
                 ("words + 2-grams", {"ngram_range": (1, 2), "min_df": 2})]:
    model = make_pipeline(TfidfVectorizer(sublinear_tf=True, **kw), LogisticRegression(C=10))
    model.fit(Xtr, ytr)                                    # fit on TRAIN text only
    print(f"{name:26}: test accuracy {model.score(Xte, yte):.1%}")
for r in ["the acting was not good and the plot was a mess",
          "a beautiful, moving film with a brilliant cast"]:
    print(f"{r!r} -> {model.predict([r])[0]}")

Output:

words                     : test accuracy 88.4%
words, stop words removed : test accuracy 87.6%
words + 2-grams           : test accuracy 87.8%
'the acting was not good and the plot was a mess' -> neg
'a beautiful, moving film with a brilliant cast' -> pos

The same three settings with 5-fold cross-validation (Episode 26) on all 2,000 reviews: plain words 87.8% (± 0.6%), stop words removed 87.0% (± 1.1%), words + 2-grams 87.7% (± 0.9%). No real winner: measure, don't assume.

What the model learned

from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
docs = [mr.raw(f) for f in mr.fileids()]
y = [mr.categories(f)[0] for f in mr.fileids()]
vec = TfidfVectorizer(sublinear_tf=True)
X = vec.fit_transform(docs)                              # rows = reviews, columns = words
print("matrix:", X.shape, "| non-zero:", X.nnz, f"({X.nnz / (X.shape[0] * X.shape[1]):.2%})")
clf = LogisticRegression(C=10).fit(X, y)
w, words = clf.coef_[0], vec.get_feature_names_out()     # weight > 0 pushes towards "pos"
order = w.argsort()
print("most negative:", [words[i] for i in order[:8]])
print("most positive:", [words[i] for i in order[-8:][::-1]])

Output:

matrix: (2000, 39659) | non-zero: 666842 (0.84%)
most negative: ['bad', 'worst', 'boring', 'supposed', 'nothing', 'unfortunately', 'plot', 'script']
most positive: ['great', 'fun', 'well', 'hilarious', 'very', 'also', 'memorable', 'excellent']

The matrix is sparse: only 0.84% of its cells are non-zero, so scikit-learn stores only those. Nobody gave the model word lists; it found bad, worst, boring and great, fun, hilarious in the data.

Common mistakes

from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
vec.fit("The dal was hot and the roti was soft")      # one review, not a list
Traceback (most recent call last):
  ...
ValueError: Iterable over raw text documents expected, string object received.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.transform(["the dal was hot"])               # transform before fit
Traceback (most recent call last):
  ...
sklearn.exceptions.NotFittedError: The TF-IDF vectorizer is not fitted
Mistake Exact message / symptom Why Fix
vec.fit("one review") ValueError: Iterable over raw text documents expected, string object received. it needs a list of documents vec.fit(["one review"])
transform before fit NotFittedError: The TF-IDF vectorizer is not fitted no vocabulary learned yet fit on training text, then transform
Removing stop words for sentiment "not good" → "good", silently not and no are on the lists keep negations, and measure
Fitting the vectorizer on all the text a test score that is too optimistic vocabulary and IDF leak from the test set a Pipeline, fitted on train only
Lemmatising without the part of speech cookies → cooky, ran → ran the default is noun pass pos, or use a part-of-speech tagger
Keyword rules 65.3% case, spelling, negation, few words learn weights from labelled data

Mini project: explain each prediction

Clean the text inside the vectorizer (preprocessor=), train on all 2,000 reviews, and for a new review print the probability and the words that pushed most for and against (TF-IDF value × weight).

import re
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
def clean(t):                                     # the cleaning steps from the video
    t = re.sub(r"https?://\S+", " ", t.lower())
    t = re.sub(r"\d+", " num ", t)
    return re.sub(r"(.)\1{2,}", r"\1\1", t)
model = make_pipeline(TfidfVectorizer(sublinear_tf=True, preprocessor=clean),
                      LogisticRegression(C=10)).fit(docs, y)
vec, clf = model[0], model[1]
def explain(review):                              # which words pushed the decision?
    row = vec.transform([review]); words = vec.get_feature_names_out()
    parts = sorted(((row[0, j] * clf.coef_[0][j], words[j]) for j in row.indices), reverse=True)
    p = model.predict_proba([review])[0][list(clf.classes_).index("pos")]
    print(f"{review!r}\n   P(pos) = {p:.2f} | for: {[w for _, w in parts[:3]]} | against: {[w for _, w in parts[-3:]]}")
for r in ["What a brilliant, funny film. Loved it!!!",
          "The plot was boring and the ending was the worst I have seen",
          "Not bad at all, actually quite good"]:
    explain(r)

Output:

'What a brilliant, funny film. Loved it!!!'
   P(pos) = 0.90 | for: ['brilliant', 'loved', 'film'] | against: ['it', 'what', 'funny']
'The plot was boring and the ending was the worst I have seen'
   P(pos) = 0.01 | for: ['seen', 'ending', 'and'] | against: ['plot', 'boring', 'worst']
'Not bad at all, actually quite good'
   P(pos) = 0.19 | for: ['quite', 'good', 'not'] | against: ['at', 'actually', 'bad']

The third review shows the limit of single words: "Not bad at all, actually quite good" gets only 0.19, because bad has a strong negative weight and not is just another word. Try ngram_range=(1, 2) and see whether "not bad" helps; the next episodes give the model a real sense of word meaning and order.

Practice

1. Remove NLTK's stop words except the negations. Does it beat removing all of them?

Solution
from nltk.corpus import movie_reviews as mr, stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
keep = {"not", "no", "nor", "don't", "didn't", "wasn't", "isn't"}
stop = sorted(set(stopwords.words("english")) - keep)          # NLTK list minus negations
for name, sw in [("no removal", None), ("NLTK list", sorted(stopwords.words("english"))),
                 ("NLTK list minus negations", stop)]:
    m = make_pipeline(TfidfVectorizer(sublinear_tf=True, stop_words=sw), LogisticRegression(C=10))
    print(f"{name:26}: {m.fit(Xtr, ytr).score(Xte, yte):.1%}")
no removal                : 88.4%
NLTK list                 : 88.0%
NLTK list minus negations : 88.4%

Removing the full list costs 0.4 points; keeping the negations brings it back to the no-removal score.

2. Add a Porter stemmer to the pipeline.

Solution
import re
from nltk.corpus import movie_reviews as mr
from nltk.stem import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
stem = PorterStemmer()
def stem_tokens(text): return [stem.stem(w) for w in re.findall(r"[a-z']+", text.lower())]
for name, kw in [("words", {}), ("Porter stems", {"tokenizer": stem_tokens, "token_pattern": None})]:
    m = make_pipeline(TfidfVectorizer(sublinear_tf=True, **kw), LogisticRegression(C=10)).fit(Xtr, ytr)
    print(f"{name:13}: {len(m[0].vocabulary_):6} columns, test accuracy {m.score(Xte, yte):.1%}")
words        :  35327 columns, test accuracy 88.4%
Porter stems :  25784 columns, test accuracy 86.6%

Stemming cut the vocabulary by about a quarter but lowered accuracy here; with plenty of data, TF-IDF + logistic regression often does fine without it.

3. Compare ngram_range (1, 1), (1, 2) and (1, 3) with 5-fold cross-validation.

Solution
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
for rng in [(1, 1), (1, 2), (1, 3)]:
    m = make_pipeline(TfidfVectorizer(sublinear_tf=True, ngram_range=rng, min_df=2), LogisticRegression(C=10))
    s = cross_val_score(m, docs, y, cv=5)
    print(f"ngram_range={rng}: 5-fold accuracy {s.mean():.1%} ± {s.std():.1%}")
ngram_range=(1, 1): 5-fold accuracy 88.0% ± 0.7%
ngram_range=(1, 2): 5-fold accuracy 87.7% ± 0.9%
ngram_range=(1, 3): 5-fold accuracy 87.3% ± 1.4%

On 2,000 reviews, more n-grams added columns faster than useful signal.

4. How many 3-grams does a 10-word sentence have?

Solution

10 − 3 + 1 = 8. In general n words have n − k + 1 k-grams.

Quick quiz

from nltk.util import ngrams
grams = list(ngrams("the food was cold".split(), 2))
# len(grams) = ?
Answer

3: ("the", "food"), ("food", "was"), ("was", "cold").

Interview questions

  1. Stemming vs lemmatisation? Stemming chops endings by rules (fast, may produce non-words like "studi"); lemmatisation maps to the dictionary form using a vocabulary and the part of speech (slower, real words).
  2. Should you always remove stop words? No. For topic tasks it can save space; for sentiment it can delete negations. Measure both.
  3. Why TF-IDF instead of raw counts? Words common to every document (the, movie) carry little information; IDF down-weights them.
  4. Why is the TF-IDF matrix sparse and why does it matter? Each document uses a tiny fraction of the vocabulary; storing only non-zeros saves memory and makes linear models fast.
  5. How do n-grams help, and what do they cost? They capture short phrases and negation ("not good"); the number of columns grows quickly, so use min_df and validation.
  6. Where can leakage happen in a text pipeline? Fitting the vectorizer (vocabulary and IDF) on test documents; fit it inside a pipeline on training folds only.

Glossary

Term Meaning, with an example
Natural language processing (NLP) Getting computers to work with human language; classifying reviews
Corpus A collection of texts used for learning; 2,000 movie reviews
Sentiment analysis Deciding whether a text is positive or negative
Text normalisation Making different spellings identical: "GREAT" → "great", "20" → "num"
Regular expression (regex) A text-search pattern; \d+ matches "20"
Regex cleaning Using re.sub to remove or replace patterns such as links
Token One unit of text, usually a word: "wasn't" → "was", "n't"
Word tokenisation Splitting text into tokens (word_tokenize)
Stop words Very common words often removed: the, is, and (and also not, no!)
Stemming Chopping word endings by rules: studies → studi (Porter, 1980)
Stem What is left after stemming; may not be a real word
Lemmatisation Mapping a word to its dictionary form: ran → run (WordNet)
Lemma A word's dictionary form
Part of speech A word's grammatical role: noun, verb, adjective
WordNet An English dictionary database used by NLTK's lemmatiser
N-gram n tokens in a row; "not good" is a 2-gram (bigram)
ngram_range Which n-gram lengths a vectorizer counts; (1, 2) = words and pairs
sublinear_tf Use 1 + log(count) as the term frequency
TF-IDF Term frequency × inverse document frequency (Episode 39)
Sparse matrix A matrix stored without its zeros; 0.84% non-zero here
Text classifier A model that assigns a label to a text; TF-IDF + logistic regression

Summary

  • A keyword rule reached 65.3% because text is just characters: case, spelling, word forms and negation break it.
  • Normalise and clean with regex, tokenise (NLTK separates "n't"), and be careful: stop-word lists contain not and no.
  • Stemming chops (studi), lemmatisation looks up (study) and needs the part of speech.
  • N-grams keep a little order; "good not bad" and "bad not good" get different rows.
  • TF-IDF + logistic regression reached 88.4% on held-out reviews; its learned weights found bad/worst/boring and great/fun/hilarious by itself.

Next: Episode 72, word embeddings: dense word vectors where good and great are close, word2vec from scratch, GloVe and subword tokenisation.