Machine Learning Full Course · Episode 53. Every code sample below was run with Python 3.13, NumPy 2.5.3 and pandas 3.0.5; the output shown is the real output. Meera's tiffin app and all its ratings are made up.
The big idea: a recommender system picks, for each person, the few items they are most likely to want. Sorting by average stars puts a dish with one 5-star rating on top; the popularity baseline (most ratings) shows Chicken Biryani to everyone, even a vegetarian who gave it 1 star. Arranging ratings as a user-item matrix shows the real job: fill the empty cells of a sparse matrix. Content-based filtering matches dish tags to a user's taste profile (Asha → Rajma Chawal, cosine 0.80). Collaborative filtering uses other people's ratings: user-based (Asha's nearest neighbour Ravi, 0.84 → Rajma Chawal 4.19 stars) and item-based. On 500 made-up customers, guessing hidden ratings: global average MAE 0.873, dish average 0.899, content-based 0.839, user-based 0.662, item-based 0.642. A new user or new dish has no history: the cold-start problem (similarity = nan).
Sources: P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom and J. Riedl, "GroupLens: An Open Architecture for Collaborative Filtering of Netnews", ACM CSCW Conference, Chapel Hill, 1994 (paper page); G. Linden, B. Smith and J. York, "Amazon.com recommendations: item-to-item collaborative filtering", IEEE Internet Computing 7(1):76-80, 2003 (Crossref); Y. Koren, R. Bell and C. Volinsky, "Matrix Factorization Techniques for Recommender Systems", Computer (IEEE), August 2009, pp. 42-49, for the Netflix Prize data size (PDF).
Words you need first
- Vector, dot product, norm, cosine (Episode 3): a list of numbers; multiply pairwise and add; the length of a vector; dot product ÷ (length × length), 1 = same direction, 0 = nothing in common, −1 = opposite.
- Matrix (Episode 4): a table of numbers with rows and columns.
- Mean, Pearson correlation (Episode 11): the average; how strongly two lists rise and fall together (−1 to 1).
- MAE (Episode 19): mean absolute error, the average size of a miss. Guesses 4 and 2 for real ratings 5 and 3 → MAE 1.
- Train/test split (Episode 15): learn from some data, judge on data the model never saw.
- Baseline: the simplest method; any real model must beat it.
pivot: pandas, turns a long list of (row, column, value) into a table.
The real-life hook: Meera's tiffin app
Meera's app lists 60 dishes. Customers scroll, get tired, and leave without ordering. She wants the top of the screen to show five dishes this person will probably love. Software that does this is a recommender system.
What can the app learn from? Feedback
| Kind | What it is | Examples | Good | Bad |
|---|---|---|---|---|
| Explicit feedback | the user tells you | 1-5 stars, like, thumbs down | clear meaning | rare: most people never rate |
| Implicit feedback | you watch what they do | orders, clicks, re-orders, time spent | plentiful | noisy: one order isn't love, no order isn't hate |
Today we use explicit star ratings.
These lines create the toy data (TOY), 22 ratings from 6 customers:
import numpy as np, pandas as pd
R = pd.DataFrame([("Asha", "Masala Dosa", 5), ("Asha", "Idli Sambar", 5), ("Asha", "Paneer Tikka", 4),
("Ravi", "Masala Dosa", 4), ("Ravi", "Idli Sambar", 5), ("Ravi", "Paneer Tikka", 4), ("Ravi", "Rajma Chawal", 5),
("Ravi", "Chicken Biryani", 2), ("Priya", "Masala Dosa", 2), ("Priya", "Idli Sambar", 1), ("Priya", "Paneer Tikka", 2),
("Priya", "Chicken Biryani", 5), ("Priya", "Rajma Chawal", 3), ("Arjun", "Masala Dosa", 1), ("Arjun", "Idli Sambar", 2),
("Arjun", "Paneer Tikka", 3), ("Arjun", "Chicken Biryani", 5), ("Neha", "Chicken Biryani", 4), ("Neha", "Rasmalai", 5),
("Vikram", "Chicken Biryani", 4), ("Vikram", "Rajma Chawal", 2), ("Asha", "Chicken Biryani", 1)],
columns=["user", "dish", "stars"]) # 22 ratings, 1 to 5 stars
pd.set_option("display.width", 130, "display.max_columns", None)
The obvious attempt and the popularity baseline
# TOY, then:
by_dish = R.groupby("dish")["stars"].agg(["mean", "count"])
print(by_dish.sort_values("mean", ascending=False).round(2)) # obvious attempt: highest average first
print("most popular:", by_dish["count"].idxmax()) # popularity baseline: most ratings
Output:
mean count
dish
Rasmalai 5.00 1
Chicken Biryani 3.50 6
Rajma Chawal 3.33 3
Idli Sambar 3.25 4
Paneer Tikka 3.25 4
Masala Dosa 3.00 4
most popular: Chicken Biryani
- Sorting by average puts Rasmalai first with one rating. An average of one rating is luck.
- The popularity baseline recommends what most people chose (most ratings) or what is rated highest among items with enough ratings. Here: Chicken Biryani, for everyone, including Asha, a vegetarian who gave it 1 star.
- It is not personal, but keep it: it is the bar to beat and the fallback for new users.
The user-item matrix and sparsity
# TOY, then:
M = R.pivot(index="user", columns="dish", values="stars") # the user-item matrix
print(M.rename(columns=lambda d: "Biryani" if d.startswith("Chicken") else d.split()[0])) # short names to fit
print("empty cells:", M.isna().sum().sum(), "of", M.size, f"= {M.isna().mean().mean():.0%}")
Output:
dish Biryani Idli Masala Paneer Rajma Rasmalai
user
Arjun 5.0 2.0 1.0 3.0 NaN NaN
Asha 1.0 5.0 5.0 4.0 NaN NaN
Neha 4.0 NaN NaN NaN NaN 5.0
Priya 5.0 1.0 2.0 2.0 3.0 NaN
Ravi 2.0 5.0 4.0 4.0 5.0 NaN
Vikram 4.0 NaN NaN NaN 2.0 NaN
empty cells: 14 of 36 = 39%
- User-item matrix: one row per user, one column per item, the rating in each cell, empty (NaN) where there is none.
- Recommending = guessing the empty cells.
- Sparsity: the share of empty cells. Our toy is 39% empty. The Netflix Prize data had more than 100 million ratings from about 500,000 customers on more than 17,000 movies: about 8.5 billion cells, roughly 1% filled, 99% empty.
Content-based filtering
- Describe each item with features (here 4 tags, 1 or 0: spicy, veg, sweet, south).
- The user's profile = the average features of the items they rated 4 or 5.
- Score each unseen item by its similarity to the profile and recommend the highest. We use cosine similarity.
# TOY, then:
tags = pd.DataFrame({"spicy": [0, 0, 1, 1, 0, 0], "veg": [1, 1, 1, 0, 1, 1], "sweet": [0, 0, 0, 0, 0, 1],
"south": [1, 1, 0, 0, 0, 0]}, # content: what each dish IS
index=["Masala Dosa", "Idli Sambar", "Paneer Tikka", "Chicken Biryani", "Rajma Chawal", "Rasmalai"])
liked = R[(R.user == "Asha") & (R.stars >= 4)].dish # dishes Asha gave 4 or 5 stars
profile = tags.loc[liked].mean() # her taste profile: average of their tags
print("Asha:", profile.round(2).to_dict())
unseen = tags.drop(R[R.user == "Asha"].dish) # dishes she hasn't rated yet
cos = unseen @ profile / (np.linalg.norm(unseen, axis=1) * np.linalg.norm(profile)) # cosine similarity
print(cos.sort_values(ascending=False).round(2))
Output:
Asha: {'spicy': 0.33, 'veg': 1.0, 'sweet': 0.0, 'south': 0.67}
Rajma Chawal 0.80
Rasmalai 0.57
dtype: float64
| Strength | Weakness |
|---|---|
| Works for a new item with no ratings (only its features are needed) | Someone must describe every item |
| Easy to explain: "because you like veg South Indian" | Features miss taste and quality |
| Recommends more of the same (no surprises) |
Collaborative filtering
Uses only the user-item matrix, no item features: "people like you liked this". The idea goes back to GroupLens (1994), which recommended news articles from other readers' ratings.
User-based
- Centre each user's ratings (subtract their own average): some people rate everything high.
- Similarity between users = cosine of the centred rows (close to Pearson correlation).
- Predict = the user's average + the similarity-weighted average of the neighbours' centred ratings for that item.
# TOY, then:
M = R.pivot(index="user", columns="dish", values="stars")
C = M.sub(M.mean(axis=1), axis=0).fillna(0) # centre each user on their own average
sim = (C @ C.T) / np.outer(np.linalg.norm(C, axis=1), np.linalg.norm(C, axis=1)) # cosine between users
print(sim["Asha"].drop("Asha").round(2).sort_values(ascending=False))
who = M["Rajma Chawal"].dropna().index # people who rated Rajma Chawal
s = sim.loc["Asha", who]
guess = M.loc["Asha"].mean() + (s * C.loc[who, "Rajma Chawal"]).sum() / s.abs().sum()
print("Asha's predicted stars for Rajma Chawal:", round(guess, 2))
Output:
user
Ravi 0.84
Neha 0.59
Vikram -0.59
Arjun -0.95
Priya -0.96
Name: Asha, dtype: float64
Asha's predicted stars for Rajma Chawal: 4.19
Working it out for Asha (average 3.75) and Rajma Chawal:
| Neighbour | Similarity | Their centred rating of Rajma Chawal | Product |
|---|---|---|---|
| Ravi | 0.84 | 5 − 4.0 = +1.0 | +0.84 |
| Priya | −0.96 | 3 − 2.6 = +0.4 | −0.38 |
| Vikram | −0.59 | 2 − 3.0 = −1.0 | +0.59 |
(0.84 − 0.38 + 0.59) / (0.84 + 0.96 + 0.59) = 0.44, so 3.75 + 0.44 = 4.19 stars. A neighbour with the opposite taste who disliked the dish counts as a vote for it.
Item-based
Two items are similar if the same people rate them alike (cosine between centred columns). To predict Asha's rating for a dish, look at her ratings of the dishes most similar to it. Shops usually have far fewer items than customers, and item similarities change slowly, so they can be computed ahead of time. Amazon described this item-to-item collaborative filtering in 2003.
A fair scoreboard on 500 customers
A bigger made-up app: 500 customers, 60 dishes, hidden tastes we built in, tags that capture only part of them. We hide 20% of the ratings and guess them. These lines are BIG:
import numpy as np, pandas as pd; r = np.random.default_rng(53) # Meera's tiffin app: made-up ratings
U, I = 500, 60; F, P = r.normal(0, 1, (I, 2)), r.normal(0, 1, (U, 2)) # 500 customers, 60 dishes, hidden tastes
tags = pd.DataFrame(np.c_[F + r.normal(0, 0.7, (I, 2)), r.normal(0, 1, (I, 2))] > 0, dtype=int,
columns=["spicy", "veg", "sweet", "south"], index=[f"dish{i:02}" for i in range(I)])
true = 3 + r.normal(0, 0.3, (U, 1)) + r.normal(0, 0.4, (1, I)) + 0.9 * P @ F.T
R = pd.DataFrame([(u, tags.index[j], int(np.clip(round(true[u, j] + r.normal(0, 0.3)), 1, 5))) for u in range(U)
for j in r.choice(I, r.poisson(12) + 1, replace=False)], columns=["user", "dish", "stars"])
test = R.sample(frac=0.2, random_state=0); train = R.drop(test.index) # hide 20% of the ratings
M = train.pivot(index="user", columns="dish", values="stars").reindex(index=range(U), columns=tags.index)
mu, um = train.stars.mean(), M.mean(axis=1); mae = lambda p: np.abs(np.clip(p, 1, 5) - test.stars.values).mean().round(3)
# BIG, then:
print(len(R), "ratings;", f"user-item matrix {M.shape}, {M.isna().mean().mean():.1%} empty")
print("global average: MAE", mae(np.full(len(test), mu)))
print("dish average: MAE", mae(M.mean()[test.dish].fillna(mu).values)) # the popularity-style baseline
def content(u, j): # learn how much user u likes each tag
d = train[train.user == u]; X = tags.loc[d.dish].values
w = np.linalg.lstsq(X - X.mean(0), d.stars.values - um[u], rcond=None)[0]
return um[u] + (tags.loc[j].values - X.mean(0)) @ w * 0.5
print("content-based: MAE", mae([content(u, j) for u, j in zip(test.user, test.dish)]))
Output:
6452 ratings; user-item matrix (500, 60), 82.8% empty
global average: MAE 0.873
dish average: MAE 0.899
content-based: MAE 0.839
# BIG, then:
C = M.sub(um, axis=0).fillna(0).values # centred matrix, empty = 0
def cosine(A): n = np.linalg.norm(A, axis=1) + 1e-9; S = A @ A.T / np.outer(n, n); np.fill_diagonal(S, 0); return S
Su, Si = cosine(C), cosine(C.T); col = {d: k for k, d in enumerate(M.columns)} # user-user, dish-dish similarity
def predict(u, j, user_based, k=20):
s, x = (Su[u], C[:, col[j]]) if user_based else (Si[col[j]], C[u]) # similar users who rated j / dishes u rated
s = np.where(x != 0, s, 0); top = np.argsort(-np.abs(s))[:k]
return um[u] + (s[top] * x[top]).sum() / (np.abs(s[top]).sum() + 1e-9)
for name, ub in [("user-based CF", True), ("item-based CF", False)]:
print(f"{name}: MAE", mae([predict(u, j, ub) for u, j in zip(test.user, test.dish)]))
Output:
user-based CF: MAE 0.662
item-based CF: MAE 0.642
| Method | MAE (stars) |
|---|---|
| Global average | 0.873 |
| Dish average (popularity-style) | 0.899 |
| Content-based (tags) | 0.839 |
| User-based collaborative filtering (k = 20) | 0.662 |
| Item-based collaborative filtering (k = 20) | 0.642 |
The cold-start problem
# BIG, then:
C = M.sub(um, axis=0).fillna(0).values
kabir = np.zeros(I) # new user Kabir: no ratings yet
sim = C @ kabir / (np.linalg.norm(C, axis=1) * np.linalg.norm(kabir)) # how similar is he to everyone?
print("Kabir vs the first 5 users:", sim[:5])
lassi = pd.Series({"spicy": 0, "veg": 1, "sweet": 1, "south": 0}) # new dish Mango Lassi: tags known, 0 ratings
print("Mango Lassi has 0 ratings: CF has nothing to compare")
likes = train[train.stars >= 4].merge(tags, left_on="dish", right_index=True).groupby("user")[list(tags.columns)].mean()
match = likes @ lassi / (np.linalg.norm(likes, axis=1) * np.linalg.norm(lassi)) # content-based still works
print("content matches for Mango Lassi:", list(match.nlargest(5).index))
Output:
Kabir vs the first 5 users: [nan nan nan nan nan]
Mango Lassi has 0 ratings: CF has nothing to compare
content matches for Mango Lassi: [10, 23, 43, 159, 387]
main.py:13: RuntimeWarning: invalid value encountered in divide
- A new user (Kabir) has an all-zero vector: its length is 0, cosine divides by zero →
nanandRuntimeWarning: invalid value encountered in divide. - A new item (Mango Lassi) has no ratings: nobody to compare.
- Cold-start problem: no history → collaborative filtering can't recommend for, or recommend, the newcomer.
- Fixes: the popularity baseline for new users; ask at sign-up ("pick 3 dishes you like"); content-based for new items; switch to collaborative filtering as ratings arrive. Combining methods is a hybrid recommender (Episode 54).
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Rank by average stars | Rasmalai 5.0 from 1 rating on top | tiny counts are luck | require a minimum count |
| Same list for everyone | Chicken Biryani for a vegetarian | not personal | content-based or collaborative filtering |
| Raw (uncentred) ratings in CF | MAE 0.818 / 0.887 instead of 0.662 / 0.642 (Practice 2) | generous raters look similar to everyone | subtract each user's average |
| CF for a brand-new user | RuntimeWarning: invalid value encountered in divide, similarity nan |
zero-length vector | popularity, onboarding questions |
| Only content tags | MAE 0.839 | tags miss taste | collaborative filtering |
| Wide table printed by pandas | columns replaced by ... |
display width | pd.set_option("display.width", 130, "display.max_columns", None) |
Mini project: a top 5 for one customer
Recommend five dishes for customer 7 with item-based collaborative filtering.
# BIG, then the Q6 helper lines (C, cosine, Su, Si, col, predict), then:
u = 7; seen = set(train.dish[train.user == u])
scores = {j: predict(u, j, False) for j in M.columns if j not in seen} # item-based guess for every unseen dish
top5 = sorted(scores, key=scores.get, reverse=True)[:5]
print("user 7 rated:", train[train.user == u].sort_values("stars", ascending=False).values[:, 1:].tolist())
print("top 5 for user 7:", [(j, float(round(scores[j], 2))) for j in top5])
Output:
user 7 rated: [['dish52', 5], ['dish22', 5], ['dish55', 5], ['dish57', 5], ['dish18', 4], ['dish05', 4], ['dish31', 3], ['dish06', 3], ['dish39', 3], ['dish09', 2], ['dish17', 1], ['dish28', 1]]
top 5 for user 7: [('dish23', 4.94), ('dish36', 4.77), ('dish54', 4.75), ('dish32', 4.73), ('dish45', 4.7)]
Practice
1. Change the number of neighbours k to 5 and 100.
Solution
Replace the last loop of the Q6 program with:
for k in [5, 20, 100]:
print(f"k = {k:3}: user-based", mae([predict(u, j, True, k) for u, j in zip(test.user, test.dish)]),
" item-based", mae([predict(u, j, False, k) for u, j in zip(test.user, test.dish)]))
k = 5: user-based 0.692 item-based 0.643
k = 20: user-based 0.662 item-based 0.642
k = 100: user-based 0.678 item-based 0.642
User-based is best around k = 20: too few neighbours is noisy, too many adds dissimilar people. Item-based barely changes because each user rated only about 13 dishes, so there are few neighbours to choose from.
2. Skip the centring: use raw stars (empty = 0) and a plain weighted average.
Solution
Change C = M.sub(um, axis=0).fillna(0).values to C = M.fillna(0).values and return (s[top] * x[top]).sum() / (np.abs(s[top]).sum() + 1e-9):
user-based CF: MAE 0.818
item-based CF: MAE 0.887
Much worse: with raw stars every user looks similar to everyone (all ratings are positive), and empty cells count as 0 stars.
3. Rank the toy dishes by average, but only those with at least 3 ratings.
Solution
# TOY, then:
by_dish = R.groupby("dish")["stars"].agg(["mean", "count"])
print(by_dish[by_dish["count"] >= 3].sort_values("mean", ascending=False).round(2)) # only dishes with 3+ ratings
mean count
dish
Chicken Biryani 3.50 6
Rajma Chawal 3.33 3
Idli Sambar 3.25 4
Paneer Tikka 3.25 4
Masala Dosa 3.00 4
Rasmalai (1 rating) disappears from the list.
Quick quiz
A new book arrives in an online shop. Nobody has bought or rated it yet; its genre, author and length are known. Which method can recommend it today? A) User-based CF B) Item-based CF C) Content-based D) None
Answer
C, content-based. With zero ratings, both kinds of collaborative filtering have nothing to compare; the features are known, so it can be matched to readers' profiles.
More questions:
- Is "watched 90% of a video" explicit or implicit feedback?
Answer
Implicit: the user did not tell you, you observed it. - A matrix of 1,000 users × 200 items has 6,000 ratings. How sparse is it?
Answer
6,000 / 200,000 = 3% filled, so 97% empty. - Why centre ratings before computing user similarity?
Answer
So a generous rater's 4 and a strict rater's 4 are compared by what they mean relative to each person's own average. - Two users have similarity −0.9 and one of them hated a dish. What does user-based CF predict for the other?
Answer
Above their average: a negative similarity times a negative centred rating gives a positive contribution.
Interview questions
- Content-based vs collaborative filtering? Content-based uses item features and the user's own history; collaborative uses the ratings of many users and no item features. Content handles new items and explains itself; collaborative finds tastes features miss but needs history.
- User-based vs item-based CF? User-based finds similar users; item-based finds similar items. Item-based is often preferred when there are far more users than items, because item similarities are fewer and more stable, so they can be precomputed (Amazon, 2003).
- What is the cold-start problem and how do you handle it? New users or items have no interactions. Use popularity, onboarding questions, content features, or a hybrid, and switch to CF as data arrives.
- Why is the user-item matrix sparse, and why does it matter? Each user interacts with a tiny fraction of items; similarities rest on few overlapping ratings and dense storage is wasteful.
- Explicit vs implicit feedback: which is more common? Implicit: every visit produces clicks and views, few users rate. It is noisy and has no negatives.
- Which similarity measures are used? Cosine, cosine on centred ratings (adjusted cosine), Pearson correlation, Jaccard for sets of interactions.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Recommender system | Software that picks, for each person, the items they're most likely to want: "you may also like" |
| Item | Anything recommended: a dish, a movie, a book |
| Explicit feedback | The user tells you their opinion: Asha gives Masala Dosa 5 stars |
| Implicit feedback | You infer it from behaviour: Asha ordered Masala Dosa three times |
| User-item matrix | Table with a row per user, a column per item, ratings in the cells |
| Sparsity | Share of empty cells; the Netflix Prize data was about 99% empty |
| Sparse matrix | A matrix where most cells are empty |
| Popularity baseline | Recommend the most-rated (or best-rated with enough ratings) items to everyone |
| Content-based filtering | Recommend items whose features match the user's profile |
| Item features | Descriptions of an item: tags like spicy, veg, sweet, south |
| User profile | Average features of the items the user liked: veg 1.0, south 0.67 |
| Collaborative filtering (CF) | Recommend from other people's ratings: "people like you liked this" |
| User-based CF | Find the most similar users (neighbours), predict from their ratings |
| Item-based CF | Find items rated alike by the same people, predict from the user's ratings of them |
| Neighbour | One of the k most similar users (or items) |
| Similarity (for recommendations) | A number saying how alike two users or items are; cosine here |
| Cosine similarity | a · b / (|a| × |b|): 1 same direction, 0 unrelated, −1 opposite |
| Centring | Subtracting each user's average rating, so +1 means "more than I usually like" |
| Cold-start problem | A new user or item has no history, so CF can't use it |
| Onboarding | Asking a new user a few questions at sign-up ("pick 3 dishes") |
| Hybrid recommender | Combines content-based and collaborative methods |
| MAE | Mean absolute error: average size of the miss, here in stars |
| NaN | "Not a number": a missing value, or the result of 0 / 0 |
Summary
- A recommender system fills the empty cells of a sparse user-item matrix; feedback is explicit (stars) or implicit (orders, clicks).
- The popularity baseline isn't personal; averages of tiny counts are luck.
- Content-based matches item features to a user profile with cosine similarity: works for new items, plays it safe, needs good tags (MAE 0.839).
- Collaborative filtering learns from similar users (0.662) or similar items (0.642), with ratings centred per user.
- The cold-start problem: no history, no CF (
nan). Use popularity, onboarding and content features, then a hybrid.
Next episode: Recommender systems 2: matrix factorisation, latent factors, user and item embeddings, ALS and SGD, hybrid recommenders, precision@k, recall@k, MAP, nDCG, and offline vs online evaluation.