SyntaxPilot
Courses/Machine Learning/Lesson 53
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 53 of 90

Recommender systems 1

recommender systemexplicit vs implicit feedbackuser-item matrixsparsitypopularity baselinecontent-based filteringcollaborative filteringuser-based vs item-basedsimilarity for recommendationscold-start problem

Machine Learning Full Course · Episode 53. Every code sample below was run with Python 3.13, NumPy 2.5.3 and pandas 3.0.5; the output shown is the real output. Meera's tiffin app and all its ratings are made up.

The big idea: a recommender system picks, for each person, the few items they are most likely to want. Sorting by average stars puts a dish with one 5-star rating on top; the popularity baseline (most ratings) shows Chicken Biryani to everyone, even a vegetarian who gave it 1 star. Arranging ratings as a user-item matrix shows the real job: fill the empty cells of a sparse matrix. Content-based filtering matches dish tags to a user's taste profile (Asha → Rajma Chawal, cosine 0.80). Collaborative filtering uses other people's ratings: user-based (Asha's nearest neighbour Ravi, 0.84 → Rajma Chawal 4.19 stars) and item-based. On 500 made-up customers, guessing hidden ratings: global average MAE 0.873, dish average 0.899, content-based 0.839, user-based 0.662, item-based 0.642. A new user or new dish has no history: the cold-start problem (similarity = nan).

Sources: P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom and J. Riedl, "GroupLens: An Open Architecture for Collaborative Filtering of Netnews", ACM CSCW Conference, Chapel Hill, 1994 (paper page); G. Linden, B. Smith and J. York, "Amazon.com recommendations: item-to-item collaborative filtering", IEEE Internet Computing 7(1):76-80, 2003 (Crossref); Y. Koren, R. Bell and C. Volinsky, "Matrix Factorization Techniques for Recommender Systems", Computer (IEEE), August 2009, pp. 42-49, for the Netflix Prize data size (PDF).

Words you need first

  • Vector, dot product, norm, cosine (Episode 3): a list of numbers; multiply pairwise and add; the length of a vector; dot product ÷ (length × length), 1 = same direction, 0 = nothing in common, −1 = opposite.
  • Matrix (Episode 4): a table of numbers with rows and columns.
  • Mean, Pearson correlation (Episode 11): the average; how strongly two lists rise and fall together (−1 to 1).
  • MAE (Episode 19): mean absolute error, the average size of a miss. Guesses 4 and 2 for real ratings 5 and 3 → MAE 1.
  • Train/test split (Episode 15): learn from some data, judge on data the model never saw.
  • Baseline: the simplest method; any real model must beat it.
  • pivot: pandas, turns a long list of (row, column, value) into a table.

The real-life hook: Meera's tiffin app

Meera's app lists 60 dishes. Customers scroll, get tired, and leave without ordering. She wants the top of the screen to show five dishes this person will probably love. Software that does this is a recommender system.

What can the app learn from? Feedback

Kind What it is Examples Good Bad
Explicit feedback the user tells you 1-5 stars, like, thumbs down clear meaning rare: most people never rate
Implicit feedback you watch what they do orders, clicks, re-orders, time spent plentiful noisy: one order isn't love, no order isn't hate

Today we use explicit star ratings.

These lines create the toy data (TOY), 22 ratings from 6 customers:

import numpy as np, pandas as pd
R = pd.DataFrame([("Asha", "Masala Dosa", 5), ("Asha", "Idli Sambar", 5), ("Asha", "Paneer Tikka", 4),
    ("Ravi", "Masala Dosa", 4), ("Ravi", "Idli Sambar", 5), ("Ravi", "Paneer Tikka", 4), ("Ravi", "Rajma Chawal", 5),
    ("Ravi", "Chicken Biryani", 2), ("Priya", "Masala Dosa", 2), ("Priya", "Idli Sambar", 1), ("Priya", "Paneer Tikka", 2),
    ("Priya", "Chicken Biryani", 5), ("Priya", "Rajma Chawal", 3), ("Arjun", "Masala Dosa", 1), ("Arjun", "Idli Sambar", 2),
    ("Arjun", "Paneer Tikka", 3), ("Arjun", "Chicken Biryani", 5), ("Neha", "Chicken Biryani", 4), ("Neha", "Rasmalai", 5),
    ("Vikram", "Chicken Biryani", 4), ("Vikram", "Rajma Chawal", 2), ("Asha", "Chicken Biryani", 1)],
    columns=["user", "dish", "stars"])                               # 22 ratings, 1 to 5 stars
pd.set_option("display.width", 130, "display.max_columns", None)

The obvious attempt and the popularity baseline

# TOY, then:
by_dish = R.groupby("dish")["stars"].agg(["mean", "count"])
print(by_dish.sort_values("mean", ascending=False).round(2))           # obvious attempt: highest average first
print("most popular:", by_dish["count"].idxmax())                       # popularity baseline: most ratings

Output:

                 mean  count
dish                        
Rasmalai         5.00      1
Chicken Biryani  3.50      6
Rajma Chawal     3.33      3
Idli Sambar      3.25      4
Paneer Tikka     3.25      4
Masala Dosa      3.00      4
most popular: Chicken Biryani
  • Sorting by average puts Rasmalai first with one rating. An average of one rating is luck.
  • The popularity baseline recommends what most people chose (most ratings) or what is rated highest among items with enough ratings. Here: Chicken Biryani, for everyone, including Asha, a vegetarian who gave it 1 star.
  • It is not personal, but keep it: it is the bar to beat and the fallback for new users.

The user-item matrix and sparsity

# TOY, then:
M = R.pivot(index="user", columns="dish", values="stars")             # the user-item matrix
print(M.rename(columns=lambda d: "Biryani" if d.startswith("Chicken") else d.split()[0]))   # short names to fit
print("empty cells:", M.isna().sum().sum(), "of", M.size, f"= {M.isna().mean().mean():.0%}")

Output:

dish    Biryani  Idli  Masala  Paneer  Rajma  Rasmalai
user                                                  
Arjun       5.0   2.0     1.0     3.0    NaN       NaN
Asha        1.0   5.0     5.0     4.0    NaN       NaN
Neha        4.0   NaN     NaN     NaN    NaN       5.0
Priya       5.0   1.0     2.0     2.0    3.0       NaN
Ravi        2.0   5.0     4.0     4.0    5.0       NaN
Vikram      4.0   NaN     NaN     NaN    2.0       NaN
empty cells: 14 of 36 = 39%
  • User-item matrix: one row per user, one column per item, the rating in each cell, empty (NaN) where there is none.
  • Recommending = guessing the empty cells.
  • Sparsity: the share of empty cells. Our toy is 39% empty. The Netflix Prize data had more than 100 million ratings from about 500,000 customers on more than 17,000 movies: about 8.5 billion cells, roughly 1% filled, 99% empty.

Content-based filtering

  1. Describe each item with features (here 4 tags, 1 or 0: spicy, veg, sweet, south).
  2. The user's profile = the average features of the items they rated 4 or 5.
  3. Score each unseen item by its similarity to the profile and recommend the highest. We use cosine similarity.
# TOY, then:
tags = pd.DataFrame({"spicy": [0, 0, 1, 1, 0, 0], "veg": [1, 1, 1, 0, 1, 1], "sweet": [0, 0, 0, 0, 0, 1],
    "south": [1, 1, 0, 0, 0, 0]},                                    # content: what each dish IS
    index=["Masala Dosa", "Idli Sambar", "Paneer Tikka", "Chicken Biryani", "Rajma Chawal", "Rasmalai"])
liked = R[(R.user == "Asha") & (R.stars >= 4)].dish                  # dishes Asha gave 4 or 5 stars
profile = tags.loc[liked].mean()                                     # her taste profile: average of their tags
print("Asha:", profile.round(2).to_dict())
unseen = tags.drop(R[R.user == "Asha"].dish)                         # dishes she hasn't rated yet
cos = unseen @ profile / (np.linalg.norm(unseen, axis=1) * np.linalg.norm(profile))   # cosine similarity
print(cos.sort_values(ascending=False).round(2))

Output:

Asha: {'spicy': 0.33, 'veg': 1.0, 'sweet': 0.0, 'south': 0.67}
Rajma Chawal    0.80
Rasmalai        0.57
dtype: float64
Strength Weakness
Works for a new item with no ratings (only its features are needed) Someone must describe every item
Easy to explain: "because you like veg South Indian" Features miss taste and quality
Recommends more of the same (no surprises)

Collaborative filtering

Uses only the user-item matrix, no item features: "people like you liked this". The idea goes back to GroupLens (1994), which recommended news articles from other readers' ratings.

User-based

  1. Centre each user's ratings (subtract their own average): some people rate everything high.
  2. Similarity between users = cosine of the centred rows (close to Pearson correlation).
  3. Predict = the user's average + the similarity-weighted average of the neighbours' centred ratings for that item.
# TOY, then:
M = R.pivot(index="user", columns="dish", values="stars")
C = M.sub(M.mean(axis=1), axis=0).fillna(0)                          # centre each user on their own average
sim = (C @ C.T) / np.outer(np.linalg.norm(C, axis=1), np.linalg.norm(C, axis=1))   # cosine between users
print(sim["Asha"].drop("Asha").round(2).sort_values(ascending=False))
who = M["Rajma Chawal"].dropna().index                               # people who rated Rajma Chawal
s = sim.loc["Asha", who]
guess = M.loc["Asha"].mean() + (s * C.loc[who, "Rajma Chawal"]).sum() / s.abs().sum()
print("Asha's predicted stars for Rajma Chawal:", round(guess, 2))

Output:

user
Ravi      0.84
Neha      0.59
Vikram   -0.59
Arjun    -0.95
Priya    -0.96
Name: Asha, dtype: float64
Asha's predicted stars for Rajma Chawal: 4.19

Working it out for Asha (average 3.75) and Rajma Chawal:

Neighbour Similarity Their centred rating of Rajma Chawal Product
Ravi 0.84 5 − 4.0 = +1.0 +0.84
Priya −0.96 3 − 2.6 = +0.4 −0.38
Vikram −0.59 2 − 3.0 = −1.0 +0.59

(0.84 − 0.38 + 0.59) / (0.84 + 0.96 + 0.59) = 0.44, so 3.75 + 0.44 = 4.19 stars. A neighbour with the opposite taste who disliked the dish counts as a vote for it.

Item-based

Two items are similar if the same people rate them alike (cosine between centred columns). To predict Asha's rating for a dish, look at her ratings of the dishes most similar to it. Shops usually have far fewer items than customers, and item similarities change slowly, so they can be computed ahead of time. Amazon described this item-to-item collaborative filtering in 2003.

A fair scoreboard on 500 customers

A bigger made-up app: 500 customers, 60 dishes, hidden tastes we built in, tags that capture only part of them. We hide 20% of the ratings and guess them. These lines are BIG:

import numpy as np, pandas as pd; r = np.random.default_rng(53)   # Meera's tiffin app: made-up ratings
U, I = 500, 60; F, P = r.normal(0, 1, (I, 2)), r.normal(0, 1, (U, 2))   # 500 customers, 60 dishes, hidden tastes
tags = pd.DataFrame(np.c_[F + r.normal(0, 0.7, (I, 2)), r.normal(0, 1, (I, 2))] > 0, dtype=int,
                    columns=["spicy", "veg", "sweet", "south"], index=[f"dish{i:02}" for i in range(I)])
true = 3 + r.normal(0, 0.3, (U, 1)) + r.normal(0, 0.4, (1, I)) + 0.9 * P @ F.T
R = pd.DataFrame([(u, tags.index[j], int(np.clip(round(true[u, j] + r.normal(0, 0.3)), 1, 5))) for u in range(U)
                  for j in r.choice(I, r.poisson(12) + 1, replace=False)], columns=["user", "dish", "stars"])
test = R.sample(frac=0.2, random_state=0); train = R.drop(test.index)   # hide 20% of the ratings
M = train.pivot(index="user", columns="dish", values="stars").reindex(index=range(U), columns=tags.index)
mu, um = train.stars.mean(), M.mean(axis=1); mae = lambda p: np.abs(np.clip(p, 1, 5) - test.stars.values).mean().round(3)
# BIG, then:
print(len(R), "ratings;", f"user-item matrix {M.shape}, {M.isna().mean().mean():.1%} empty")
print("global average:   MAE", mae(np.full(len(test), mu)))
print("dish average:     MAE", mae(M.mean()[test.dish].fillna(mu).values))   # the popularity-style baseline
def content(u, j):                                                   # learn how much user u likes each tag
    d = train[train.user == u]; X = tags.loc[d.dish].values
    w = np.linalg.lstsq(X - X.mean(0), d.stars.values - um[u], rcond=None)[0]
    return um[u] + (tags.loc[j].values - X.mean(0)) @ w * 0.5
print("content-based:    MAE", mae([content(u, j) for u, j in zip(test.user, test.dish)]))

Output:

6452 ratings; user-item matrix (500, 60), 82.8% empty
global average:   MAE 0.873
dish average:     MAE 0.899
content-based:    MAE 0.839
# BIG, then:
C = M.sub(um, axis=0).fillna(0).values                               # centred matrix, empty = 0
def cosine(A): n = np.linalg.norm(A, axis=1) + 1e-9; S = A @ A.T / np.outer(n, n); np.fill_diagonal(S, 0); return S
Su, Si = cosine(C), cosine(C.T); col = {d: k for k, d in enumerate(M.columns)}   # user-user, dish-dish similarity
def predict(u, j, user_based, k=20):
    s, x = (Su[u], C[:, col[j]]) if user_based else (Si[col[j]], C[u])   # similar users who rated j / dishes u rated
    s = np.where(x != 0, s, 0); top = np.argsort(-np.abs(s))[:k]
    return um[u] + (s[top] * x[top]).sum() / (np.abs(s[top]).sum() + 1e-9)
for name, ub in [("user-based CF", True), ("item-based CF", False)]:
    print(f"{name}:  MAE", mae([predict(u, j, ub) for u, j in zip(test.user, test.dish)]))

Output:

user-based CF:  MAE 0.662
item-based CF:  MAE 0.642
Method MAE (stars)
Global average 0.873
Dish average (popularity-style) 0.899
Content-based (tags) 0.839
User-based collaborative filtering (k = 20) 0.662
Item-based collaborative filtering (k = 20) 0.642

The cold-start problem

# BIG, then:
C = M.sub(um, axis=0).fillna(0).values
kabir = np.zeros(I)                                                  # new user Kabir: no ratings yet
sim = C @ kabir / (np.linalg.norm(C, axis=1) * np.linalg.norm(kabir))   # how similar is he to everyone?
print("Kabir vs the first 5 users:", sim[:5])
lassi = pd.Series({"spicy": 0, "veg": 1, "sweet": 1, "south": 0})   # new dish Mango Lassi: tags known, 0 ratings
print("Mango Lassi has 0 ratings: CF has nothing to compare")
likes = train[train.stars >= 4].merge(tags, left_on="dish", right_index=True).groupby("user")[list(tags.columns)].mean()
match = likes @ lassi / (np.linalg.norm(likes, axis=1) * np.linalg.norm(lassi))   # content-based still works
print("content matches for Mango Lassi:", list(match.nlargest(5).index))

Output:

Kabir vs the first 5 users: [nan nan nan nan nan]
Mango Lassi has 0 ratings: CF has nothing to compare
content matches for Mango Lassi: [10, 23, 43, 159, 387]
main.py:13: RuntimeWarning: invalid value encountered in divide
  • A new user (Kabir) has an all-zero vector: its length is 0, cosine divides by zero → nan and RuntimeWarning: invalid value encountered in divide.
  • A new item (Mango Lassi) has no ratings: nobody to compare.
  • Cold-start problem: no history → collaborative filtering can't recommend for, or recommend, the newcomer.
  • Fixes: the popularity baseline for new users; ask at sign-up ("pick 3 dishes you like"); content-based for new items; switch to collaborative filtering as ratings arrive. Combining methods is a hybrid recommender (Episode 54).

Common mistakes

Mistake Exact message / symptom Why Fix
Rank by average stars Rasmalai 5.0 from 1 rating on top tiny counts are luck require a minimum count
Same list for everyone Chicken Biryani for a vegetarian not personal content-based or collaborative filtering
Raw (uncentred) ratings in CF MAE 0.818 / 0.887 instead of 0.662 / 0.642 (Practice 2) generous raters look similar to everyone subtract each user's average
CF for a brand-new user RuntimeWarning: invalid value encountered in divide, similarity nan zero-length vector popularity, onboarding questions
Only content tags MAE 0.839 tags miss taste collaborative filtering
Wide table printed by pandas columns replaced by ... display width pd.set_option("display.width", 130, "display.max_columns", None)

Mini project: a top 5 for one customer

Recommend five dishes for customer 7 with item-based collaborative filtering.

# BIG, then the Q6 helper lines (C, cosine, Su, Si, col, predict), then:
u = 7; seen = set(train.dish[train.user == u])
scores = {j: predict(u, j, False) for j in M.columns if j not in seen}    # item-based guess for every unseen dish
top5 = sorted(scores, key=scores.get, reverse=True)[:5]
print("user 7 rated:", train[train.user == u].sort_values("stars", ascending=False).values[:, 1:].tolist())
print("top 5 for user 7:", [(j, float(round(scores[j], 2))) for j in top5])

Output:

user 7 rated: [['dish52', 5], ['dish22', 5], ['dish55', 5], ['dish57', 5], ['dish18', 4], ['dish05', 4], ['dish31', 3], ['dish06', 3], ['dish39', 3], ['dish09', 2], ['dish17', 1], ['dish28', 1]]
top 5 for user 7: [('dish23', 4.94), ('dish36', 4.77), ('dish54', 4.75), ('dish32', 4.73), ('dish45', 4.7)]

Practice

1. Change the number of neighbours k to 5 and 100.

Solution

Replace the last loop of the Q6 program with:

for k in [5, 20, 100]:
    print(f"k = {k:3}:  user-based", mae([predict(u, j, True, k) for u, j in zip(test.user, test.dish)]),
          "  item-based", mae([predict(u, j, False, k) for u, j in zip(test.user, test.dish)]))
k =   5:  user-based 0.692   item-based 0.643
k =  20:  user-based 0.662   item-based 0.642
k = 100:  user-based 0.678   item-based 0.642

User-based is best around k = 20: too few neighbours is noisy, too many adds dissimilar people. Item-based barely changes because each user rated only about 13 dishes, so there are few neighbours to choose from.

2. Skip the centring: use raw stars (empty = 0) and a plain weighted average.

Solution

Change C = M.sub(um, axis=0).fillna(0).values to C = M.fillna(0).values and return (s[top] * x[top]).sum() / (np.abs(s[top]).sum() + 1e-9):

user-based CF:  MAE 0.818
item-based CF:  MAE 0.887

Much worse: with raw stars every user looks similar to everyone (all ratings are positive), and empty cells count as 0 stars.

3. Rank the toy dishes by average, but only those with at least 3 ratings.

Solution
# TOY, then:
by_dish = R.groupby("dish")["stars"].agg(["mean", "count"])
print(by_dish[by_dish["count"] >= 3].sort_values("mean", ascending=False).round(2))   # only dishes with 3+ ratings
                 mean  count
dish                        
Chicken Biryani  3.50      6
Rajma Chawal     3.33      3
Idli Sambar      3.25      4
Paneer Tikka     3.25      4
Masala Dosa      3.00      4

Rasmalai (1 rating) disappears from the list.

Quick quiz

A new book arrives in an online shop. Nobody has bought or rated it yet; its genre, author and length are known. Which method can recommend it today? A) User-based CF B) Item-based CF C) Content-based D) None

Answer

C, content-based. With zero ratings, both kinds of collaborative filtering have nothing to compare; the features are known, so it can be matched to readers' profiles.

More questions:

  1. Is "watched 90% of a video" explicit or implicit feedback?
    AnswerImplicit: the user did not tell you, you observed it.
  2. A matrix of 1,000 users × 200 items has 6,000 ratings. How sparse is it?
    Answer6,000 / 200,000 = 3% filled, so 97% empty.
  3. Why centre ratings before computing user similarity?
    AnswerSo a generous rater's 4 and a strict rater's 4 are compared by what they mean relative to each person's own average.
  4. Two users have similarity −0.9 and one of them hated a dish. What does user-based CF predict for the other?
    AnswerAbove their average: a negative similarity times a negative centred rating gives a positive contribution.

Interview questions

  1. Content-based vs collaborative filtering? Content-based uses item features and the user's own history; collaborative uses the ratings of many users and no item features. Content handles new items and explains itself; collaborative finds tastes features miss but needs history.
  2. User-based vs item-based CF? User-based finds similar users; item-based finds similar items. Item-based is often preferred when there are far more users than items, because item similarities are fewer and more stable, so they can be precomputed (Amazon, 2003).
  3. What is the cold-start problem and how do you handle it? New users or items have no interactions. Use popularity, onboarding questions, content features, or a hybrid, and switch to CF as data arrives.
  4. Why is the user-item matrix sparse, and why does it matter? Each user interacts with a tiny fraction of items; similarities rest on few overlapping ratings and dense storage is wasteful.
  5. Explicit vs implicit feedback: which is more common? Implicit: every visit produces clicks and views, few users rate. It is noisy and has no negatives.
  6. Which similarity measures are used? Cosine, cosine on centred ratings (adjusted cosine), Pearson correlation, Jaccard for sets of interactions.

Glossary

Term Meaning (plain words + example)
Recommender system Software that picks, for each person, the items they're most likely to want: "you may also like"
Item Anything recommended: a dish, a movie, a book
Explicit feedback The user tells you their opinion: Asha gives Masala Dosa 5 stars
Implicit feedback You infer it from behaviour: Asha ordered Masala Dosa three times
User-item matrix Table with a row per user, a column per item, ratings in the cells
Sparsity Share of empty cells; the Netflix Prize data was about 99% empty
Sparse matrix A matrix where most cells are empty
Popularity baseline Recommend the most-rated (or best-rated with enough ratings) items to everyone
Content-based filtering Recommend items whose features match the user's profile
Item features Descriptions of an item: tags like spicy, veg, sweet, south
User profile Average features of the items the user liked: veg 1.0, south 0.67
Collaborative filtering (CF) Recommend from other people's ratings: "people like you liked this"
User-based CF Find the most similar users (neighbours), predict from their ratings
Item-based CF Find items rated alike by the same people, predict from the user's ratings of them
Neighbour One of the k most similar users (or items)
Similarity (for recommendations) A number saying how alike two users or items are; cosine here
Cosine similarity a · b / (|a| × |b|): 1 same direction, 0 unrelated, −1 opposite
Centring Subtracting each user's average rating, so +1 means "more than I usually like"
Cold-start problem A new user or item has no history, so CF can't use it
Onboarding Asking a new user a few questions at sign-up ("pick 3 dishes")
Hybrid recommender Combines content-based and collaborative methods
MAE Mean absolute error: average size of the miss, here in stars
NaN "Not a number": a missing value, or the result of 0 / 0

Summary

  • A recommender system fills the empty cells of a sparse user-item matrix; feedback is explicit (stars) or implicit (orders, clicks).
  • The popularity baseline isn't personal; averages of tiny counts are luck.
  • Content-based matches item features to a user profile with cosine similarity: works for new items, plays it safe, needs good tags (MAE 0.839).
  • Collaborative filtering learns from similar users (0.662) or similar items (0.642), with ratings centred per user.
  • The cold-start problem: no history, no CF (nan). Use popularity, onboarding and content features, then a hybrid.

Next episode: Recommender systems 2: matrix factorisation, latent factors, user and item embeddings, ALS and SGD, hybrid recommenders, precision@k, recall@k, MAP, nDCG, and offline vs online evaluation.