Machine Learning Full Course · Episode 9. Every code sample below was run with Python 3.13 using only the standard library (fractions, random, collections); the output shown is the real output. The medical-test numbers are made up for teaching; no real test is described.
The puzzle: a positive result on a 99% test
A health camp screens everyone for a rare illness.
- 1 person in 1,000 has it.
- The test catches 99% of sick people.
- It wrongly flags 1% of healthy people.
Your result is positive. How likely is it that you are actually sick? Most people answer 99%. To check, we build probability from scratch, and the real answer turns out to be about 9%.
Experiment, outcome, sample space, event
| Term | Meaning | Ludo die example |
|---|---|---|
| Experiment | an action whose result can't be known in advance | rolling a die |
| Outcome | one possible result | a 4 |
| Sample space | the set of all possible outcomes | {1, 2, 3, 4, 5, 6} |
| Event | a group of outcomes we care about | "even" = {2, 4, 6}; "big" (> 3) = {4, 5, 6} |
When every outcome is equally likely, the probability of an event A is
P(A) = (number of favourable outcomes) ÷ (total number of outcomes)
from fractions import Fraction
space = [1, 2, 3, 4, 5, 6] # sample space of one die roll
even = [x for x in space if x % 2 == 0] # an event
big = [x for x in space if x > 3] # another event
def P(event): # equally likely outcomes
return Fraction(len(event), len(space))
print("P(even) =", P(even), " P(big) =", P(big))
print("P(7) =", P([x for x in space if x == 7]), " P(any) =", P(space))
print("P(not 6) =", 1 - P([6]))
Output:
P(even) = 1/2 P(big) = 1/2
P(7) = 0 P(any) = 1
P(not 6) = 5/6
Fraction(3, 6)keeps exact fractions (shown as1/2) instead of decimals.[x for x in space if x % 2 == 0]is a list comprehension: keep the x's that pass the test (%is the remainder).
The probability rules
- Range: 0 ≤ P(A) ≤ 1. An impossible event (a 7) has probability 0; a certain event (the whole sample space) has probability 1.
- Complement: P(not A) = 1 − P(A). "Not A" (the complement) is every outcome outside A. P(not 6) = 5/6.
- Addition rule: P(A or B) = P(A) + P(B) − P(A and B).
Why the addition rule subtracts (the obvious way fails)
from fractions import Fraction
space = [1, 2, 3, 4, 5, 6]
even = {x for x in space if x % 2 == 0}
big = {x for x in space if x > 3}
def P(event):
return Fraction(len(event), len(space))
print("just add: ", P(even) + P(big))
print("count it: ", P(even | big), sorted(even | big))
print("subtract both:", P(even) + P(big) - P(even & big))
Output:
just add: 1
count it: 2/3 [2, 4, 5, 6]
subtract both: 2/3
Adding gives 1 ("certain"), but a 1 or a 3 is neither even nor big. 4 and 6 are in both events, so adding counted them twice. Subtract the overlap once. If two events can't both happen (mutually exclusive, e.g. "roll a 1" and "roll a 2"), the overlap is 0 and you may simply add.
{...}makes a set;|is set union (OR),&is intersection (AND).
Random variables
A random variable is a rule that gives a number for each outcome. Roll two dice in Ludo: 36 equally likely pairs, but your token moves by the sum, X = 2 … 12. X is "random" because it depends on the roll. Each value has a probability: a 7 can come 6 ways (1+6, 2+5, 3+4, 4+3, 5+2, 6+1), so P(X = 7) = 6/36; a 2 only one way (1/36).
import random
from collections import Counter
random.seed(7) # same "random" numbers every run
N = 100_000
sums = Counter(random.randint(1, 6) + random.randint(1, 6) for _ in range(N))
def exact(t): # count the 36 equally likely pairs
return sum(1 for a in range(1, 7) for b in range(1, 7) if a + b == t) / 36
for t in [2, 7, 12]:
print(t, "simulated:", round(sums[t] / N, 4), " exact:", round(exact(t), 4))
Output:
2 simulated: 0.0277 exact: 0.0278
7 simulated: 0.1682 exact: 0.1667
12 simulated: 0.0274 exact: 0.0278
random.randint(1, 6)gives a whole number from 1 to 6 inclusive.random.seed(7)fixes the starting point of Python's pseudo-random numbers, so everyone gets the same output (reproducibility, episode 2).Countercounts how many times each sum appeared.- Simulation (the long-run frequency view of probability): the more rolls, the closer the frequency gets to the exact probability. When unsure of an answer, simulate it.
Joint and marginal probability
Raju's order book for one week, 250 orders:
| Tea | Coffee | Total | |
|---|---|---|---|
| Morning | 120 | 30 | 150 |
| Evening | 60 | 40 | 100 |
| Total | 180 | 70 | 250 |
- Joint probability, two things together: P(morning and coffee) = 30 / 250 = 0.12.
- Marginal probability, one thing whatever the other: P(coffee) = (30 + 40) / 250 = 0.28; P(morning) = 150 / 250 = 0.6. Found by adding the joint table over the variable you don't care about; the name comes from the totals written in the table's margin.
Conditional probability
Given that it is morning, what is the chance of coffee? Look only at the morning row: 30 / 150 = 0.2. A probability computed when something is already known is a conditional probability, written P(A | B), read "A given B":
P(A | B) = P(A and B) ÷ P(B)
orders = {("morning", "tea"): 120, ("morning", "coffee"): 30,
("evening", "tea"): 60, ("evening", "coffee"): 40}
total = sum(orders.values())
def P(time=None, drink=None): # probability from counts
n = sum(c for (t, d), c in orders.items()
if (time is None or t == time) and (drink is None or d == drink))
return n / total
print("joint P(morning and coffee):", P("morning", "coffee"))
print("marginal P(coffee):", P(drink="coffee"), " P(morning):", P("morning"))
print("conditional P(coffee | morning):", P("morning", "coffee") / P("morning"))
print("conditional P(morning | coffee):", round(P("morning", "coffee") / P(drink="coffee"), 4))
Output:
joint P(morning and coffee): 0.12
marginal P(coffee): 0.28 P(morning): 0.6
conditional P(coffee | morning): 0.2
conditional P(morning | coffee): 0.4286
Noneas a default means "don't filter on this".orders.items()gives each ((time, drink), count) pair.- P(A | B) ≠ P(B | A). Of the morning orders, 20% are coffee; of the coffee orders, 43% are in the morning. Different questions, different denominators. Swapping them is the most common probability mistake, and it is the trap in the health-camp puzzle.
Independence
Two events are independent when knowing one tells you nothing about the other (two separate coin tosses). Then, and only then:
P(A and B) = P(A) × P(B)
import random
random.seed(1)
N = 100_000
a = [random.random() < 0.5 for _ in range(N)] # coin 1 lands heads?
b = [random.random() < 0.5 for _ in range(N)] # coin 2 lands heads?
both = sum(x and y for x, y in zip(a, b)) / N
print("coins: P(A and B) =", round(both, 3), " P(A)P(B) =", round(sum(a) / N * sum(b) / N, 3))
p_coffee, p_morning, p_both = 0.28, 0.6, 0.12
print("chai: P(A and B) =", p_both, " P(A)P(B) =", round(p_coffee * p_morning, 3))
Output:
coins: P(A and B) = 0.251 P(A)P(B) = 0.25
chai: P(A and B) = 0.12 P(A)P(B) = 0.168
random.random()gives a decimal in [0, 1);< 0.5is True half the time.sumof True/False counts the Trues.- Coins: they match (independent). Chai: 0.12 ≠ 0.168, so time of day and drink are dependent (equivalently, P(coffee | morning) = 0.2 ≠ P(coffee) = 0.28).
Back to the puzzle: simulate a million people
import random
random.seed(42)
N = 1_000_000
sick_pos = healthy_pos = 0
for _ in range(N):
sick = random.random() < 0.001 # 1 in 1,000 people have it
if sick:
positive = random.random() < 0.99 # catches 99% of sick people
else:
positive = random.random() < 0.01 # wrongly flags 1% of healthy
if positive:
sick_pos += sick
healthy_pos += not sick
print("positives:", sick_pos + healthy_pos, "sick:", sick_pos, "healthy:", healthy_pos)
print("P(sick | positive) =", round(sick_pos / (sick_pos + healthy_pos), 3))
Output:
positives: 10969 sick: 935 healthy: 10034
P(sick | positive) = 0.085
Of 10,969 positives, only 935 are sick. Why the gut fails: healthy people outnumber sick people 999 to 1, so 1% of the huge healthy crowd (≈ 10,000 false alarms) swamps 99% of the tiny sick group (≈ 990 true alarms). The 99% was P(positive | sick); the question asked P(sick | positive).
Bayes' theorem
P(H | E) = P(E | H) × P(H) ÷ P(E)
| Piece | Meaning | Puzzle value |
|---|---|---|
| H | the hypothesis | "I am sick" |
| E | the evidence | "the test is positive" |
| Prior P(H) | belief before the evidence | 0.001 |
| Likelihood P(E | H) | how likely the evidence is if H is true | 0.99 |
| Evidence P(E) | how likely the evidence is overall | 0.99 × 0.001 + 0.01 × 0.999 |
| Posterior P(H | E) | belief after the evidence | ≈ 0.09 |
P(E) uses the law of total probability: a positive can come from a sick person or from a healthy person, two separate ways, so add them.
prior = 0.001 # P(sick) before the test
p_pos_sick = 0.99 # P(positive | sick)
p_pos_healthy = 0.01 # P(positive | healthy)
p_pos = p_pos_sick * prior + p_pos_healthy * (1 - prior) # all positives
posterior = p_pos_sick * prior / p_pos # Bayes' theorem
print("P(positive) =", round(p_pos, 5))
print("P(sick | positive) =", round(posterior, 4))
second = p_pos_sick * posterior / (p_pos_sick * posterior + p_pos_healthy * (1 - posterior))
print("after a 2nd positive test:", round(second, 4))
Output:
P(positive) = 0.01098
P(sick | positive) = 0.0902
after a 2nd positive test: 0.9075
- The exact posterior 0.0902 matches the simulation (0.085 is within sampling noise for ~1,000 sick people).
- Today's posterior is tomorrow's prior: a second independent positive test lifts the belief to 90.75%. This is why rare diagnoses are confirmed with a second test.
- Base-rate fallacy: ignoring how common something is to begin with (the base rate, here 1 in 1,000). A fallacy is a common thinking mistake.
- Where it shows up in ML: spam filters (episode 28, naive Bayes), fraud alerts and any detector of rare events: with a rare positive class, even an accurate model raises mostly false alarms (precision, episode 24).
Common mistakes
| Mistake | Result | Why | Fix |
|---|---|---|---|
| P(even or big) = 1/2 + 1/2 | 1 instead of 2/3 |
{4, 6} counted twice | subtract P(A and B) |
| P(coffee | morning) = P(morning | coffee) | 0.2 vs 0.4286 | different denominators | divide by the probability of what you KNOW |
| P(A and B) = P(A) × P(B) always | 0.168 instead of 0.12 | only true for independent events | check independence first |
| positive on a 99% test ⇒ 99% sick | really ≈ 0.09 | base-rate fallacy | Bayes: include the prior |
| a probability above 1 or below 0 | impossible answer | a counting or formula error | check 0 ≤ P ≤ 1 |
simulation without random.seed |
different numbers every run | no fixed starting point | set a seed for repeatable results |
random.randint(1, 6) thought to exclude 6 |
wrong sample space | randint includes both ends (unlike range) |
randint(a, b) → a … b |
Mini project: the three-box puzzle
A prize is hidden in one of three boxes. You pick one. The host, who knows where the prize is, opens one of the other boxes and shows it is empty, then offers you a switch. Stay or switch?
import random
random.seed(11)
N = 100_000
stay_wins = switch_wins = 0
for _ in range(N):
prize = random.randint(1, 3) # the box with the prize
pick = random.randint(1, 3) # your first choice
# the host opens an empty box that you didn't pick
opened = random.choice([b for b in (1, 2, 3) if b != pick and b != prize])
switched = next(b for b in (1, 2, 3) if b != pick and b != opened)
stay_wins += pick == prize
switch_wins += switched == prize
print("stay:", stay_wins / N, " switch:", switch_wins / N)
Output:
stay: 0.33531 switch: 0.66469
Switching wins 2/3 of the time. Your first pick is right with probability 1/3 (the prior), and the host's choice is not independent of where the prize is: he must avoid it. That information moves the remaining 2/3 onto the other closed box. random.choice(list) picks one item; next(...) takes the first item that matches.
Practice
- P(sum of two dice ≥ 10), exact and simulated.
Solution
Pairs: (4,6), (5,5), (6,4), (5,6), (6,5), (6,6) = 6 of 36 = 1/6.
import random
from fractions import Fraction
exact = Fraction(sum(1 for a in range(1, 7) for b in range(1, 7) if a + b >= 10), 36)
random.seed(5)
N = 100_000
hits = sum(random.randint(1, 6) + random.randint(1, 6) >= 10 for _ in range(N))
print("exact:", exact, "=", round(float(exact), 4), " simulated:", hits / N)
Output:
exact: 1/6 = 0.1667 simulated: 0.16637
- P(tea | evening) and P(evening | tea) from Raju's table.
Solution
P(tea | evening) = 60 / 100 = 0.6; P(evening | tea) = 60 / 180 = 0.3333. Different denominators, different answers.
- Redo the test puzzle with 1 in 100 people sick.
Solution
posterior = 0.99 × 0.01 ÷ (0.99 × 0.01 + 0.01 × 0.99) = 0.5. A 10× higher base rate turns 9% into 50%: the prior matters enormously.
def bayes(prior, p_e_h, p_e_not_h):
return p_e_h * prior / (p_e_h * prior + p_e_not_h * (1 - prior))
print(round(bayes(0.01, 0.99, 0.01), 4))
prints 0.5.
- On one die, are "even" and "big" independent?
Solution
P(even and big) = P({4, 6}) = 1/3, but P(even) × P(big) = 1/2 × 1/2 = 1/4. Not equal, so not independent: knowing the roll is big makes even more likely (2 of 3 big numbers are even). In Python, P(even & big), P(even) * P(big) prints 1/3 1/4.
Quick quiz
from fractions import Fraction
space = range(1, 7)
A = {x for x in space if x <= 2}
B = {x for x in space if x % 2 == 1}
P = lambda e: Fraction(len(e), 6)
print(P(A | B), P(A & B) / P(B))
A) 2/3 1/3 B) 5/6 1/3 C) 2/3 1/2 D) 1 1/3
Answer
A) 2/3 1/3. A = {1, 2}, B = {1, 3, 5}. A or B = {1, 2, 3, 5}: 4/6 = 2/3. A and B = {1}: 1/6, so P(A | B) = (1/6) ÷ (3/6) = 1/3.
Interview questions
- What is conditional probability? The probability of A when B is known to have happened: P(A | B) = P(A and B) / P(B).
- State Bayes' theorem and name its parts. P(H | E) = P(E | H)·P(H) / P(E): posterior = likelihood × prior / evidence.
- What does independence mean, and how do you test it? Knowing one event doesn't change the other's probability: P(A | B) = P(A), equivalently P(A and B) = P(A)·P(B).
- Independent vs mutually exclusive? Mutually exclusive events can't happen together (P(A and B) = 0); independent events don't influence each other. Two mutually exclusive events with non-zero probabilities are always dependent.
- What is the base-rate fallacy, and where does it hurt in ML? Ignoring the prior frequency. With rare positives (fraud, disease), even an accurate classifier's alerts are mostly false positives, so accuracy alone misleads.
- Joint vs marginal vs conditional? Joint: P(A and B). Marginal: P(A), summing the joint over the other variable. Conditional: P(A | B), the joint divided by the marginal of B.
- How would you check a probability answer quickly? Simulate it with a fixed seed and compare the frequency with the formula.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Probability | A number from 0 to 1 saying how likely something is | P(even) = 1/2 |
| Experiment | An action with an uncertain result | rolling a die |
| Outcome | One possible result | a 4 |
| Sample space | The set of all possible outcomes | {1, …, 6} |
| Event | A group of outcomes | even = {2, 4, 6} |
| Equally likely | Every outcome has the same chance | a fair die |
| Complement | Everything outside an event; P(not A) = 1 − P(A) | not 6: 5/6 |
| Addition rule | P(A or B) = P(A) + P(B) − P(A and B) | 2/3 |
| Mutually exclusive | Events that can't happen together | roll a 1 / roll a 2 |
| Random variable | A number assigned to each outcome | X = sum of two dice |
| Simulation | Estimating probabilities by repeating random trials | 100,000 rolls |
| Seed | The starting point of pseudo-random numbers; same seed, same numbers | random.seed(7) |
| Joint probability | P(A and B) | P(morning and coffee) = 0.12 |
| Marginal probability | P(A) whatever the other variable is | P(coffee) = 0.28 |
| Conditional probability | P(A | B) = P(A and B) / P(B) | P(coffee | morning) = 0.2 |
| Independent events | Knowing one doesn't change the other; P(A and B) = P(A)·P(B) | two coins |
| Dependent events | Knowing one changes the other | morning and coffee |
| Hypothesis | A claim we want the probability of | "I am sick" |
| Evidence | What we observed | a positive test |
| Prior | Belief before the evidence | 0.001 |
| Likelihood | P(evidence | hypothesis) | 0.99 |
| Posterior | Belief after the evidence | 0.0902 |
| Law of total probability | Add up every separate way an event can happen | P(positive) = 0.01098 |
| Bayes' theorem | P(H | E) = P(E | H)·P(H) / P(E) | 9% sick |
| Base rate | How common something is in the whole population | 1 in 1,000 |
| Base-rate fallacy | Ignoring the base rate when judging evidence | "99% sick" |
| False positive | A positive result for someone who doesn't have it | 10,034 healthy positives |
Fraction |
Python's exact fraction type | Fraction(1, 2) |
Set, \|, & |
Python set; union (or); intersection (and) | even \| big |
Summary
- An experiment has a sample space of outcomes; an event is a group of them; P = favourable ÷ total when outcomes are equally likely.
- Rules: 0 ≤ P ≤ 1, complement 1 − P(A), addition rule subtracts the overlap.
- A random variable puts a number on each outcome; simulation with a seed checks any answer.
- From a table: joint, marginal and conditional probability; P(A | B) ≠ P(B | A).
- Independent events multiply; dependent ones don't.
- Bayes' theorem turns a prior into a posterior; ignoring the base rate (the base-rate fallacy) turns 9% into a mistaken 99%.
Next episode: Maths for ML 8, probability distributions: PMF, PDF, CDF, Bernoulli, binomial, Poisson, uniform and normal, mean and variance, sampled with NumPy.