Machine Learning Full Course · Episode 4. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output. You only need episode 3 (vectors, the dot product) and basic arithmetic.
The problem: bill 200 customers at once
From episode 3, one chai-stall order is a vector: Asha's order of 1 chai, 1 samosa, 0 cake is [1, 1, 0]. Tonight Raju has 200 customers and wants every bill before closing. Doing 200 separate dot products by hand is slow and error-prone. We need a way to store all the orders together and multiply them in one go.
Matrix, shape and entries
Write the order vectors one under another, like rows in a notebook:
chai samosa cake
Asha [ 1 1 0 ]
Ravi [ 6 6 0 ]
Meena [ 0 1 2 ]
| Term | Meaning | Example |
|---|---|---|
| Matrix (plural matrices) | a rectangular table of numbers | the orders table above |
| Row | one horizontal line; here one customer | Ravi's [6, 6, 0] |
| Column | one vertical line; here one item for all customers | the chai column [1, 6, 0] |
| Shape | (number of rows, number of columns), rows always first | (3, 3); 200 customers × 3 items → (200, 3) |
| Entry | one number, found by row and column (counting from 0) | orders[2, 2] = Meena's 2 cakes |
A vector is one list of numbers; a matrix is a stack of vectors. In NumPy a matrix is a 2-D array: a list of rows.
The obvious way fails: orders * prices
import numpy as np
# rows = customers, columns = chai, samosa, cake
orders = np.array([[1, 1, 0], # Asha
[6, 6, 0], # Ravi
[0, 1, 2]]) # Meena
prices = np.array([15, 20, 60])
print(orders.shape)
print(orders * prices)
Output:
(3, 3)
[[ 15 20 0]
[ 90 120 0]
[ 0 20 120]]
Nine numbers, not three bills. Why: * multiplies matching items one by one (broadcasting stretched the price vector across every row, episode 2) but it never adds. A bill needs multiply, then add, which is the dot product from episode 3.
The matrix-vector product
Dot each row with the price vector:
- Asha: 1×15 + 1×20 + 0×60 = 35
- Ravi: 6×15 + 6×20 + 0×60 = 210
- Meena: 0×15 + 1×20 + 2×60 = 140
One number per row. Multiplying a matrix by a vector this way, every row dotted with the vector, is the matrix-vector product. In NumPy it is the @ operator (or np.matmul):
import numpy as np
orders = np.array([[1, 1, 0],
[6, 6, 0],
[0, 1, 2]])
prices = np.array([15, 20, 60])
for row in orders:
print(row @ prices)
print(orders @ prices) # matrix-vector product
Output:
35
210
140
[ 35 210 140]
Shape rule: (3 × 3) @ (3,) → (3,). The matrix's number of columns must equal the vector's length (3 items, 3 prices). With 200 customers it is still one line, (200 × 3) @ (3,) → (200,), and NumPy runs it in fast compiled code.
Matrix multiplication
Diwali is coming and Raju has a second price list (festival: Rs 20, 25, 80). Each customer wants both bills. Write each price list as a column and stack the columns side by side: a 3 × 2 matrix. Every customer row dotted with every price column gives one bill. Doing every row with every column is matrix multiplication: many matrix-vector products at once.
First, the natural mistake: writing each price list as a row.
import numpy as np
orders = np.array([[1, 1, 0],
[6, 6, 0],
[0, 1, 2]])
# one row per price list: normal day, festival day
lists = np.array([[15, 20, 60],
[20, 25, 80]])
print(orders @ lists)
Output (the real error):
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)
The shape rule: (n × k) @ (k × m) = (n × m). The two inner numbers must match and disappear; the outer numbers give the answer's shape. Here (3 × 3) @ (2 × 3): inner 3 vs 2, so it fails. The signature in the message, (n?,k),(k,m?)->(n?,m?), is exactly this rule.
Transpose
Turn the table on its side so rows become columns, like a teacher's register turned sideways (students along the top instead of down the side). That's the transpose, A.T in NumPy, Aᵀ in maths. A (2 × 3) matrix becomes (3 × 2), and entry [i, j] moves to [j, i].
import numpy as np
orders = np.array([[1, 1, 0],
[6, 6, 0],
[0, 1, 2]])
lists = np.array([[15, 20, 60],
[20, 25, 80]])
print(lists.shape, lists.T.shape)
print(orders @ lists.T) # (3x3) @ (3x2) -> (3x2)
Output:
(2, 3) (3, 2)
[[ 35 45]
[210 270]
[140 185]]
Row = customer, column = price list: Asha pays 35 normally and 45 at the festival; Meena 140 and 185 (0×20 + 1×25 + 2×80 = 185).
Order matters, and the identity matrix
import numpy as np
A = np.array([[1, 2],
[3, 4]])
B = np.array([[0, 1],
[1, 0]])
I = np.eye(2, dtype=int) # identity matrix
print(A @ B)
print(B @ A)
print(A @ I)
Output:
[[2 1]
[4 3]]
[[3 4]
[1 2]]
[[1 2]
[3 4]]
A @ B≠B @ A. With ordinary numbers 2 × 3 = 3 × 2, but matrix multiplication is not commutative: keep the order you mean. (HereBswaps columns when it's on the right and swaps rows when it's on the left.)- The identity matrix
Ihas 1s on the main diagonal (top-left to bottom-right) and 0s elsewhere.A @ I = I @ A = A. It's a photocopier at 100%: it changes nothing, the matrix version of the number 1.np.eye(n)makes an n × n identity.
Systems of linear equations
Raju's price list is ruined by rain. Yesterday's bills: 2 chai + 1 samosa = Rs 50; 1 chai + 3 samosa = Rs 75. With unknown prices c and s:
2c + 1s = 50
1c + 3s = 75
[2 1] [c] [50]
[1 3] [s] = [75] A x = b
- A linear equation has unknowns only multiplied by numbers and added (no squares, roots, or unknowns multiplied together).
- A system of linear equations is several of them that must all be true at once. Its left side is a matrix-vector product, so the whole system is A x = b.
Linear regression (episode 18) solves one big system like this.
The inverse matrix
For numbers, 5x = 10 is solved by multiplying by 1/5. Matrices can't be divided, but many square matrices have an inverse A⁻¹ that undoes A, like an undo button: A A⁻¹ = A⁻¹ A = I. Then x = A⁻¹ b.
import numpy as np
# 2 chai + 1 samosa = Rs 50 ; 1 chai + 3 samosa = Rs 75
A = np.array([[2, 1],
[1, 3]])
b = np.array([50, 75])
A_inv = np.linalg.inv(A)
print(A_inv.round(2))
print((A @ A_inv).round(2))
print(A_inv @ b)
print(np.linalg.solve(A, b))
Output:
[[ 0.6 -0.2]
[-0.2 0.4]]
[[ 1. 0.]
[-0. 1.]]
[15. 20.]
[15. 20.]
Chai Rs 15, samosa Rs 20 (check: 2×15 + 20 = 50 ✓, 15 + 3×20 = 75 ✓). The -0. is a tiny negative rounding leftover (about −5.6 × 10⁻¹⁷) from floating-point arithmetic. Prefer np.linalg.solve(A, b) over inv(A) @ b: it gives the same answer, is faster, and has smaller rounding errors.
Singular matrices, determinant and rank
What if the second clue is useless? The second customer ordered exactly double: 4 chai + 2 samosa = Rs 100.
import numpy as np
# 2 chai + 1 samosa = Rs 50 ; 4 chai + 2 samosa = Rs 100
A = np.array([[2, 1],
[4, 2]])
b = np.array([50, 100])
print(np.linalg.det(A), np.linalg.matrix_rank(A))
print(np.linalg.solve(A, b))
Output:
0.0 1
Traceback (most recent call last):
...
numpy.linalg.LinAlgError: Singular matrix
Double the order, double the bill: the second equation tells us nothing new. One clue, two unknowns, so infinitely many answers (chai 15 & samosa 20 works, so does chai 20 & samosa 10). There's no unique answer and no inverse. A matrix with no inverse is singular.
| Idea | Plain meaning | Our matrices |
|---|---|---|
| Rank | number of genuinely different rows (rows not made by scaling and adding others): the number of real clues | [[2,1],[4,2]] → 1; [[2,1],[1,3]] → 2 |
| Full rank | rank = number of rows (for a square matrix) | [[2,1],[1,3]] |
| Determinant | one number from a square matrix; for 2 × 2, a·d − b·c |
2·2 − 1·4 = 0; 2·3 − 1·1 = 5 |
| Singular | det = 0 ⇔ not full rank ⇔ no inverse | [[2,1],[4,2]] |
In ML, two features that copy each other (rent in rupees and rent in thousands) make the data matrix lose rank: this is called multicollinearity and comes back in episode 19.
Linear transformations: a matrix moves points
A matrix times a point gives a new point, so a matrix is a machine that moves points. Every matrix is a linear transformation: it keeps straight lines straight and keeps the origin (0, 0) fixed. The determinant has a picture: how much the matrix scales area (and a negative determinant also flips orientation, like a mirror).
import numpy as np
stretch = np.array([[2, 0], # x doubles
[0, 1]]) # y stays the same
squash = np.array([[1, 2],
[2, 4]])
for p in [[1, 0], [0, 1], [1, 1]]:
print(p, "->", stretch @ p, squash @ p)
print(np.linalg.det(stretch), np.linalg.det(squash))
Output:
[1, 0] -> [2 0] [1 2]
[0, 1] -> [0 1] [2 4]
[1, 1] -> [2 1] [3 6]
2.0 0.0
- Stretch (like stretching a photo to twice its width): x doubles, det = 2, area doubles; it can be undone.
- Squash: every point lands on the line y = 2x. Two dimensions collapse into one, different points land on the same spot, and nothing can undo it: det = 0, no inverse.
Neural networks are chains of such matrix machines with a small non-linear "bend" between them (later in the course).
The data matrix X
A dataset stored with one row per example and one column per feature is the data matrix, by convention a capital X (shape n_samples × n_features). Neha's flats from episode 3:
import numpy as np
# data matrix X: one row per flat (example),
# columns = size, distance, rent (features)
X = np.array([[12, 1, 25],
[11, 6, 21],
[ 6, 2, 18],
[ 9, 5, 24]])
w = np.array([2, -1, -0.5]) # Neha's weights: big, close, cheap
print(X.shape, w.shape)
print(X @ w) # one score per flat
print(X[:, 2], X.T.shape) # column 2 = every rent
Output:
(4, 3) (3,)
[10.5 5.5 1. 1. ]
[25 21 18 24] (3, 4)
X @ wscores every row at once:(4 × 3) @ (3,)→(4,). Flat 0: 2×12 − 1 − 0.5×25 = 10.5.X[:, 2]means "all rows, column 2": every rent.- A linear model is just
X @ w(plus a constant). Learning means finding the bestwautomatically instead of guessing it: that's linear regression (episode 18).
Common mistakes (and the real error messages)
| Mistake | Error / result | Why | Fix |
|---|---|---|---|
orders * prices for bills |
a 3 × 3 grid, not 3 bills | * is item by item, no adding |
orders @ prices |
(3×3) @ (2×3) |
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3) |
inner sizes differ | check shapes; transpose (lists.T) |
np.linalg.solve([[2, 1], [4, 2]], b) |
numpy.linalg.LinAlgError: Singular matrix |
row 2 = 2 × row 1: det 0, rank 1 | add a genuinely new equation; drop duplicate features |
assuming A @ B == B @ A |
different matrices | matrix multiplication isn't commutative | keep the order you mean |
inv(A) @ b everywhere |
works, but slower and less accurate | builds the inverse needlessly | np.linalg.solve(A, b) |
comparing A @ A_inv == I exactly |
False because of -5.55e-17 |
floating-point rounding | np.allclose(A @ A_inv, I) |
Mini project: recover a lost price list, then bill new customers
Three items, three bills, three unknown prices:
import numpy as np
# 3 customers' orders (chai, samosa, cake) and their bills
orders = np.array([[2, 1, 0],
[1, 3, 1],
[0, 2, 2]])
bills = np.array([50, 135, 160])
print("rank:", np.linalg.matrix_rank(orders), "det:", round(np.linalg.det(orders), 2))
prices = np.linalg.solve(orders, bills)
print("prices:", prices)
print("check:", orders @ prices)
# new customers, normal and festival lists
new = np.array([[3, 0, 1], [0, 4, 0]])
lists = np.column_stack([prices, [20, 25, 80]])
print(lists.shape)
print(new @ lists)
Output:
rank: 3 det: 6.0
prices: [15. 20. 60.]
check: [ 50. 135. 160.]
(3, 2)
[[105. 140.]
[ 80. 100.]]
Full rank (3) and det ≠ 0, so there's exactly one answer. np.column_stack puts vectors side by side as columns. Try changing the third order to [4, 2, 0] (twice the first) and watch the rank drop to 2 and solve raise Singular matrix.
Practice
- By hand: the determinant of
[[3, 1], [2, 4]].
Solution
3·4 − 1·2 = 10. NumPy: round(np.linalg.det(np.array([[3, 1], [2, 4]])), 2) → 10.0. Not zero, so it has an inverse.
- Make a 2 × 2 matrix of rank 1 and check it.
Solution
Make row 2 a multiple of row 1, e.g. R = np.array([[1, 3], [2, 6]]). np.linalg.matrix_rank(R) → 1, np.linalg.det(R) → 0.0.
- What is the shape of
(5 × 2) @ (2 × 7)? And(2 × 7) @ (5 × 2)?
Solution
(5, 7): the inner 2s match. The reverse has inner sizes 7 and 5, so NumPy raises ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 5 is different from 7).
- Your own lost price list: invent two orders of chai and cake with bills, and solve for both prices.
Solution
E.g. 1 chai + 1 cake = Rs 75 and 2 chai + 3 cake = Rs 210: np.linalg.solve(np.array([[1, 1], [2, 3]]), np.array([75, 210])) → [15. 60.].
Quick quiz
import numpy as np
A = np.ones((4, 3))
B = np.ones((3, 5))
print((A @ B).shape, (A @ B)[0, 0])
A) (4, 5) 3.0 B) (3, 3) 5.0 C) (4, 5) 1.0 D) Error: shapes don't match
Answer
A) (4, 5) 3.0. (4 × 3) @ (3 × 5): the inner 3s match, giving (4, 5). Each entry is a row of three 1s dotted with a column of three 1s = 3.0. (np.ones makes an array full of 1.0.)
Interview questions
- What is the shape rule for matrix multiplication?
(n × k) @ (k × m) = (n × m); the inner dimensions must match. - Element-wise
*vs@?*multiplies matching entries (with broadcasting);@is the matrix product (dot products of rows with columns). - Is matrix multiplication commutative? No, generally AB ≠ BA. It is associative: (AB)C = A(BC).
- When does a matrix have an inverse? When it is square and full rank, equivalently det ≠ 0.
- Why use
solveinstead ofinv? It solves A x = b directly (by factorisation), which is faster and numerically more stable than forming A⁻¹. - What does the determinant mean geometrically? The factor by which the transformation scales area (volume in 3-D); 0 means space is squashed into a lower dimension.
- What does rank tell you about a data matrix? How many independent columns (features) it really has; rank lower than the number of features means some features are combinations of others (multicollinearity).
- Write a linear model in matrix form. ŷ = X w + b: one prediction per row of X.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Matrix | A rectangular table of numbers | the orders table |
| Row / column | A horizontal / vertical line of a matrix | Ravi's row; the chai column |
| Shape | (rows, columns), rows first | (3, 3), (200, 3) |
| Entry | One number, by row and column | orders[2, 2] = 2 |
| 2-D array | NumPy's matrix: a list of rows | np.array([[1, 1, 0], ...]) |
Element-wise product (*) |
Multiply matching entries, no adding | orders * prices → 3 × 3 grid |
| Matrix-vector product | Each row dotted with the vector | orders @ prices → [35 210 140] |
| Matrix multiplication | Every row of A dotted with every column of B | orders @ lists.T → 3 × 2 bills |
@ / np.matmul |
Python/NumPy matrix product operator | A @ B |
| Inner dimensions | The two middle numbers in (n × k) @ (k × m); must match |
k |
Transpose (Aᵀ, A.T) |
Swap rows and columns | (2, 3) → (3, 2) |
| Commutative | Order doesn't matter (true for numbers, not matrices) | 2×3 = 3×2, but AB ≠ BA |
| Main diagonal | Entries from top-left to bottom-right | the 1s of I |
| Identity matrix (I) | 1s on the diagonal, 0s elsewhere; changes nothing | np.eye(2) |
| Linear equation | Unknowns only multiplied by numbers and added | 2c + s = 50 |
| System of linear equations | Several linear equations true at once; A x = b | the lost price list |
| Inverse (A⁻¹) | The matrix that undoes A; A A⁻¹ = I | np.linalg.inv(A) |
np.linalg.solve |
Solves A x = b directly | solve(A, b) → [15. 20.] |
| Singular matrix | A matrix with no inverse | [[2, 1], [4, 2]] |
| Determinant | One number from a square matrix; area-scaling factor; 0 ⇔ singular | a·d − b·c |
| Rank | Number of genuinely different rows/columns (real clues) | rank 1 for [[2, 1], [4, 2]] |
| Full rank | Rank equals the number of rows (square case) | [[2, 1], [1, 3]] |
| Linear transformation | A map that keeps lines straight and the origin fixed; what a matrix does | stretch, squash |
| Data matrix X | Dataset as a matrix: rows = examples, columns = features | Neha's flats, (4, 3) |
| Weights (w) | One number per feature saying how much it counts | [2, -1, -0.5] |
| Multicollinearity | Features that are (near) copies or combinations of each other | rent in Rs and in thousands |
| Floating point | Computers store decimals approximately | -5.55e-17 instead of 0 |
np.allclose |
Compares arrays allowing tiny rounding differences | np.allclose(A @ A_inv, I) |
Summary
- A matrix is a table of numbers with a shape (rows, columns); stacked data points form the data matrix X.
*multiplies item by item;@does the matrix-vector product (each row · vector) and matrix multiplication (every row · every column).- Shape rule:
(n × k) @ (k × m) = (n × m); use the transpose to fix orientation. Order matters: AB ≠ BA. - The identity changes nothing; the inverse undoes a matrix and solves A x = b (prefer
np.linalg.solve). - det = 0 ⇔ singular ⇔ not full rank ⇔ no inverse; rank counts the real clues; the determinant scales area.
- Every matrix is a linear transformation, and a linear model is just
X @ w.
Next episode: Maths for ML 3, eigenvectors and SVD: projections, orthogonal directions, the directions a matrix only stretches, and compressing a table with the singular value decomposition.