SyntaxPilot
Courses/Machine Learning/Lesson 4

Machine Learning · Lesson 4 of 90 · 15 min

Maths 2: matrices

matrixshape (rows by columns)transposematrix-vector productmatrix multiplicationidentity matrixinverse matrixdeterminant (intuition)rank (intuition)systems of linear equationslinear transformationdata matrix X

Machine Learning Full Course · Episode 4. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output. You only need episode 3 (vectors, the dot product) and basic arithmetic.

The problem: bill 200 customers at once

From episode 3, one chai-stall order is a vector: Asha's order of 1 chai, 1 samosa, 0 cake is [1, 1, 0]. Tonight Raju has 200 customers and wants every bill before closing. Doing 200 separate dot products by hand is slow and error-prone. We need a way to store all the orders together and multiply them in one go.

Matrix, shape and entries

Write the order vectors one under another, like rows in a notebook:

          chai samosa cake
Asha   [   1     1     0  ]
Ravi   [   6     6     0  ]
Meena  [   0     1     2  ]
Term Meaning Example
Matrix (plural matrices) a rectangular table of numbers the orders table above
Row one horizontal line; here one customer Ravi's [6, 6, 0]
Column one vertical line; here one item for all customers the chai column [1, 6, 0]
Shape (number of rows, number of columns), rows always first (3, 3); 200 customers × 3 items → (200, 3)
Entry one number, found by row and column (counting from 0) orders[2, 2] = Meena's 2 cakes

A vector is one list of numbers; a matrix is a stack of vectors. In NumPy a matrix is a 2-D array: a list of rows.

The obvious way fails: orders * prices

import numpy as np

# rows = customers, columns = chai, samosa, cake
orders = np.array([[1, 1, 0],    # Asha
                   [6, 6, 0],    # Ravi
                   [0, 1, 2]])   # Meena
prices = np.array([15, 20, 60])

print(orders.shape)
print(orders * prices)

Output:

(3, 3)
[[ 15  20   0]
 [ 90 120   0]
 [  0  20 120]]

Nine numbers, not three bills. Why: * multiplies matching items one by one (broadcasting stretched the price vector across every row, episode 2) but it never adds. A bill needs multiply, then add, which is the dot product from episode 3.

The matrix-vector product

Dot each row with the price vector:

  • Asha: 1×15 + 1×20 + 0×60 = 35
  • Ravi: 6×15 + 6×20 + 0×60 = 210
  • Meena: 0×15 + 1×20 + 2×60 = 140

One number per row. Multiplying a matrix by a vector this way, every row dotted with the vector, is the matrix-vector product. In NumPy it is the @ operator (or np.matmul):

import numpy as np

orders = np.array([[1, 1, 0],
                   [6, 6, 0],
                   [0, 1, 2]])
prices = np.array([15, 20, 60])

for row in orders:
    print(row @ prices)
print(orders @ prices)       # matrix-vector product

Output:

35
210
140
[ 35 210 140]

Shape rule: (3 × 3) @ (3,) → (3,). The matrix's number of columns must equal the vector's length (3 items, 3 prices). With 200 customers it is still one line, (200 × 3) @ (3,) → (200,), and NumPy runs it in fast compiled code.

Matrix multiplication

Diwali is coming and Raju has a second price list (festival: Rs 20, 25, 80). Each customer wants both bills. Write each price list as a column and stack the columns side by side: a 3 × 2 matrix. Every customer row dotted with every price column gives one bill. Doing every row with every column is matrix multiplication: many matrix-vector products at once.

First, the natural mistake: writing each price list as a row.

import numpy as np

orders = np.array([[1, 1, 0],
                   [6, 6, 0],
                   [0, 1, 2]])
# one row per price list: normal day, festival day
lists = np.array([[15, 20, 60],
                  [20, 25, 80]])

print(orders @ lists)

Output (the real error):

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)

The shape rule: (n × k) @ (k × m) = (n × m). The two inner numbers must match and disappear; the outer numbers give the answer's shape. Here (3 × 3) @ (2 × 3): inner 3 vs 2, so it fails. The signature in the message, (n?,k),(k,m?)->(n?,m?), is exactly this rule.

Transpose

Turn the table on its side so rows become columns, like a teacher's register turned sideways (students along the top instead of down the side). That's the transpose, A.T in NumPy, Aᵀ in maths. A (2 × 3) matrix becomes (3 × 2), and entry [i, j] moves to [j, i].

import numpy as np

orders = np.array([[1, 1, 0],
                   [6, 6, 0],
                   [0, 1, 2]])
lists = np.array([[15, 20, 60],
                  [20, 25, 80]])

print(lists.shape, lists.T.shape)
print(orders @ lists.T)      # (3x3) @ (3x2) -> (3x2)

Output:

(2, 3) (3, 2)
[[ 35  45]
 [210 270]
 [140 185]]

Row = customer, column = price list: Asha pays 35 normally and 45 at the festival; Meena 140 and 185 (0×20 + 1×25 + 2×80 = 185).

Order matters, and the identity matrix

import numpy as np

A = np.array([[1, 2],
              [3, 4]])
B = np.array([[0, 1],
              [1, 0]])
I = np.eye(2, dtype=int)     # identity matrix

print(A @ B)
print(B @ A)
print(A @ I)

Output:

[[2 1]
 [4 3]]
[[3 4]
 [1 2]]
[[1 2]
 [3 4]]
  • A @ B ≠ B @ A. With ordinary numbers 2 × 3 = 3 × 2, but matrix multiplication is not commutative: keep the order you mean. (Here B swaps columns when it's on the right and swaps rows when it's on the left.)
  • The identity matrix I has 1s on the main diagonal (top-left to bottom-right) and 0s elsewhere. A @ I = I @ A = A. It's a photocopier at 100%: it changes nothing, the matrix version of the number 1. np.eye(n) makes an n × n identity.

Systems of linear equations

Raju's price list is ruined by rain. Yesterday's bills: 2 chai + 1 samosa = Rs 50; 1 chai + 3 samosa = Rs 75. With unknown prices c and s:

2c + 1s = 50
1c + 3s = 75

[2 1] [c]   [50]
[1 3] [s] = [75]        A x = b
  • A linear equation has unknowns only multiplied by numbers and added (no squares, roots, or unknowns multiplied together).
  • A system of linear equations is several of them that must all be true at once. Its left side is a matrix-vector product, so the whole system is A x = b.

Linear regression (episode 18) solves one big system like this.

The inverse matrix

For numbers, 5x = 10 is solved by multiplying by 1/5. Matrices can't be divided, but many square matrices have an inverse A⁻¹ that undoes A, like an undo button: A A⁻¹ = A⁻¹ A = I. Then x = A⁻¹ b.

import numpy as np

# 2 chai + 1 samosa = Rs 50 ; 1 chai + 3 samosa = Rs 75
A = np.array([[2, 1],
              [1, 3]])
b = np.array([50, 75])

A_inv = np.linalg.inv(A)
print(A_inv.round(2))
print((A @ A_inv).round(2))
print(A_inv @ b)
print(np.linalg.solve(A, b))

Output:

[[ 0.6 -0.2]
 [-0.2  0.4]]
[[ 1.  0.]
 [-0.  1.]]
[15. 20.]
[15. 20.]

Chai Rs 15, samosa Rs 20 (check: 2×15 + 20 = 50 ✓, 15 + 3×20 = 75 ✓). The -0. is a tiny negative rounding leftover (about −5.6 × 10⁻¹⁷) from floating-point arithmetic. Prefer np.linalg.solve(A, b) over inv(A) @ b: it gives the same answer, is faster, and has smaller rounding errors.

Singular matrices, determinant and rank

What if the second clue is useless? The second customer ordered exactly double: 4 chai + 2 samosa = Rs 100.

import numpy as np

# 2 chai + 1 samosa = Rs 50 ; 4 chai + 2 samosa = Rs 100
A = np.array([[2, 1],
              [4, 2]])
b = np.array([50, 100])

print(np.linalg.det(A), np.linalg.matrix_rank(A))
print(np.linalg.solve(A, b))

Output:

0.0 1
Traceback (most recent call last):
  ...
numpy.linalg.LinAlgError: Singular matrix

Double the order, double the bill: the second equation tells us nothing new. One clue, two unknowns, so infinitely many answers (chai 15 & samosa 20 works, so does chai 20 & samosa 10). There's no unique answer and no inverse. A matrix with no inverse is singular.

Idea Plain meaning Our matrices
Rank number of genuinely different rows (rows not made by scaling and adding others): the number of real clues [[2,1],[4,2]] → 1; [[2,1],[1,3]] → 2
Full rank rank = number of rows (for a square matrix) [[2,1],[1,3]]
Determinant one number from a square matrix; for 2 × 2, a·d − b·c 2·2 − 1·4 = 0; 2·3 − 1·1 = 5
Singular det = 0 ⇔ not full rank ⇔ no inverse [[2,1],[4,2]]

In ML, two features that copy each other (rent in rupees and rent in thousands) make the data matrix lose rank: this is called multicollinearity and comes back in episode 19.

Linear transformations: a matrix moves points

A matrix times a point gives a new point, so a matrix is a machine that moves points. Every matrix is a linear transformation: it keeps straight lines straight and keeps the origin (0, 0) fixed. The determinant has a picture: how much the matrix scales area (and a negative determinant also flips orientation, like a mirror).

import numpy as np

stretch = np.array([[2, 0],      # x doubles
                    [0, 1]])     # y stays the same
squash = np.array([[1, 2],
                   [2, 4]])

for p in [[1, 0], [0, 1], [1, 1]]:
    print(p, "->", stretch @ p, squash @ p)

print(np.linalg.det(stretch), np.linalg.det(squash))

Output:

[1, 0] -> [2 0] [1 2]
[0, 1] -> [0 1] [2 4]
[1, 1] -> [2 1] [3 6]
2.0 0.0
  • Stretch (like stretching a photo to twice its width): x doubles, det = 2, area doubles; it can be undone.
  • Squash: every point lands on the line y = 2x. Two dimensions collapse into one, different points land on the same spot, and nothing can undo it: det = 0, no inverse.

Neural networks are chains of such matrix machines with a small non-linear "bend" between them (later in the course).

The data matrix X

A dataset stored with one row per example and one column per feature is the data matrix, by convention a capital X (shape n_samples × n_features). Neha's flats from episode 3:

import numpy as np

# data matrix X: one row per flat (example),
# columns = size, distance, rent (features)
X = np.array([[12, 1, 25],
              [11, 6, 21],
              [ 6, 2, 18],
              [ 9, 5, 24]])
w = np.array([2, -1, -0.5])   # Neha's weights: big, close, cheap

print(X.shape, w.shape)
print(X @ w)                  # one score per flat
print(X[:, 2], X.T.shape)     # column 2 = every rent

Output:

(4, 3) (3,)
[10.5  5.5  1.   1. ]
[25 21 18 24] (3, 4)
  • X @ w scores every row at once: (4 × 3) @ (3,) → (4,). Flat 0: 2×12 − 1 − 0.5×25 = 10.5.
  • X[:, 2] means "all rows, column 2": every rent.
  • A linear model is just X @ w (plus a constant). Learning means finding the best w automatically instead of guessing it: that's linear regression (episode 18).

Common mistakes (and the real error messages)

Mistake Error / result Why Fix
orders * prices for bills a 3 × 3 grid, not 3 bills * is item by item, no adding orders @ prices
(3×3) @ (2×3) ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3) inner sizes differ check shapes; transpose (lists.T)
np.linalg.solve([[2, 1], [4, 2]], b) numpy.linalg.LinAlgError: Singular matrix row 2 = 2 × row 1: det 0, rank 1 add a genuinely new equation; drop duplicate features
assuming A @ B == B @ A different matrices matrix multiplication isn't commutative keep the order you mean
inv(A) @ b everywhere works, but slower and less accurate builds the inverse needlessly np.linalg.solve(A, b)
comparing A @ A_inv == I exactly False because of -5.55e-17 floating-point rounding np.allclose(A @ A_inv, I)

Mini project: recover a lost price list, then bill new customers

Three items, three bills, three unknown prices:

import numpy as np

# 3 customers' orders (chai, samosa, cake) and their bills
orders = np.array([[2, 1, 0],
                   [1, 3, 1],
                   [0, 2, 2]])
bills = np.array([50, 135, 160])

print("rank:", np.linalg.matrix_rank(orders), "det:", round(np.linalg.det(orders), 2))
prices = np.linalg.solve(orders, bills)
print("prices:", prices)
print("check:", orders @ prices)

# new customers, normal and festival lists
new = np.array([[3, 0, 1], [0, 4, 0]])
lists = np.column_stack([prices, [20, 25, 80]])
print(lists.shape)
print(new @ lists)

Output:

rank: 3 det: 6.0
prices: [15. 20. 60.]
check: [ 50. 135. 160.]
(3, 2)
[[105. 140.]
 [ 80. 100.]]

Full rank (3) and det ≠ 0, so there's exactly one answer. np.column_stack puts vectors side by side as columns. Try changing the third order to [4, 2, 0] (twice the first) and watch the rank drop to 2 and solve raise Singular matrix.

Practice

  1. By hand: the determinant of [[3, 1], [2, 4]].
Solution

3·4 − 1·2 = 10. NumPy: round(np.linalg.det(np.array([[3, 1], [2, 4]])), 2) → 10.0. Not zero, so it has an inverse.

  1. Make a 2 × 2 matrix of rank 1 and check it.
Solution

Make row 2 a multiple of row 1, e.g. R = np.array([[1, 3], [2, 6]]). np.linalg.matrix_rank(R) → 1, np.linalg.det(R) → 0.0.

  1. What is the shape of (5 × 2) @ (2 × 7)? And (2 × 7) @ (5 × 2)?
Solution

(5, 7): the inner 2s match. The reverse has inner sizes 7 and 5, so NumPy raises ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 5 is different from 7).

  1. Your own lost price list: invent two orders of chai and cake with bills, and solve for both prices.
Solution

E.g. 1 chai + 1 cake = Rs 75 and 2 chai + 3 cake = Rs 210: np.linalg.solve(np.array([[1, 1], [2, 3]]), np.array([75, 210])) → [15. 60.].

Quick quiz

import numpy as np

A = np.ones((4, 3))
B = np.ones((3, 5))
print((A @ B).shape, (A @ B)[0, 0])

A) (4, 5) 3.0 B) (3, 3) 5.0 C) (4, 5) 1.0 D) Error: shapes don't match

Answer

A) (4, 5) 3.0. (4 × 3) @ (3 × 5): the inner 3s match, giving (4, 5). Each entry is a row of three 1s dotted with a column of three 1s = 3.0. (np.ones makes an array full of 1.0.)

Interview questions

  1. What is the shape rule for matrix multiplication? (n × k) @ (k × m) = (n × m); the inner dimensions must match.
  2. Element-wise * vs @? * multiplies matching entries (with broadcasting); @ is the matrix product (dot products of rows with columns).
  3. Is matrix multiplication commutative? No, generally AB ≠ BA. It is associative: (AB)C = A(BC).
  4. When does a matrix have an inverse? When it is square and full rank, equivalently det ≠ 0.
  5. Why use solve instead of inv? It solves A x = b directly (by factorisation), which is faster and numerically more stable than forming A⁻¹.
  6. What does the determinant mean geometrically? The factor by which the transformation scales area (volume in 3-D); 0 means space is squashed into a lower dimension.
  7. What does rank tell you about a data matrix? How many independent columns (features) it really has; rank lower than the number of features means some features are combinations of others (multicollinearity).
  8. Write a linear model in matrix form. ŷ = X w + b: one prediction per row of X.

Glossary

Term Meaning Example
Matrix A rectangular table of numbers the orders table
Row / column A horizontal / vertical line of a matrix Ravi's row; the chai column
Shape (rows, columns), rows first (3, 3), (200, 3)
Entry One number, by row and column orders[2, 2] = 2
2-D array NumPy's matrix: a list of rows np.array([[1, 1, 0], ...])
Element-wise product (*) Multiply matching entries, no adding orders * prices → 3 × 3 grid
Matrix-vector product Each row dotted with the vector orders @ prices → [35 210 140]
Matrix multiplication Every row of A dotted with every column of B orders @ lists.T → 3 × 2 bills
@ / np.matmul Python/NumPy matrix product operator A @ B
Inner dimensions The two middle numbers in (n × k) @ (k × m); must match k
Transpose (Aᵀ, A.T) Swap rows and columns (2, 3) → (3, 2)
Commutative Order doesn't matter (true for numbers, not matrices) 2×3 = 3×2, but AB ≠ BA
Main diagonal Entries from top-left to bottom-right the 1s of I
Identity matrix (I) 1s on the diagonal, 0s elsewhere; changes nothing np.eye(2)
Linear equation Unknowns only multiplied by numbers and added 2c + s = 50
System of linear equations Several linear equations true at once; A x = b the lost price list
Inverse (A⁻¹) The matrix that undoes A; A A⁻¹ = I np.linalg.inv(A)
np.linalg.solve Solves A x = b directly solve(A, b) → [15. 20.]
Singular matrix A matrix with no inverse [[2, 1], [4, 2]]
Determinant One number from a square matrix; area-scaling factor; 0 ⇔ singular a·d − b·c
Rank Number of genuinely different rows/columns (real clues) rank 1 for [[2, 1], [4, 2]]
Full rank Rank equals the number of rows (square case) [[2, 1], [1, 3]]
Linear transformation A map that keeps lines straight and the origin fixed; what a matrix does stretch, squash
Data matrix X Dataset as a matrix: rows = examples, columns = features Neha's flats, (4, 3)
Weights (w) One number per feature saying how much it counts [2, -1, -0.5]
Multicollinearity Features that are (near) copies or combinations of each other rent in Rs and in thousands
Floating point Computers store decimals approximately -5.55e-17 instead of 0
np.allclose Compares arrays allowing tiny rounding differences np.allclose(A @ A_inv, I)

Summary

  • A matrix is a table of numbers with a shape (rows, columns); stacked data points form the data matrix X.
  • * multiplies item by item; @ does the matrix-vector product (each row · vector) and matrix multiplication (every row · every column).
  • Shape rule: (n × k) @ (k × m) = (n × m); use the transpose to fix orientation. Order matters: AB ≠ BA.
  • The identity changes nothing; the inverse undoes a matrix and solves A x = b (prefer np.linalg.solve).
  • det = 0 ⇔ singular ⇔ not full rank ⇔ no inverse; rank counts the real clues; the determinant scales area.
  • Every matrix is a linear transformation, and a linear model is just X @ w.

Next episode: Maths for ML 3, eigenvectors and SVD: projections, orthogonal directions, the directions a matrix only stretches, and compressing a table with the singular value decomposition.