Machine Learning Full Course · Episode 56. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output.
The big idea: an artificial neuron multiplies its inputs by weights, adds a bias, and passes the sum through an activation function. One neuron draws one straight line, so the perceptron learns AND and OR (4/4) but never XOR (2/4 after 1,000 epochs). Put neurons between the input and the output, a hidden layer, and XOR is solved (0, 1, 1, 0): that's a multilayer perceptron (MLP). Without an activation, two layers collapse into one (same? True), so every neuron needs a non-linear bend: step, sigmoid, tanh, ReLU. The forward pass is a few matrix multiplications, and the universal approximation theorem says one hidden layer with enough neurons can copy any continuous function (a sine wave: worst error 0.9406 with 1 unit, 0.0001 with 100). But someone still has to find the weights: that's next episode.
Sources: the perceptron was developed by Frank Rosenblatt in 1957, and Minsky and Papert's 1969 book Perceptrons showed that a single perceptron cannot learn XOR because XOR is not linearly separable (University of Waterloo CS 486/686 lecture notes on the history of neural networks, PDF); G. Cybenko, "Approximation by Superpositions of a Sigmoidal Function", Mathematics of Control, Signals and Systems 2, 303-314 (1989) (record).
Words you need first
- Vector, dot product (Episode 3): w · x = w1·x1 + w2·x2 + ...: (3, −2) · (1, 1) = 1.
- Matrix multiplication and shapes (Episode 4): (4×2) @ (2×3) gives 4×3; the inner numbers must match.
- Sigmoid (Episode 22): 1 / (1 + e^(−z)), an S-curve from 0 to 1.
- Linear regression, least squares (Episodes 18-19): fitting weights that minimise squared error;
np.linalg.lstsqdoes it in one call. - Epoch: one pass over all training examples.
- Gradient descent (Episode 8): improving weights by following the slope of the loss.
The artificial neuron: should Asha go for chai?
| Asha's decision | Neuron part |
|---|---|
| Reasons: friends coming? raining? exam tomorrow? (1 or 0) | inputs x |
| How much each matters: friends +3, rain −2, exam −4 | weights w |
| She loves chai anyway: +1 | bias b |
| Total above 0 → go | activation function (here a step) |
z = w1·x1 + w2·x2 + w3·x3 + b
output = step(z) = 1 if z > 0 else 0
- Friends yes, rain yes, exam no: 3 − 2 − 0 + 1 = 2 > 0 → go. Add the exam: 3 − 2 − 4 + 1 = −2 → stay home.
- Artificial neuron: weighted sum of inputs plus a bias, then an activation function.
- Weight: how strongly one input pushes the decision (positive = towards yes, negative = towards no).
- Bias: a fixed push up or down before any input, which moves the threshold.
- Perceptron: a neuron with a step activation plus a rule that learns w and b from examples (Rosenblatt, 1957).
- Perceptron learning rule: for each example, error = target − output (+1, 0 or −1); w ← w + rate × error × input; b ← b + rate × error. Correct answers change nothing.
The XOR wall
The staircase light: one light, a switch at the top and one at the bottom. Flip either and the light changes; both up or both down is off. The light is on when exactly one switch is up: XOR (exclusive or). Compare AND (on only when both are on) and OR (on when at least one is on).
| x1 | x2 | AND | OR | XOR |
|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 0 | 1 | 1 |
| 1 | 0 | 0 | 1 | 1 |
| 1 | 1 | 1 | 1 | 0 |
X = [(0, 0), (0, 1), (1, 0), (1, 1)] # two switches
gates = {"AND": [0, 0, 0, 1], "OR": [0, 1, 1, 1], "XOR": [0, 1, 1, 0]}
def neuron(x, w, b): return 1 if w[0] * x[0] + w[1] * x[1] + b > 0 else 0 # sum, then a step
for name, y in gates.items():
w, b = [0.0, 0.0], 0.0
for epoch in range(1000): # perceptron learning rule
for x, t in zip(X, y):
err = t - neuron(x, w, b) # +1, 0 or -1
w = [w[0] + 0.1 * err * x[0], w[1] + 0.1 * err * x[1]]; b += 0.1 * err
correct = sum(neuron(x, w, b) == t for x, t in zip(X, y))
print(f"{name:3}: {correct}/4 correct after 1000 epochs w = {[round(v, 1) for v in w]} b = {round(b, 1)}")
Output:
AND: 4/4 correct after 1000 epochs w = [0.2, 0.1] b = -0.2
OR : 4/4 correct after 1000 epochs w = [0.1, 0.1] b = 0.0
XOR: 2/4 correct after 1000 epochs w = [-0.1, 0.0] b = 0.1
Why XOR fails: the points where w1·x1 + w2·x2 + b = 0 form a straight line; the neuron outputs 1 on one side and 0 on the other. For AND, (1,1) sits alone in a corner. For XOR, the 1s, (0,1) and (1,0), lie on one diagonal and the 0s on the other: no single straight line splits them.
- Linearly separable: two classes that one straight line (in more dimensions, one flat plane) can split. AND and OR are; XOR isn't.
- Minsky and Papert's book Perceptrons (1969) pointed this out, and interest in neural networks dropped for years.
Hidden layers: neurons feeding neurons
XOR = OR, but not AND. Feed both switches into an OR neuron and an AND neuron, then combine them with a third neuron: "OR on and AND off".
def step(z): return 1 if z > 0 else 0
def neuron(x, w, b): return step(sum(wi * xi for wi, xi in zip(w, x)) + b)
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
h1 = neuron(x, [1, 1], -0.5) # hidden neuron 1: OR (at least one on)
h2 = neuron(x, [1, 1], -1.5) # hidden neuron 2: AND (both on)
out = neuron([h1, h2], [1, -1], -0.5) # output: OR but not AND
print(x, "-> hidden", (h1, h2), "-> XOR =", out)
Output:
(0, 0) -> hidden (0, 0) -> XOR = 0
(0, 1) -> hidden (1, 0) -> XOR = 1
(1, 0) -> hidden (1, 0) -> XOR = 1
(1, 1) -> hidden (1, 1) -> XOR = 0
- Hidden layer: neurons between the inputs and the output; we never see their values directly. Here: h1 = OR, h2 = AND.
- Each hidden neuron draws one line; the output neuron combines the lines into a shape no single line could make.
Layers and the multilayer perceptron
- Layer: a group of neurons that all read the same inputs.
- Input layer: the features. Output layer: the neurons that give the prediction.
- Multilayer perceptron (MLP): input → one or more hidden layers → output.
- Fully connected (dense): every neuron reads every output of the layer before.
- Deep network: many hidden layers; that's what "deep" in deep learning means.
Why every neuron needs an activation function
Drop the step and stack two layers of plain weighted sums:
import numpy as np; rng = np.random.default_rng(56)
W1, b1 = rng.normal(size=(2, 3)), rng.normal(size=3) # layer 1: 2 inputs -> 3 neurons, NO activation
W2, b2 = rng.normal(size=(3, 1)), rng.normal(size=1) # layer 2: 3 -> 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
two_layers = (X @ W1 + b1) @ W2 + b2
W, b = W1 @ W2, b1 @ W2 + b2 # multiply the layers out
one_layer = X @ W + b
print("two layers:", two_layers.ravel().round(4))
print("one layer :", one_layer.ravel().round(4))
print("same?", np.allclose(two_layers, one_layer), "| still a straight line: XOR impossible")
Output:
two layers: [2.1447 1.4547 1.0079 0.3179]
one layer : [2.1447 1.4547 1.0079 0.3179]
same? True | still a straight line: XOR impossible
(X·W1 + b1)·W2 + b2 = X·(W1·W2) + (b1·W2 + b2): one weight matrix and one bias. A line of a line is still a line, so the extra layer adds nothing.
- Activation function: a non-linear function applied to each neuron's weighted sum z.
- Non-linear: its graph is not a straight line; it bends somewhere. The bend is what lets each layer build new shapes.
Four activation functions
| Name | Formula | Range | Notes |
|---|---|---|---|
| Step | 1 if z > 0 else 0 | 0 or 1 | a hard jump; slope 0 everywhere else |
| Sigmoid | 1 / (1 + e^(−z)) | 0 to 1 | smooth; output can be read as a probability |
| tanh (hyperbolic tangent) | (e^z − e^(−z)) / (e^z + e^(−z)) | −1 to 1 | S-shaped, centred at 0 |
| ReLU (rectified linear unit) | max(0, z) | 0 to ∞ | cheap; the usual choice for hidden layers |
import numpy as np
z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0]) # weighted sums coming into a neuron
print("z :", z)
print("step :", (z > 0).astype(float)) # 0 or 1, a hard jump
print("sigmoid:", (1 / (1 + np.exp(-z))).round(3)) # smooth, between 0 and 1
print("tanh :", np.tanh(z).round(3)) # smooth, between -1 and 1
print("ReLU :", np.maximum(0, z)) # 0 for negatives, z itself otherwise
Output:
z : [-3. -1. 0. 1. 3.]
step : [0. 0. 0. 1. 1.]
sigmoid: [0.047 0.269 0.5 0.731 0.953]
tanh : [-0.995 -0.762 0. 0.762 0.995]
ReLU : [0. 0. 0. 1. 3.]
Which one where? ReLU in hidden layers; sigmoid at the output for a yes/no probability; no activation (linear) at the output for a number such as a price. The step is flat almost everywhere, so its slope is 0 and gradient descent has nothing to follow: smooth activations make learning possible (Episode 57).
Forward propagation with matrices
Z1 = X @ W1 + b1 # (4×2) @ (2×2) → 4×2
H = relu(Z1) # hidden layer outputs
out = H @ W2 + b2 # (4×2) @ (2×1) → 4×1
- Forward propagation (forward pass): pushing inputs through the layers, one after another, to get the output.
- Each row of X is one example, so a whole batch goes through at once.
- W1 has shape (inputs × neurons): column j holds the weights of hidden neuron j.
import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) # 4 examples, 2 features: a batch
W1 = np.array([[1, 1], [1, 1]]); b1 = np.array([0, -1]) # hidden layer: 2 neurons
W2 = np.array([[1], [-2]]); b2 = np.array([0]) # output layer: 1 neuron
relu = lambda z: np.maximum(0, z)
Z1 = X @ W1 + b1; print("Z1 = X @ W1 + b1 :", Z1.tolist(), Z1.shape)
H = relu(Z1); print("H = relu(Z1) :", H.tolist())
out = H @ W2 + b2; print("out = H @ W2 + b2:", out.ravel().tolist(), out.shape)
Output:
Z1 = X @ W1 + b1 : [[0, -1], [1, 0], [1, 0], [2, 1]] (4, 2)
H = relu(Z1) : [[0, 0], [1, 0], [1, 0], [2, 1]]
out = H @ W2 + b2: [0, 1, 1, 0] (4, 1)
XOR with two ReLU neurons: h1 = relu(x1 + x2), h2 = relu(x1 + x2 − 1), out = h1 − 2·h2.
A wrong shape
import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) # 4 examples, 2 features
W1 = np.ones((3, 2)); b1 = np.zeros(3) # 3 hidden neurons... written as (neurons, inputs)
Z1 = X @ W1 + b1
print(Z1)
Output:
Traceback (most recent call last):
File "main.py", line 4, in <module>
Z1 = X @ W1 + b1
~~^~~~
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2)
W1 must be (inputs × neurons) = (2 × 3), not (3 × 2).
The universal approximation theorem
Each ReLU neuron adds one bend to the output's graph; with more bends, a chain of straight pieces can follow any curve more closely, like drawing a circle with many short lines.
Universal approximation theorem (Cybenko, 1989): a network with one hidden layer of sigmoid neurons can approximate any continuous function on a bounded region as closely as you like, given enough neurons. Later results extended this to other activations such as ReLU. It says nothing about how many neurons, or how to find the weights.
import numpy as np
x = np.linspace(0, 2 * np.pi, 200)[:, None]; y = np.sin(x).ravel() # the curve to copy: one sine wave
for units in [1, 3, 5, 20, 100]: # hidden ReLU neurons
W, b = np.ones((1, units)), -np.linspace(0, 2 * np.pi, units, endpoint=False) # each neuron bends at one spot
H = np.c_[np.maximum(0, x @ W + b), np.ones(len(x))] # forward pass to the hidden layer
w_out = np.linalg.lstsq(H, y, rcond=None)[0] # best output weights
print(f"{units:3} hidden ReLU units: worst error {np.abs(H @ w_out - y).max():.4f}")
Output:
1 hidden ReLU units: worst error 0.9406
3 hidden ReLU units: worst error 0.2894
5 hidden ReLU units: worst error 0.1143
20 hidden ReLU units: worst error 0.0071
100 hidden ReLU units: worst error 0.0001
Here we placed the bends evenly by hand and fitted only the output weights with least squares; in a real network every weight is learned.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| One neuron for XOR | XOR: 2/4 correct after 1000 epochs |
one straight line | add a hidden layer |
| Layers with no activation | same? True |
a line of a line is a line | ReLU (or another non-linearity) after each hidden layer |
| Step activation in a network you train | gradient 0 everywhere | flat function | ReLU, sigmoid, tanh |
| Weight matrix in the wrong shape | ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2) |
(4×2) @ (3×2) | W is (inputs × neurons) |
| Forgetting the bias | the line must pass through the origin | no shift possible | add b |
| Reading "universal" as "easy" | no recipe for the weights | the theorem only says weights exist | training (Episode 57) |
Mini project: let a perceptron learn Asha's rule
import itertools
X = list(itertools.product([0, 1], repeat=3)) # friends, rain, exam: all 8 days
Y = [1 if 3 * f - 2 * r - 4 * e + 1 > 0 else 0 for f, r, e in X] # Asha's real decisions
def neuron(x, w, b): return 1 if sum(wi * xi for wi, xi in zip(w, x)) + b > 0 else 0
w, b = [0.0, 0.0, 0.0], 0.0
for epoch in range(100): # perceptron learning rule
for x, t in zip(X, Y):
err = t - neuron(x, w, b)
w = [wi + 0.1 * err * xi for wi, xi in zip(w, x)]; b += 0.1 * err
print("learned w =", [round(v, 1) for v in w], " b =", round(b, 1))
print("correct:", sum(neuron(x, w, b) == t for x, t in zip(X, Y)), "of 8")
Output:
learned w = [0.1, -0.1, -0.3] b = 0.1
correct: 8 of 8
The learned weights differ from Asha's (3, −2, −4, +1) but make the same 8 decisions: many weight settings draw an equally good line.
Practice
1. Hand-build NAND and NOR neurons.
Solution
def neuron(x, w, b): return 1 if sum(wi * xi for wi, xi in zip(w, x)) + b > 0 else 0
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
print(x, "NAND =", neuron(x, [-1, -1], 1.5), " NOR =", neuron(x, [-1, -1], 0.5))
Output:
(0, 0) NAND = 1 NOR = 1
(0, 1) NAND = 1 NOR = 0
(1, 0) NAND = 1 NOR = 0
(1, 1) NAND = 0 NOR = 0
NAND = not AND, NOR = not OR: flip the signs of the weights and pick the threshold.
2. XOR with sigmoid instead of step.
Solution
Scale the step weights up so the sigmoid is nearly a step:
import math
sig = lambda z: 1 / (1 + math.exp(-z))
def neuron(x, w, b): return sig(sum(wi * xi for wi, xi in zip(w, x)) + b)
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
h1 = neuron(x, [10, 10], -5); h2 = neuron(x, [10, 10], -15) # the step weights, scaled up by 10
out = neuron([h1, h2], [10, -10], -5)
print(x, "->", round(out, 3))
Output:
(0, 0) -> 0.007
(0, 1) -> 0.992
(1, 0) -> 0.992
(1, 1) -> 0.007
3. Forward pass 2 → 4 → 1 with random weights.
Solution
import numpy as np; rng = np.random.default_rng(56)
X = rng.random((5, 2)) # 5 examples, 2 features
W1, b1 = rng.normal(size=(2, 4)), np.zeros(4) # 2 inputs -> 4 hidden neurons
W2, b2 = rng.normal(size=(4, 1)), np.zeros(1) # 4 -> 1 output
H = np.maximum(0, X @ W1 + b1); out = 1 / (1 + np.exp(-(H @ W2 + b2))) # ReLU hidden, sigmoid output
print("H:", H.shape, " out:", out.shape)
print(out.ravel().round(3))
Output:
H: (5, 4) out: (5, 1)
[0.461 0.444 0.5 0.493 0.5 ]
Outputs of exactly 0.5 mean all four ReLUs were 0 for that example, so the output sum is 0 and sigmoid(0) = 0.5.
4. Sine wave with 10 bends: worst error?
Solution
import numpy as np
x = np.linspace(0, 2 * np.pi, 200)[:, None]; y = np.sin(x).ravel()
units = 10
W, b = np.ones((1, units)), -np.linspace(0, 2 * np.pi, units, endpoint=False)
H = np.c_[np.maximum(0, x @ W + b), np.ones(len(x))]
w_out = np.linalg.lstsq(H, y, rcond=None)[0]
print(f"{units} hidden ReLU units: worst error {np.abs(H @ w_out - y).max():.4f}")
Output:
10 hidden ReLU units: worst error 0.0301
Between 5 units (0.1143) and 20 units (0.0071).
Quick quiz
Inputs x = [1, 1], weights w = [2, −1], bias b = −3, ReLU. Output?
Answer
0. z = 2 − 1 − 3 = −2; ReLU(−2) = max(0, −2) = 0.
More questions:
- Why can't one perceptron learn XOR?
Answer
It draws one straight line, and XOR's classes are not linearly separable. - What happens if an MLP has no activation functions?
Answer
The layers multiply out into a single linear layer. - A batch of 32 examples with 10 features goes into a layer of 64 neurons. Shape of W and of the output?
Answer
W: (10, 64); output: (32, 64). - Which activation would you use at the output to predict a house price?
Answer
None (linear), because the price can be any number. - What does the universal approximation theorem not tell you?
Answer
How many neurons are needed or how to find the weights.
Interview questions
- What does a single neuron compute? activation(w · x + b).
- Why do neural networks need non-linear activations? Without them, any stack of layers equals one linear layer.
- Sigmoid vs tanh vs ReLU? Sigmoid: 0 to 1, saturates; tanh: −1 to 1, zero-centred, saturates; ReLU: max(0, z), cheap, doesn't saturate for positive inputs, the default for hidden layers.
- Explain the XOR problem. XOR isn't linearly separable, so a single-layer perceptron fails; one hidden layer solves it.
- What is forward propagation? Computing each layer's outputs from the previous layer's: Z = XW + b, A = f(Z), until the output.
- State the universal approximation theorem. A feed-forward network with one hidden layer and enough units can approximate any continuous function on a compact set arbitrarily well.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Artificial neuron | Weighted sum of inputs plus bias, through an activation: Asha's chai decision |
| Input | A number fed into a neuron: "raining" = 1 |
| Weight | How strongly an input pushes the output: rain −2 |
| Bias | A fixed push before any input: +1 for loving chai |
| Perceptron | A step-activation neuron with a learning rule for w and b (Rosenblatt, 1957) |
| Perceptron learning rule | w ← w + rate × (target − output) × input |
| Activation function | Non-linear function applied to the weighted sum |
| Non-linear | Not a straight line; bends somewhere |
| Step function | 1 if z > 0 else 0 |
| Sigmoid | 1 / (1 + e^(−z)), 0 to 1: sigmoid(0) = 0.5 |
| tanh | Hyperbolic tangent, −1 to 1: tanh(1) = 0.762 |
| ReLU | Rectified linear unit, max(0, z): ReLU(−3) = 0, ReLU(3) = 3 |
| AND / OR / XOR | Logic rules for two switches; XOR = exactly one on (staircase light) |
| Linearly separable | Classes that one straight line (plane) can split |
| Layer | Neurons that all read the same inputs |
| Input layer | The features going into the network |
| Hidden layer | Neurons between input and output: the OR and AND neurons |
| Output layer | Neurons that give the prediction |
| Multilayer perceptron (MLP) | Input → hidden layer(s) → output |
| Fully connected (dense) layer | Every neuron reads every output of the previous layer |
| Deep network | A network with many hidden layers |
| Forward propagation | Computing the output layer by layer: Z1 = X @ W1 + b1, H = relu(Z1), ... |
| Batch | Several examples processed together, one per row of X |
| Universal approximation theorem | One hidden layer with enough neurons can approximate any continuous function (Cybenko, 1989) |
| Least squares | Choosing weights that minimise the sum of squared errors |
Summary
- A neuron = activation(w · x + b). One neuron = one straight line: the perceptron learns AND and OR but not XOR.
- A hidden layer fixes XOR: the multilayer perceptron.
- Without activation functions, layers collapse into one. Know step, sigmoid, tanh, ReLU; use ReLU in hidden layers.
- The forward pass is Z = XW + b, A = f(Z), layer by layer, for a whole batch at once.
- Universal approximation: enough hidden neurons can copy any smooth curve (sine: 0.9406 → 0.0001), but the theorem doesn't find the weights.
Next episode: Neural networks from scratch 2: backpropagation: local gradients, the chain rule through layers, gradient checks, and training XOR in plain Python.