SyntaxPilot
Courses/Machine Learning/Lesson 56
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 56 of 90

Neural networks from scratch 1: forward pass

perceptronartificial neuronweights and biasactivation functionstepsigmoidtanhReLUlayerhidden layermultilayer perceptron (MLP)forward propagationuniversal approximation theorem

Machine Learning Full Course · Episode 56. Every code sample below was run with Python 3.13 and NumPy 2.5.3; the output shown is the real output.

The big idea: an artificial neuron multiplies its inputs by weights, adds a bias, and passes the sum through an activation function. One neuron draws one straight line, so the perceptron learns AND and OR (4/4) but never XOR (2/4 after 1,000 epochs). Put neurons between the input and the output, a hidden layer, and XOR is solved (0, 1, 1, 0): that's a multilayer perceptron (MLP). Without an activation, two layers collapse into one (same? True), so every neuron needs a non-linear bend: step, sigmoid, tanh, ReLU. The forward pass is a few matrix multiplications, and the universal approximation theorem says one hidden layer with enough neurons can copy any continuous function (a sine wave: worst error 0.9406 with 1 unit, 0.0001 with 100). But someone still has to find the weights: that's next episode.

Sources: the perceptron was developed by Frank Rosenblatt in 1957, and Minsky and Papert's 1969 book Perceptrons showed that a single perceptron cannot learn XOR because XOR is not linearly separable (University of Waterloo CS 486/686 lecture notes on the history of neural networks, PDF); G. Cybenko, "Approximation by Superpositions of a Sigmoidal Function", Mathematics of Control, Signals and Systems 2, 303-314 (1989) (record).

Words you need first

  • Vector, dot product (Episode 3): w · x = w1·x1 + w2·x2 + ...: (3, −2) · (1, 1) = 1.
  • Matrix multiplication and shapes (Episode 4): (4×2) @ (2×3) gives 4×3; the inner numbers must match.
  • Sigmoid (Episode 22): 1 / (1 + e^(−z)), an S-curve from 0 to 1.
  • Linear regression, least squares (Episodes 18-19): fitting weights that minimise squared error; np.linalg.lstsq does it in one call.
  • Epoch: one pass over all training examples.
  • Gradient descent (Episode 8): improving weights by following the slope of the loss.

The artificial neuron: should Asha go for chai?

Asha's decision Neuron part
Reasons: friends coming? raining? exam tomorrow? (1 or 0) inputs x
How much each matters: friends +3, rain −2, exam −4 weights w
She loves chai anyway: +1 bias b
Total above 0 → go activation function (here a step)
z = w1·x1 + w2·x2 + w3·x3 + b
output = step(z) = 1 if z > 0 else 0
  • Friends yes, rain yes, exam no: 3 − 2 − 0 + 1 = 2 > 0 → go. Add the exam: 3 − 2 − 4 + 1 = −2 → stay home.
  • Artificial neuron: weighted sum of inputs plus a bias, then an activation function.
  • Weight: how strongly one input pushes the decision (positive = towards yes, negative = towards no).
  • Bias: a fixed push up or down before any input, which moves the threshold.
  • Perceptron: a neuron with a step activation plus a rule that learns w and b from examples (Rosenblatt, 1957).
  • Perceptron learning rule: for each example, error = target − output (+1, 0 or −1); w ← w + rate × error × input; b ← b + rate × error. Correct answers change nothing.

The XOR wall

The staircase light: one light, a switch at the top and one at the bottom. Flip either and the light changes; both up or both down is off. The light is on when exactly one switch is up: XOR (exclusive or). Compare AND (on only when both are on) and OR (on when at least one is on).

x1 x2 AND OR XOR
0 0 0 0 0
0 1 0 1 1
1 0 0 1 1
1 1 1 1 0
X = [(0, 0), (0, 1), (1, 0), (1, 1)]                       # two switches
gates = {"AND": [0, 0, 0, 1], "OR": [0, 1, 1, 1], "XOR": [0, 1, 1, 0]}
def neuron(x, w, b): return 1 if w[0] * x[0] + w[1] * x[1] + b > 0 else 0   # sum, then a step
for name, y in gates.items():
    w, b = [0.0, 0.0], 0.0
    for epoch in range(1000):                              # perceptron learning rule
        for x, t in zip(X, y):
            err = t - neuron(x, w, b)                      # +1, 0 or -1
            w = [w[0] + 0.1 * err * x[0], w[1] + 0.1 * err * x[1]]; b += 0.1 * err
    correct = sum(neuron(x, w, b) == t for x, t in zip(X, y))
    print(f"{name:3}: {correct}/4 correct after 1000 epochs  w = {[round(v, 1) for v in w]} b = {round(b, 1)}")

Output:

AND: 4/4 correct after 1000 epochs  w = [0.2, 0.1] b = -0.2
OR : 4/4 correct after 1000 epochs  w = [0.1, 0.1] b = 0.0
XOR: 2/4 correct after 1000 epochs  w = [-0.1, 0.0] b = 0.1

Why XOR fails: the points where w1·x1 + w2·x2 + b = 0 form a straight line; the neuron outputs 1 on one side and 0 on the other. For AND, (1,1) sits alone in a corner. For XOR, the 1s, (0,1) and (1,0), lie on one diagonal and the 0s on the other: no single straight line splits them.

  • Linearly separable: two classes that one straight line (in more dimensions, one flat plane) can split. AND and OR are; XOR isn't.
  • Minsky and Papert's book Perceptrons (1969) pointed this out, and interest in neural networks dropped for years.

Hidden layers: neurons feeding neurons

XOR = OR, but not AND. Feed both switches into an OR neuron and an AND neuron, then combine them with a third neuron: "OR on and AND off".

def step(z): return 1 if z > 0 else 0
def neuron(x, w, b): return step(sum(wi * xi for wi, xi in zip(w, x)) + b)
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h1 = neuron(x, [1, 1], -0.5)                           # hidden neuron 1: OR   (at least one on)
    h2 = neuron(x, [1, 1], -1.5)                           # hidden neuron 2: AND  (both on)
    out = neuron([h1, h2], [1, -1], -0.5)                  # output: OR but not AND
    print(x, "-> hidden", (h1, h2), "-> XOR =", out)

Output:

(0, 0) -> hidden (0, 0) -> XOR = 0
(0, 1) -> hidden (1, 0) -> XOR = 1
(1, 0) -> hidden (1, 0) -> XOR = 1
(1, 1) -> hidden (1, 1) -> XOR = 0
  • Hidden layer: neurons between the inputs and the output; we never see their values directly. Here: h1 = OR, h2 = AND.
  • Each hidden neuron draws one line; the output neuron combines the lines into a shape no single line could make.

Layers and the multilayer perceptron

  • Layer: a group of neurons that all read the same inputs.
  • Input layer: the features. Output layer: the neurons that give the prediction.
  • Multilayer perceptron (MLP): input → one or more hidden layers → output.
  • Fully connected (dense): every neuron reads every output of the layer before.
  • Deep network: many hidden layers; that's what "deep" in deep learning means.

Why every neuron needs an activation function

Drop the step and stack two layers of plain weighted sums:

import numpy as np; rng = np.random.default_rng(56)
W1, b1 = rng.normal(size=(2, 3)), rng.normal(size=3)       # layer 1: 2 inputs -> 3 neurons, NO activation
W2, b2 = rng.normal(size=(3, 1)), rng.normal(size=1)       # layer 2: 3 -> 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
two_layers = (X @ W1 + b1) @ W2 + b2
W, b = W1 @ W2, b1 @ W2 + b2                               # multiply the layers out
one_layer = X @ W + b
print("two layers:", two_layers.ravel().round(4))
print("one layer :", one_layer.ravel().round(4))
print("same?", np.allclose(two_layers, one_layer), "| still a straight line: XOR impossible")

Output:

two layers: [2.1447 1.4547 1.0079 0.3179]
one layer : [2.1447 1.4547 1.0079 0.3179]
same? True | still a straight line: XOR impossible

(X·W1 + b1)·W2 + b2 = X·(W1·W2) + (b1·W2 + b2): one weight matrix and one bias. A line of a line is still a line, so the extra layer adds nothing.

  • Activation function: a non-linear function applied to each neuron's weighted sum z.
  • Non-linear: its graph is not a straight line; it bends somewhere. The bend is what lets each layer build new shapes.

Four activation functions

Name Formula Range Notes
Step 1 if z > 0 else 0 0 or 1 a hard jump; slope 0 everywhere else
Sigmoid 1 / (1 + e^(−z)) 0 to 1 smooth; output can be read as a probability
tanh (hyperbolic tangent) (e^z − e^(−z)) / (e^z + e^(−z)) −1 to 1 S-shaped, centred at 0
ReLU (rectified linear unit) max(0, z) 0 to ∞ cheap; the usual choice for hidden layers
import numpy as np
z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])                  # weighted sums coming into a neuron
print("z      :", z)
print("step   :", (z > 0).astype(float))                   # 0 or 1, a hard jump
print("sigmoid:", (1 / (1 + np.exp(-z))).round(3))         # smooth, between 0 and 1
print("tanh   :", np.tanh(z).round(3))                     # smooth, between -1 and 1
print("ReLU   :", np.maximum(0, z))                        # 0 for negatives, z itself otherwise

Output:

z      : [-3. -1.  0.  1.  3.]
step   : [0. 0. 0. 1. 1.]
sigmoid: [0.047 0.269 0.5   0.731 0.953]
tanh   : [-0.995 -0.762  0.     0.762  0.995]
ReLU   : [0. 0. 0. 1. 3.]

Which one where? ReLU in hidden layers; sigmoid at the output for a yes/no probability; no activation (linear) at the output for a number such as a price. The step is flat almost everywhere, so its slope is 0 and gradient descent has nothing to follow: smooth activations make learning possible (Episode 57).

Forward propagation with matrices

Z1 = X @ W1 + b1      # (4×2) @ (2×2) → 4×2
H  = relu(Z1)         # hidden layer outputs
out = H @ W2 + b2     # (4×2) @ (2×1) → 4×1
  • Forward propagation (forward pass): pushing inputs through the layers, one after another, to get the output.
  • Each row of X is one example, so a whole batch goes through at once.
  • W1 has shape (inputs × neurons): column j holds the weights of hidden neuron j.
import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])             # 4 examples, 2 features: a batch
W1 = np.array([[1, 1], [1, 1]]); b1 = np.array([0, -1])    # hidden layer: 2 neurons
W2 = np.array([[1], [-2]]);      b2 = np.array([0])        # output layer: 1 neuron
relu = lambda z: np.maximum(0, z)
Z1 = X @ W1 + b1;  print("Z1 = X @ W1 + b1 :", Z1.tolist(), Z1.shape)
H  = relu(Z1);     print("H  = relu(Z1)    :", H.tolist())
out = H @ W2 + b2; print("out = H @ W2 + b2:", out.ravel().tolist(), out.shape)

Output:

Z1 = X @ W1 + b1 : [[0, -1], [1, 0], [1, 0], [2, 1]] (4, 2)
H  = relu(Z1)    : [[0, 0], [1, 0], [1, 0], [2, 1]]
out = H @ W2 + b2: [0, 1, 1, 0] (4, 1)

XOR with two ReLU neurons: h1 = relu(x1 + x2), h2 = relu(x1 + x2 − 1), out = h1 − 2·h2.

A wrong shape

import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])             # 4 examples, 2 features
W1 = np.ones((3, 2)); b1 = np.zeros(3)                     # 3 hidden neurons... written as (neurons, inputs)
Z1 = X @ W1 + b1
print(Z1)

Output:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    Z1 = X @ W1 + b1
         ~~^~~~
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2)

W1 must be (inputs × neurons) = (2 × 3), not (3 × 2).

The universal approximation theorem

Each ReLU neuron adds one bend to the output's graph; with more bends, a chain of straight pieces can follow any curve more closely, like drawing a circle with many short lines.

Universal approximation theorem (Cybenko, 1989): a network with one hidden layer of sigmoid neurons can approximate any continuous function on a bounded region as closely as you like, given enough neurons. Later results extended this to other activations such as ReLU. It says nothing about how many neurons, or how to find the weights.

import numpy as np
x = np.linspace(0, 2 * np.pi, 200)[:, None]; y = np.sin(x).ravel()          # the curve to copy: one sine wave
for units in [1, 3, 5, 20, 100]:                                             # hidden ReLU neurons
    W, b = np.ones((1, units)), -np.linspace(0, 2 * np.pi, units, endpoint=False)   # each neuron bends at one spot
    H = np.c_[np.maximum(0, x @ W + b), np.ones(len(x))]                     # forward pass to the hidden layer
    w_out = np.linalg.lstsq(H, y, rcond=None)[0]                             # best output weights
    print(f"{units:3} hidden ReLU units: worst error {np.abs(H @ w_out - y).max():.4f}")

Output:

  1 hidden ReLU units: worst error 0.9406
  3 hidden ReLU units: worst error 0.2894
  5 hidden ReLU units: worst error 0.1143
 20 hidden ReLU units: worst error 0.0071
100 hidden ReLU units: worst error 0.0001

Here we placed the bends evenly by hand and fitted only the output weights with least squares; in a real network every weight is learned.

Common mistakes

Mistake Exact message / symptom Why Fix
One neuron for XOR XOR: 2/4 correct after 1000 epochs one straight line add a hidden layer
Layers with no activation same? True a line of a line is a line ReLU (or another non-linearity) after each hidden layer
Step activation in a network you train gradient 0 everywhere flat function ReLU, sigmoid, tanh
Weight matrix in the wrong shape ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2) (4×2) @ (3×2) W is (inputs × neurons)
Forgetting the bias the line must pass through the origin no shift possible add b
Reading "universal" as "easy" no recipe for the weights the theorem only says weights exist training (Episode 57)

Mini project: let a perceptron learn Asha's rule

import itertools
X = list(itertools.product([0, 1], repeat=3))                       # friends, rain, exam: all 8 days
Y = [1 if 3 * f - 2 * r - 4 * e + 1 > 0 else 0 for f, r, e in X]   # Asha's real decisions
def neuron(x, w, b): return 1 if sum(wi * xi for wi, xi in zip(w, x)) + b > 0 else 0
w, b = [0.0, 0.0, 0.0], 0.0
for epoch in range(100):                                            # perceptron learning rule
    for x, t in zip(X, Y):
        err = t - neuron(x, w, b)
        w = [wi + 0.1 * err * xi for wi, xi in zip(w, x)]; b += 0.1 * err
print("learned w =", [round(v, 1) for v in w], " b =", round(b, 1))
print("correct:", sum(neuron(x, w, b) == t for x, t in zip(X, Y)), "of 8")

Output:

learned w = [0.1, -0.1, -0.3]  b = 0.1
correct: 8 of 8

The learned weights differ from Asha's (3, −2, −4, +1) but make the same 8 decisions: many weight settings draw an equally good line.

Practice

1. Hand-build NAND and NOR neurons.

Solution
def neuron(x, w, b): return 1 if sum(wi * xi for wi, xi in zip(w, x)) + b > 0 else 0
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    print(x, "NAND =", neuron(x, [-1, -1], 1.5), " NOR =", neuron(x, [-1, -1], 0.5))

Output:

(0, 0) NAND = 1  NOR = 1
(0, 1) NAND = 1  NOR = 0
(1, 0) NAND = 1  NOR = 0
(1, 1) NAND = 0  NOR = 0

NAND = not AND, NOR = not OR: flip the signs of the weights and pick the threshold.

2. XOR with sigmoid instead of step.

Solution

Scale the step weights up so the sigmoid is nearly a step:

import math
sig = lambda z: 1 / (1 + math.exp(-z))
def neuron(x, w, b): return sig(sum(wi * xi for wi, xi in zip(w, x)) + b)
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h1 = neuron(x, [10, 10], -5); h2 = neuron(x, [10, 10], -15)     # the step weights, scaled up by 10
    out = neuron([h1, h2], [10, -10], -5)
    print(x, "->", round(out, 3))

Output:

(0, 0) -> 0.007
(0, 1) -> 0.992
(1, 0) -> 0.992
(1, 1) -> 0.007

3. Forward pass 2 → 4 → 1 with random weights.

Solution
import numpy as np; rng = np.random.default_rng(56)
X = rng.random((5, 2))                                   # 5 examples, 2 features
W1, b1 = rng.normal(size=(2, 4)), np.zeros(4)            # 2 inputs -> 4 hidden neurons
W2, b2 = rng.normal(size=(4, 1)), np.zeros(1)            # 4 -> 1 output
H = np.maximum(0, X @ W1 + b1); out = 1 / (1 + np.exp(-(H @ W2 + b2)))   # ReLU hidden, sigmoid output
print("H:", H.shape, " out:", out.shape)
print(out.ravel().round(3))

Output:

H: (5, 4)  out: (5, 1)
[0.461 0.444 0.5   0.493 0.5  ]

Outputs of exactly 0.5 mean all four ReLUs were 0 for that example, so the output sum is 0 and sigmoid(0) = 0.5.

4. Sine wave with 10 bends: worst error?

Solution
import numpy as np
x = np.linspace(0, 2 * np.pi, 200)[:, None]; y = np.sin(x).ravel()
units = 10
W, b = np.ones((1, units)), -np.linspace(0, 2 * np.pi, units, endpoint=False)
H = np.c_[np.maximum(0, x @ W + b), np.ones(len(x))]
w_out = np.linalg.lstsq(H, y, rcond=None)[0]
print(f"{units} hidden ReLU units: worst error {np.abs(H @ w_out - y).max():.4f}")

Output:

10 hidden ReLU units: worst error 0.0301

Between 5 units (0.1143) and 20 units (0.0071).

Quick quiz

Inputs x = [1, 1], weights w = [2, −1], bias b = −3, ReLU. Output?

Answer

0. z = 2 − 1 − 3 = −2; ReLU(−2) = max(0, −2) = 0.

More questions:

  1. Why can't one perceptron learn XOR?
    AnswerIt draws one straight line, and XOR's classes are not linearly separable.
  2. What happens if an MLP has no activation functions?
    AnswerThe layers multiply out into a single linear layer.
  3. A batch of 32 examples with 10 features goes into a layer of 64 neurons. Shape of W and of the output?
    AnswerW: (10, 64); output: (32, 64).
  4. Which activation would you use at the output to predict a house price?
    AnswerNone (linear), because the price can be any number.
  5. What does the universal approximation theorem not tell you?
    AnswerHow many neurons are needed or how to find the weights.

Interview questions

  1. What does a single neuron compute? activation(w · x + b).
  2. Why do neural networks need non-linear activations? Without them, any stack of layers equals one linear layer.
  3. Sigmoid vs tanh vs ReLU? Sigmoid: 0 to 1, saturates; tanh: −1 to 1, zero-centred, saturates; ReLU: max(0, z), cheap, doesn't saturate for positive inputs, the default for hidden layers.
  4. Explain the XOR problem. XOR isn't linearly separable, so a single-layer perceptron fails; one hidden layer solves it.
  5. What is forward propagation? Computing each layer's outputs from the previous layer's: Z = XW + b, A = f(Z), until the output.
  6. State the universal approximation theorem. A feed-forward network with one hidden layer and enough units can approximate any continuous function on a compact set arbitrarily well.

Glossary

Term Meaning (plain words + example)
Artificial neuron Weighted sum of inputs plus bias, through an activation: Asha's chai decision
Input A number fed into a neuron: "raining" = 1
Weight How strongly an input pushes the output: rain −2
Bias A fixed push before any input: +1 for loving chai
Perceptron A step-activation neuron with a learning rule for w and b (Rosenblatt, 1957)
Perceptron learning rule w ← w + rate × (target − output) × input
Activation function Non-linear function applied to the weighted sum
Non-linear Not a straight line; bends somewhere
Step function 1 if z > 0 else 0
Sigmoid 1 / (1 + e^(−z)), 0 to 1: sigmoid(0) = 0.5
tanh Hyperbolic tangent, −1 to 1: tanh(1) = 0.762
ReLU Rectified linear unit, max(0, z): ReLU(−3) = 0, ReLU(3) = 3
AND / OR / XOR Logic rules for two switches; XOR = exactly one on (staircase light)
Linearly separable Classes that one straight line (plane) can split
Layer Neurons that all read the same inputs
Input layer The features going into the network
Hidden layer Neurons between input and output: the OR and AND neurons
Output layer Neurons that give the prediction
Multilayer perceptron (MLP) Input → hidden layer(s) → output
Fully connected (dense) layer Every neuron reads every output of the previous layer
Deep network A network with many hidden layers
Forward propagation Computing the output layer by layer: Z1 = X @ W1 + b1, H = relu(Z1), ...
Batch Several examples processed together, one per row of X
Universal approximation theorem One hidden layer with enough neurons can approximate any continuous function (Cybenko, 1989)
Least squares Choosing weights that minimise the sum of squared errors

Summary

  • A neuron = activation(w · x + b). One neuron = one straight line: the perceptron learns AND and OR but not XOR.
  • A hidden layer fixes XOR: the multilayer perceptron.
  • Without activation functions, layers collapse into one. Know step, sigmoid, tanh, ReLU; use ReLU in hidden layers.
  • The forward pass is Z = XW + b, A = f(Z), layer by layer, for a whole batch at once.
  • Universal approximation: enough hidden neurons can copy any smooth curve (sine: 0.9406 → 0.0001), but the theorem doesn't find the weights.

Next episode: Neural networks from scratch 2: backpropagation: local gradients, the chain rule through layers, gradient checks, and training XOR in plain Python.