These are the notes for AI episode 8. So far every model we built was a straight line that predicts a number. Here you build the tiny building block behind every neural network: the artificial neuron. Along the way: inputs, weights, bias, the weighted sum, activation functions (step, sigmoid, ReLU), the threshold, the perceptron, the decision boundary and XOR, the one problem a single neuron cannot solve. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real (Python 3.13).
How you decide: should I go out for chai?
| Your decision | In a neuron |
|---|---|
| Each thing you check (is it raining? are friends going?) | an input |
| How much each one matters (rain: a lot, friends: a bit) | a weight (can be negative: rain pushes towards "no") |
| Your mood before checking anything (lazy today) | the bias: a starting push that doesn't depend on any input |
| Add it all up, go if the total is above 0 | the weighted sum, then the activation |
Where the name comes from
| A brain neuron | An artificial neuron |
|---|---|
| Dendrites receive signals from other cells | Inputs: numbers coming in |
| Strong or weak connections | Weights: multiply each input |
| Fires if the total signal is strong enough | Activation: turns the total into an output |
In 1943, Warren McCulloch and Walter Pitts described the first mathematical (artificial) neuron. It is a simplified copy, not a real brain cell, but the name stuck.
The problem: a straight line gives impossible chances
A teacher has records for ten students: hours studied, hours slept the night before, and whether they passed (1) or failed (0). This is classification (episode 4). The only model we can train so far is a straight line.
# (hours studied, hours slept) -> passed? 1 = yes, 0 = no
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
(5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
# the best straight line through this data (episode 7's method)
def line(study, sleep):
return 0.19 * study + 0.12 * sleep - 1.05
for study, sleep in [(5, 7), (1, 3), (10, 9)]:
chance = line(study, sleep)
print(f"study {study:2}h, sleep {sleep}h -> chance of passing {chance:.0%}")
Output:
study 5h, sleep 7h -> chance of passing 74%
study 1h, sleep 3h -> chance of passing -50%
study 10h, sleep 9h -> chance of passing 193%
The weights 0.19, 0.12 and −1.05 are the least-squares line through the ten students, rounded. Why it fails: a straight line never stops; it goes up and down forever. A chance must stay between 0 and 1. We need to squash the total into a yes/no.
The fix: a decision after the sum
inputs → weighted sum (w·x + bias) → activation (fire?) → output
- Weighted sum: multiply each input by its weight, add them up, add the bias.
- Step function: if the total is ≥ 0, output 1 ("fire"), else 0. The output jumps in one step, so it's called a step function.
- Threshold: the point where it jumps (here, a total of 0).
- Inputs + weights + bias + weighted sum + activation = an artificial neuron.
Demo 2: a neuron in Python (weights set by hand)
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
(5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
def neuron(study, sleep, w_study, w_sleep, bias):
total = w_study * study + w_sleep * sleep + bias # weighted sum
return 1 if total >= 0 else 0 # step: fire or not
w_study, w_sleep, bias = 1.0, 0.5, -8.0 # set by hand
correct = 0
for study, sleep, passed in students:
guess = neuron(study, sleep, w_study, w_sleep, bias)
correct += guess == passed
print(f"study {study}h sleep {sleep}h -> guess {guess} real {passed}")
print("correct:", correct, "out of", len(students))
Output:
study 1h sleep 5h -> guess 0 real 0
study 2h sleep 8h -> guess 0 real 0
study 3h sleep 4h -> guess 0 real 0
study 4h sleep 6h -> guess 0 real 0
study 4h sleep 9h -> guess 1 real 1
study 5h sleep 3h -> guess 0 real 0
study 5h sleep 7h -> guess 1 real 1
study 6h sleep 5h -> guess 1 real 1
study 7h sleep 8h -> guess 1 real 1
study 8h sleep 4h -> guess 1 real 1
correct: 10 out of 10
The rule "study + half your sleep must reach 8" gets all ten right. But the weights were chosen by a human staring at the data. With thousands of inputs, the neuron must learn them.
Demo 3: gradient descent on a step function gets stuck
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
(5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
def step(total):
return 1 if total >= 0 else 0
def loss(w): # w = [w_study, w_sleep, bias]
return sum((step(w[0] * a + w[1] * b + w[2]) - p) ** 2
for a, b, p in students) / len(students)
w = [0.0, 0.0, -1.0] # a bad start: always says "fail"
h = 0.001
for epoch in range(1, 4):
slopes = []
for i in range(3):
nudged = w.copy(); nudged[i] += h
slopes.append((loss(nudged) - loss(w)) / h)
w = [w[i] - 0.1 * slopes[i] for i in range(3)]
print(f"epoch {epoch}: slopes {slopes} loss {loss(w)}")
Output:
epoch 1: slopes [0.0, 0.0, 0.0] loss 0.5
epoch 2: slopes [0.0, 0.0, 0.0] loss 0.5
epoch 3: slopes [0.0, 0.0, 0.0] loss 0.5
Why: a tiny nudge to a weight almost never flips a 0 into a 1, so the loss doesn't change at all. The step function is flat everywhere, with one sudden cliff. On flat ground there is no slope to follow (episode 7's gradient descent needs one).
The sigmoid: a smooth switch
| At home | Activation function |
|---|---|
| A light switch: only off or on | Step: 0 or 1, jumps suddenly |
| A fan regulator / dimmer: turns smoothly | Sigmoid: a smooth S-curve from 0 to 1 |
sigmoid(total) = 1 / (1 + e^(-total)) e ≈ 2.718
| total | sigmoid(total) |
|---|---|
| −10 | 0.0000454 |
| −2 | 0.119 |
| 0 | 0.5 |
| 2 | 0.881 |
| 10 | 0.99995 |
Any total, however big, is squashed into 0..1, which we read as a chance. And it is smooth everywhere, so a small nudge always changes the output a little: gradient descent can feel the slope.
Demo 4: the neuron learns its own weights
import math
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
(5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
def sigmoid(total): # squashes any number into 0..1
return 1 / (1 + math.exp(-total))
def loss(w): # the neuron: sigmoid of the weighted sum
return sum((sigmoid(w[0] * a + w[1] * b + w[2]) - p) ** 2 for a, b, p in students) / len(students)
w, h, lr = [0.0, 0.0, -1.0], 0.001, 0.5 # same bad start
for epoch in range(1, 3001):
slopes = []
for i in range(3):
nudged = w.copy(); nudged[i] += h
slopes.append((loss(nudged) - loss(w)) / h)
w = [w[i] - lr * slopes[i] for i in range(3)]
if epoch % 500 == 0: print(f"epoch {epoch}: loss {loss(w):.4f}")
print("learned weights:", [round(x, 2) for x in w])
for a, b in [(5, 7), (1, 3), (10, 9)]:
print(f"study {a:2}h, sleep {b}h -> chance {sigmoid(w[0] * a + w[1] * b + w[2]):.0%}")
Output:
epoch 500: loss 0.1212
epoch 1000: loss 0.0394
epoch 1500: loss 0.0296
epoch 2000: loss 0.0244
epoch 2500: loss 0.0208
epoch 3000: loss 0.0182
learned weights: [1.7, 0.88, -13.0]
study 5h, sleep 7h -> chance 84%
study 1h, sleep 3h -> chance 0%
study 10h, sleep 9h -> chance 100%
The learned weights 1.7 (study) and 0.88 (sleep) keep the same ratio as the hand-made rule (sleep counts about half as much as study), but the neuron found them on its own. The chances now stay between 0% and 100%.
Three activation functions
| Activation | What it does | Where it's used |
|---|---|---|
| Step | total ≥ 0 → 1, else 0 | the first neurons (perceptron); can't be trained by gradient descent |
| Sigmoid | smooth squash into 0..1 | the final yes/no chance of a model |
| ReLU (rectified linear unit) | max(0, total): negatives become 0, positives pass through |
the default inside most deep networks; very cheap to compute |
A bit of history: the perceptron
- 1943: Warren McCulloch and Walter Pitts describe the first mathematical neuron (weights fixed by hand).
- 1957: Frank Rosenblatt, at the Cornell Aeronautical Laboratory, creates the perceptron: a neuron with a step activation and a rule to learn its weights from examples.
- 1960: the Mark I Perceptron, a machine built to run it, is shown in public on 23 June 1960.
- 1969: Marvin Minsky and Seymour Papert's book Perceptrons shows that a single-layer perceptron cannot learn XOR.
Source: Perceptron, Wikipedia.
The next problem: the staircase light (XOR)
A staircase light has a switch at the bottom and one at the top. It is on only when the two switches are different:
| Switch 1 | Switch 2 | Light |
|---|---|---|
| 0 | 0 | off (0) |
| 0 | 1 | on (1) |
| 1 | 0 | on (1) |
| 1 | 1 | off (0) |
This pattern is XOR (exclusive or).
import math
# staircase light: two switches, light is on when they differ
switches = [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]
def neuron(w, a, b):
return 1 / (1 + math.exp(-(w[0] * a + w[1] * b + w[2])))
def loss(w):
return sum((neuron(w, a, b) - on) ** 2 for a, b, on in switches) / 4
w, h, lr = [0.5, -0.3, 0.1], 0.001, 0.5
for epoch in range(1, 5001):
slopes = []
for i in range(3):
nudged = w.copy(); nudged[i] += h
slopes.append((loss(nudged) - loss(w)) / h)
w = [w[i] - lr * slopes[i] for i in range(3)]
if epoch % 1000 == 0: print(f"epoch {epoch}: loss {loss(w):.4f}")
for a, b, on in switches:
print(f"switches {a} {b} -> light {neuron(w, a, b):.2f} (should be {on})")
Output:
epoch 1000: loss 0.2500
epoch 2000: loss 0.2500
epoch 3000: loss 0.2500
epoch 4000: loss 0.2500
epoch 5000: loss 0.2500
switches 0 0 -> light 0.50 (should be 0)
switches 0 1 -> light 0.50 (should be 1)
switches 1 0 -> light 0.50 (should be 1)
switches 1 1 -> light 0.50 (should be 0)
Why: one neuron draws one straight line
- The neuron says "yes" on one side of the line where its weighted sum is 0, and "no" on the other. That line is the decision boundary.
- For the students, one straight line splits pass from fail: the data is linearly separable.
- For XOR, put the four switch positions on the corners of a square. The ON positions (0 1 and 1 0) are on opposite corners, and so are the OFF ones. No single straight line can put both ONs on one side and both OFFs on the other.
- One neuron can only draw one line, so it can never learn XOR. The fix (next episode): use a team of neurons in layers.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Neuron (biological) | a brain cell that receives signals and fires when they are strong enough | the cells in your brain |
| Artificial neuron | a tiny model: weighted sum of inputs + bias, through an activation | sigmoid(1.7*study + 0.88*sleep - 13) |
| Input | a number fed into the neuron | hours studied |
| Weight | how much an input counts; can be negative | 1.0 for study, 0.5 for sleep |
| Bias | a starting push that doesn't depend on any input | −8 |
| Weighted sum | each input × its weight, added up, + bias | 1.0*6 + 0.5*4 - 8 = 0 |
| Activation function | turns the weighted sum into the neuron's output | step, sigmoid, ReLU |
| Step function | outputs 1 if the total ≥ 0, else 0 | 1 if total >= 0 else 0 |
| Threshold | the point where a step function jumps | a total of 0 |
| Sigmoid | smooth S-curve that squashes any number into 0..1 | sigmoid(0) = 0.5 |
e |
a fixed number in maths, about 2.718 | used in math.exp |
| ReLU | rectified linear unit: max(0, total) |
relu(-3) = 0, relu(5) = 5 |
| Perceptron | Rosenblatt's 1957 neuron with a step activation that learns its weights | Mark I Perceptron, 1960 |
| Decision boundary | the line where the weighted sum is 0; one side yes, the other no | study + 0.5*sleep = 8 |
| Linearly separable | data that one straight line can split into its classes | pass/fail students, AND, OR |
| XOR (exclusive or) | true only when the two inputs differ | the staircase light |
| Classification | predicting a category (episode 4) | pass or fail |
| Epoch | one pass of training over all the data (episode 7) | for epoch in range(1, 3001) |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Huge negative total in the sigmoid | 1 / (1 + math.exp(1000)) |
OverflowError: math range error |
| Forgetting the import | math.exp(-2) without import math |
NameError: name 'math' is not defined. Did you forget to import 'math'? |
| Training a step function with gradient descent | return 1 if total >= 0 else 0 |
no error; slopes [0.0, 0.0, 0.0], loss stuck at 0.5 |
| Reading a line's output as a chance | 0.19*study + 0.12*sleep - 1.05 |
no error; chances like -50% and 193% |
Fixes: keep totals small (scale your inputs, or clip the total, for example max(-500, min(500, total))); add import math; train with a smooth activation (sigmoid, ReLU); squash the output with a sigmoid before reading it as a chance.
Mini project: your own decision neuron
Build a neuron that decides "go out for chai?" from three inputs: raining (0/1), friends going (0/1), and money in your pocket (₹). Choose weights and a bias by hand so it says yes on a dry day with friends, then write 6 days of data and train the sigmoid version with the gradient descent loop from Demo 4. Print the learned weights. Is rain's weight negative?
Practice
- In Demo 2, change the bias. For which biases are all ten students still correct?
- Train the sigmoid neuron on AND and OR (and XOR, for comparison).
- Write
relu(total)and print it for −3, 0 and 5. - Draw XOR's four corners on paper and try to split them with one line.
Solutions
1.
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
(5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
def neuron(study, sleep, w_study, w_sleep, bias):
total = w_study * study + w_sleep * sleep + bias
return 1 if total >= 0 else 0
for bias in [-6.0, -7.0, -8.0, -8.5, -9.0, -10.0]:
correct = sum(neuron(a, b, 1.0, 0.5, bias) == p for a, b, p in students)
print(f"bias {bias:5}: {correct} out of {len(students)} correct")
bias -6.0: 7 out of 10 correct
bias -7.0: 9 out of 10 correct
bias -8.0: 10 out of 10 correct
bias -8.5: 10 out of 10 correct
bias -9.0: 7 out of 10 correct
bias -10.0: 7 out of 10 correct
The bias moves the threshold: too high (−6) and failing students pass; too low (−9) and passing students fail.
2.
import math
def neuron(w, a, b):
return 1 / (1 + math.exp(-(w[0] * a + w[1] * b + w[2])))
def train(data, epochs=5000, lr=0.5, h=0.001):
def loss(w):
return sum((neuron(w, a, b) - y) ** 2 for a, b, y in data) / len(data)
w = [0.5, -0.3, 0.1]
for _ in range(epochs):
slopes = []
for i in range(3):
nudged = w.copy(); nudged[i] += h
slopes.append((loss(nudged) - loss(w)) / h)
w = [w[i] - lr * slopes[i] for i in range(3)]
return w, loss(w)
gates = {"AND": [(0, 0, 0), (0, 1, 0), (1, 0, 0), (1, 1, 1)],
"OR": [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1)],
"XOR": [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]}
for name, data in gates.items():
w, final = train(data)
outs = [round(neuron(w, a, b), 2) for a, b, _ in data]
print(f"{name}: loss {final:.4f} outputs {outs}")
AND: loss 0.0021 outputs [0.0, 0.05, 0.05, 0.94]
OR: loss 0.0011 outputs [0.05, 0.97, 0.97, 1.0]
XOR: loss 0.2500 outputs [0.5, 0.5, 0.5, 0.5]
AND and OR are linearly separable, so one neuron learns them. XOR is not.
3.
def relu(total):
return max(0, total)
for t in [-3, 0, 5]:
print(f"relu({t}) = {relu(t)}")
relu(-3) = 0
relu(0) = 0
relu(5) = 5
4. Whatever line you draw, one side always holds an ON corner and an OFF corner together. You need two lines (two neurons) and a third neuron to combine them: that is episode 9.
Quick quiz 1
Inputs: study = 6, sleep = 4. Weights 1.0 and 0.5, bias −8, step activation. What is the output? A: 0 · B: 1 · C: 0.5 · D: −8
Answer
B: 1. Weighted sum = 1.0 × 6 + 0.5 × 4 − 8 = 0, and 0 ≥ 0, so the step fires. It sits exactly on the threshold.
Quick quiz 2
Which of these can a single neuron NOT learn? A: OR · B: AND · C: XOR · D: "light on if switch 1 is up"
Answer
C: XOR. OR, AND and "switch 1 only" can each be split by one straight line. XOR's ON corners are opposite each other, so it needs two lines.
Interview questions
- What does an artificial neuron compute? A weighted sum of its inputs plus a bias, passed through an activation function.
- What is the bias for? It shifts the threshold, so the neuron can fire (or not) even when all inputs are 0.
- Why do we need activation functions? To turn the unbounded weighted sum into a useful output (a chance, a yes/no), and, in networks, to add the bends that let layers learn more than a straight line (episode 9).
- Why can't you train a step-function neuron with gradient descent? Its slope is 0 almost everywhere, so gradient descent gets no signal.
- Sigmoid vs ReLU? Sigmoid squashes into 0..1 (good for a final probability); ReLU is
max(0, x), cheap, and the usual choice in hidden layers. - What is linear separability, and why does XOR matter? Data one straight line can split. XOR is not linearly separable, so a single perceptron can't learn it (Minsky and Papert, 1969).
Summary
- A neuron multiplies inputs by weights, adds a bias (the weighted sum) and passes it through an activation.
- A plain line gave impossible chances (−50%, 193%); the step function gives yes/no but is flat, so gradient descent gets slopes of 0.
- The smooth sigmoid lets gradient descent train the neuron: it learned weights 1.7 and 0.88 on its own.
- ReLU (
max(0, total)) is the default activation inside deep networks. - One neuron draws one straight line (its decision boundary), so it can't learn XOR.
- Next episode: neural networks: layers of neurons, hidden neurons, and backpropagation.