These are the notes for AI episode 10. Last episode you built a neural network and trained it with backpropagation. Here you feed it the two things AI deals with most, photos and sentences, see exactly why a plain dense network chokes on both, and invent the two fixes: convolutional neural networks (CNNs) for images and recurrent neural networks (RNNs) for sequences. Along the way: pixels, colour channels, filters (kernels), feature maps, weight sharing, pooling, bag of words, the hidden state, tanh, the vanishing gradient problem, LSTMs and a first look at Transformers. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real (Python 3.13).
How a computer sees: a photo is a mosaic of numbers
| A rangoli made of dots | A digital image |
|---|---|
| Up close, tiny coloured dots | Pixels: tiny squares |
| Each dot mixes red, green and blue | Colour channels: 3 numbers per pixel (0 to 255 each) |
| Step back: a peacock! | The computer only ever sees a grid of numbers |
Problem 1: a photo into a dense layer
# a 1000 x 1000 colour photo into a dense layer of 1000 neurons
width, height, colours = 1000, 1000, 3 # red, green, blue per pixel
inputs = width * height * colours
neurons = 1000
weights = inputs * neurons
print(f"inputs: {inputs:,} numbers")
print(f"weights: {weights:,} in the first layer alone")
print(f"memory: {weights * 4 / 1e9:.0f} GB (4 bytes per weight)")
Output:
inputs: 3,000,000 numbers
weights: 3,000,000,000 in the first layer alone
memory: 12 GB (4 bytes per weight)
Three billion weights and 12 GB, for one layer, for one modest photo.
Problem 2: move one pixel, and it's lost
def show(img):
for row in img:
print(" ".join("#" if p else "." for p in row))
def image_with_line(col): # 5x5 image, a vertical line
return [[1 if c == col else 0 for c in range(5)] for r in range(5)]
detector = image_with_line(1) # a dense neuron's weights:
# one weight per pixel
def dense_score(img):
return sum(img[r][c] * detector[r][c] for r in range(5) for c in range(5))
for col in [1, 2]:
img = image_with_line(col)
show(img)
print("line at column", col, "-> score", dense_score(img))
Output:
. # . . .
. # . . .
. # . . .
. # . . .
. # . . .
line at column 1 -> score 5
. . # . .
. . # . .
. . # . .
. . # . .
. . # . .
line at column 2 -> score 0
Why: each dense weight is glued to one pixel position. The same line one pixel to the right scores 0. A cat in the top-left corner and one in the bottom-right would need separate weights learned from separate examples.
The fix: slide one small filter everywhere (convolution)
Real life: checking rice for stones, your mother takes one handful at a time, uses the same check on every handful, and finds a stone wherever it is.
- Filter (kernel): a small grid of weights, e.g. 3 × 3 = 9 weights.
- At each position, multiply the window of pixels by the filter and add up (a neuron's weighted sum). Slide one step, repeat, across the whole image. This is a convolution.
- The scores form a feature map: where the pattern was found.
- Weight sharing: the same 9 weights are used at every position.
- Pooling: shrink the feature map by keeping the strongest score in each region (max pooling). In the demo the region is the whole map (global max pooling).
- Feature map size per side = image size − filter size + 1 (with a step of 1 and no padding).
Demo: convolution in Python
def image_with_line(col):
return [[1 if c == col else 0 for c in range(5)] for r in range(5)]
kernel = [[0, 1, 0], # a 3x3 filter: 9 weights
[0, 1, 0], # "is there a vertical line
[0, 1, 0]] # in this little window?"
def convolve(img): # slide the filter everywhere
size = len(img) - 2
return [[sum(img[r + i][c + j] * kernel[i][j]
for i in range(3) for j in range(3))
for c in range(size)] for r in range(size)]
for col in [1, 2]:
fmap = convolve(image_with_line(col))
print("line at column", col, "-> feature map", fmap)
print(" strongest match (max pooling):", max(max(row) for row in fmap))
Output:
line at column 1 -> feature map [[3, 0, 0], [3, 0, 0], [3, 0, 0]]
strongest match (max pooling): 3
line at column 2 -> feature map [[0, 3, 0], [0, 3, 0], [0, 3, 0]]
strongest match (max pooling): 3
The match moves with the line, and the line is found wherever it is, with 9 weights instead of one per pixel.
CNNs: edges → shapes → objects
| Layers | What their filters find |
|---|---|
| First layers | edges, corners, patches of colour |
| Middle layers | shapes made of edges: circles, eyes, wheels, textures |
| Last layers | whole objects: a face, a cat, a car |
A network built from convolution layers is a convolutional neural network (CNN). The filters are not drawn by hand: they start random and are learned with backpropagation (episode 9).
History:
- 1989 · LeNet: Yann LeCun's group at AT&T Bell Labs trained a convolutional network with backpropagation to read handwritten US zip codes; the technology was later used to read amounts on bank cheques. LeNet-5 was published in 1998. (LeNet, Wikipedia)
- 2012 · AlexNet: Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton (University of Toronto) won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error of 15.3%, more than 10.8 percentage points ahead of the runner-up. About 60 million parameters, trained on two Nvidia GTX 580 GPUs. (AlexNet, Wikipedia)
Sequences: order is the meaning
"Raju pays Neha" and "Neha pays Raju" use the same three words with opposite meanings. And sentences come in any length.
from collections import Counter
a = "Raju pays Neha"
b = "Neha pays Raju"
print(Counter(a.split()))
print(Counter(b.split()))
print("same to a word-counting model?", Counter(a.split()) == Counter(b.split()))
Output:
Counter({'Raju': 1, 'pays': 1, 'Neha': 1})
Counter({'Neha': 1, 'pays': 1, 'Raju': 1})
same to a word-counting model? True
A bag of words (counting words, ignoring order) can't tell them apart. And a dense layer has a fixed number of inputs, so it can't naturally take a sentence of any length.
The fix: read one word at a time, carry a memory (RNN)
new memory = tanh(w_memory × old memory + w_word × word)
- The memory carried from word to word is the hidden state.
- The same weights are used at every word, so any length works (weight sharing across time).
- A network that loops its output back into itself is a recurrent neural network (RNN).
- tanh is an activation like the sigmoid, but squashes any number into −1..1.
import math
word = {"Raju": 1.0, "pays": 0.5, "Neha": -1.0} # each word as a number
def rnn(sentence, w_memory=0.5, w_word=1.0):
memory = 0.0 # the hidden state
for w in sentence.split(): # one word at a time
memory = math.tanh(w_memory * memory + w_word * word[w])
return memory
print("Raju pays Neha ->", round(rnn("Raju pays Neha"), 3))
print("Neha pays Raju ->", round(rnn("Neha pays Raju"), 3))
Output:
Raju pays Neha -> -0.569
Neha pays Raju -> 0.785
Different order, different memory. In a real RNN the memory is a list of hundreds of numbers and the weights are learned with backpropagation, but the loop is the same.
The next problem: memory fades
import math
def rnn(numbers, w_memory=0.5):
memory = 0.0
for x in numbers:
memory = math.tanh(w_memory * memory + x)
return memory
for length in [3, 10, 20, 50]:
middle = [0.5] * (length - 1) # the same words after it
a = rnn([1.0] + middle) # first word: Raju
b = rnn([-1.0] + middle) # first word: Neha
print(f"{length:2} words: memory of the first word = {abs(a - b):.1e}")
Output:
3 words: memory of the first word = 1.9e-01
10 words: memory of the first word = 1.8e-05
20 words: memory of the first word = 2.8e-11
50 words: memory of the first word = 1.1e-16
At 50 words the difference is as small as floating-point rounding noise: the first word is forgotten. Why: at every word the old memory is multiplied by a small weight and squashed; repeat that and anything shrinks to nothing. During training the slopes shrink the same way going backwards, so the network can't learn long-range links: the vanishing gradient problem (gradients can also blow up instead: exploding gradients).
The fix: LSTM, a notebook with rules
| Your class notebook | An LSTM cell |
|---|---|
| Erase what's no longer needed | Forget gate: decides what to drop |
| Write down what matters ("exam on Monday!") | Input gate: decides what to store |
| Read the right page when you need it | Output gate: decides what to use now |
LSTM (long short-term memory) was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to fix vanishing and exploding gradients in RNNs. Because its cell memory isn't squashed at every step, information can survive across hundreds or thousands of steps (IEEE: Long short-term memory). LSTMs powered translation and voice typing for years.
What came next
| Idea | Strength | Weakness |
|---|---|---|
| CNN | images: shared filters slide over space | made for grids, not long-range links in text |
| RNN / LSTM | sequences: read in order with a memory | one word after another: slow, hard to spread over many GPUs |
| Transformer (2017) | every word looks at every other word at once (attention) | covered from episode 14 |
"Attention Is All You Need" (Vaswani et al., submitted 12 June 2017) introduced the Transformer, "dispensing with recurrence and convolutions entirely" (arXiv:1706.03762). It is the engine inside ChatGPT.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Pixel | one tiny square of an image | a 1000 × 1000 photo has a million |
| Colour channel | one layer of numbers per colour (red, green, blue) | 3 numbers per pixel, 0–255 |
| Dense layer | every input connects to every neuron | 3 billion weights on a photo |
| Convolution | sliding a small filter over the input, a weighted sum at each position | convolve(img) |
| Filter / kernel | the small grid of weights that slides | a 3 × 3 vertical-line filter |
| Feature map | the grid of filter scores: where the pattern was found | [[3, 0, 0], ...] |
| Weight sharing | using the same weights at every position (or every time step) | 9 weights for any image size |
| Pooling / max pooling | shrinking a feature map by keeping the strongest score per region | max(...) of the map |
| CNN | convolutional neural network: layers of convolutions | LeNet, AlexNet |
| ImageNet challenge | a yearly contest to name objects in photos | AlexNet won in 2012 |
| GPU | graphics card; does many multiplications at once | AlexNet used two |
| Sequence | data where order matters | a sentence, speech, prices over time |
| Bag of words | counting words, ignoring order | Counter(text.split()) |
| RNN | recurrent neural network: reads step by step with a memory | rnn("Raju pays Neha") |
| Hidden state | the memory an RNN carries from step to step | memory |
| tanh | an activation that squashes into −1..1 | math.tanh(0.5) |
| Vanishing gradient | slopes shrinking to nothing over many steps/layers | 1.1e-16 after 50 words |
| LSTM | long short-term memory: an RNN with forget, input and output gates | 1997, Hochreiter & Schmidhuber |
| Gate | a learned switch deciding what to forget, store or output | the forget gate |
| Transformer / attention | a model where every word looks at every other word at once | 2017; the engine of ChatGPT |
| Vocabulary | the words a model knows | the word dictionary |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Letting the filter slide off the edge | size = len(img) (forgot - 2) |
IndexError: list index out of range |
| A word the model has never seen | rnn("Priya pays Raju") |
KeyError: 'Priya' |
| An image smaller than the filter | max(max(row) for row in convolve([[1, 0], [0, 1]])) |
ValueError: max() iterable argument is empty |
| Counting words for a task where order matters | Counter(a.split()) == Counter(b.split()) |
no error; True for opposite meanings |
Fixes: number of positions = image size − filter size + 1; handle unknown words (episode 12: tokens); check the image is at least as big as the filter; use a model that reads in order.
Mini project: a tiny edge finder
Make an 8 × 8 image with a filled square in the middle (1s inside, 0s outside). Write two filters, one for vertical edges ([[-1, 0, 1]] * 3) and one for horizontal edges, convolve the image with each and print both feature maps. Where do the big positive and negative numbers appear?
Practice
- Write a horizontal-line filter and test it on lines at rows 1 and 3.
- Make a 7 × 7 image and check that the feature map is 5 × 5.
- In the fade demo, try
w_memory = 0.9. Does the first word survive longer? - Add "Priya" to the RNN's word dictionary and run "Priya pays Raju".
Solutions
1.
def image_with_hline(row): # 5x5 image, a horizontal line
return [[1 if r == row else 0 for c in range(5)] for r in range(5)]
kernel = [[0, 0, 0], # "is there a horizontal line
[1, 1, 1], # in this little window?"
[0, 0, 0]]
def convolve(img):
size = len(img) - 2
return [[sum(img[r + i][c + j] * kernel[i][j]
for i in range(3) for j in range(3))
for c in range(size)] for r in range(size)]
for row in [1, 3]:
print("line at row", row, "-> feature map", convolve(image_with_hline(row)))
line at row 1 -> feature map [[3, 3, 3], [0, 0, 0], [0, 0, 0]]
line at row 3 -> feature map [[0, 0, 0], [0, 0, 0], [3, 3, 3]]
2.
kernel = [[0, 1, 0], [0, 1, 0], [0, 1, 0]]
def convolve(img):
size = len(img) - 2
return [[sum(img[r + i][c + j] * kernel[i][j]
for i in range(3) for j in range(3))
for c in range(size)] for r in range(size)]
img = [[1 if c == 4 else 0 for c in range(7)] for r in range(7)] # 7x7, line at column 4
fmap = convolve(img)
print(len(fmap), "x", len(fmap[0]))
for row in fmap:
print(row)
5 x 5
[0, 0, 0, 3, 0]
[0, 0, 0, 3, 0]
[0, 0, 0, 3, 0]
[0, 0, 0, 3, 0]
[0, 0, 0, 3, 0]
3.
import math
def rnn(numbers, w_memory):
memory = 0.0
for x in numbers:
memory = math.tanh(w_memory * memory + x)
return memory
for w_memory in [0.5, 0.9]:
for length in [3, 10, 20, 50]:
middle = [0.5] * (length - 1)
diff = abs(rnn([1.0] + middle, w_memory) - rnn([-1.0] + middle, w_memory))
print(f"w_memory {w_memory}, {length:2} words: {diff:.1e}")
w_memory 0.5, 3 words: 1.9e-01
w_memory 0.5, 10 words: 1.8e-05
w_memory 0.5, 20 words: 2.8e-11
w_memory 0.5, 50 words: 1.1e-16
w_memory 0.9, 3 words: 5.2e-01
w_memory 0.9, 10 words: 5.1e-05
w_memory 0.9, 20 words: 4.0e-11
w_memory 0.9, 50 words: 0.0e+00
A bigger memory weight barely helps: tanh's squashing at every step also shrinks the memory. That's why the LSTM keeps a cell memory that isn't squashed at every step.
4.
import math
word = {"Raju": 1.0, "pays": 0.5, "Neha": -1.0, "Priya": -0.5}
def rnn(sentence, w_memory=0.5, w_word=1.0):
memory = 0.0
for w in sentence.split():
memory = math.tanh(w_memory * memory + w_word * word[w])
return memory
for s in ["Priya pays Raju", "Raju pays Priya"]:
print(s, "->", round(rnn(s), 3))
Priya pays Raju -> 0.811
Raju pays Priya -> -0.146
Quick quiz 1
A 7 × 7 image and a 3 × 3 filter that slides one step at a time. How big is the feature map? A: 7 × 7 · B: 3 × 3 · C: 5 × 5 · D: 4 × 4
Answer
C: 5 × 5. Image size − filter size + 1 = 7 − 3 + 1 = 5.
Quick quiz 2
You need to predict the next word of a sentence, where order changes the meaning. Before Transformers, which idea fits best? A: bag of words · B: a CNN on pixels · C: an RNN / LSTM · D: one neuron
Answer
C: an RNN / LSTM. A bag of words loses the order; a sequence where order matters needs a model that reads in order and remembers.
Interview questions
- Why are CNNs better than dense networks for images? Far fewer weights (a small shared filter instead of one weight per pixel per neuron) and they find a pattern wherever it appears (weight sharing).
- What is a feature map? The grid of scores from sliding one filter over the input; it shows where that filter's pattern appears.
- What does pooling do? It shrinks a feature map by keeping the strongest (max) or average value per region, making the network cheaper and less sensitive to small shifts.
- What is the hidden state of an RNN? The memory vector carried from one time step to the next, updated from the previous state and the current input.
- What is the vanishing gradient problem? Slopes shrink exponentially as they're passed back through many steps or layers, so early inputs barely affect learning.
- How does an LSTM help, and why did Transformers replace RNNs for language? Gates control a cell memory that isn't squashed every step, so information lasts longer. Transformers drop recurrence: every word attends to every other word at once, which captures long-range links and runs in parallel on GPUs.
Summary
- Images are grids of numbers: pixels × 3 colour channels. A dense layer on a photo needs 3 billion weights, glued to positions.
- Convolution slides one small filter everywhere: feature maps, weight sharing, pooling. Stacked, they make a CNN (LeNet 1989, AlexNet 2012).
- A bag of words loses order. An RNN reads word by word with a hidden state.
- The RNN's memory fades (vanishing gradients); LSTMs (1997) add gates to remember longer.
- Transformers (2017) look at all words at once; we build up to them in episode 14.
- Next episode: parameters, hyperparameters and compute.