SyntaxPilot
Courses/Python/Lesson 24

Python · Lesson 24 of 40 · 11 min

Iterators and generators

queued

These are the notes for Python episode 24. You've written hundreds of for loops; in this episode you find out what a for loop actually does, and you invent generators: functions that hand out values one at a time, so you can process millions of items with almost no memory. Raju's chai business now has ten shops and ten million bills a year. Building all of them in a list just to add them up eats hundreds of megabytes. We'll cover iterables vs iterators, iter() and next(), StopIteration, writing an iterator class with __iter__ and __next__, yield and how a generator pauses and resumes, generator expressions, the used-up generator trap, endless generators, and a lazy pipeline that processes a file one line at a time. Every example below runs as-is in the SyntaxPilot compiler, with its real output from Python 3.13.

The problem: ten million bills

Raju wants the total of ten million bills. The obvious way, the one we've used so far, builds a full list and then sums it. Python's tracemalloc module can tell us the peak memory our program used:

import tracemalloc

def all_bills(n):
    bills = []
    for i in range(n):
        bills.append(i * 2 + 10)
    return bills

tracemalloc.start()
total = sum(all_bills(1_000_000))
print("Total:", total)
peak = tracemalloc.get_traced_memory()[1]
print(f"Peak memory: {peak:,} bytes")

Output:

Total: 1000009000000
Peak memory: 40,444,808 bytes

(tracemalloc, short for trace memory allocations, records every block of memory Python allocates; get_traced_memory() returns the current and the peak amount in bytes. Tracing slows Python down, so we use one million bills instead of ten million; ten million would need about ten times as much. 1_000_000 is just 1000000: underscores in numbers are allowed for readability.)

About 40 MB for one million bills (so about 400 MB for ten million), just to add up numbers.

Why it fails: the list keeps all ten million numbers alive at the same time. But sum only ever needs one number at a time: add it, forget it. It's like a clerk who prints every bill and stacks them all on the desk before he starts adding: the desk fills up long before he's done.

What we want: a function that hands out one bill, waits, hands out the next, and never keeps the pile. To build it, we first need to see how a for loop asks for items.

How a for loop really works

At a bank In Python
The queue of customers an iterable: anything you can loop over (list, string, dict, file, range)
The token machine that remembers who's next an iterator: it = iter(orders)
"Next, please!": one token at a time next(it) gives the next item
"No more tokens": the counter closes StopIteration tells the loop to stop

Doing the for loop's job by hand:

orders = ["chai", "samosa"]
it = iter(orders)
print(next(it))
print(next(it))
print(next(it))

Output:

chai
samosa
Traceback (most recent call last):
  File "/tmp/tmp5jy0bsj9/main.py", line 5, in <module>
    print(next(it))
          ~~~~^^^^
StopIteration

So this loop:

for order in orders:
    print(order)

is really this:

it = iter(orders)
while True:
    try:
        order = next(it)
    except StopIteration:
        break
    print(order)

for calls iter() once, then next() again and again, and quietly stops when it sees StopIteration.

Iterable vs iterator

A list is the queue, not the token machine. It doesn't remember a position, so you can't call next() on it:

orders = ["chai", "samosa"]
print(next(orders))

Output:

Traceback (most recent call last):
  File "/tmp/tmpsd8txhv0/main.py", line 2, in <module>
    print(next(orders))
          ~~~~^^^^^^^^
TypeError: 'list' object is not an iterator
  • Iterable: has __iter__, so iter(x) works. You can loop over it many times; each for gets a fresh iterator.
  • Iterator: has __next__ (and an __iter__ that returns itself). It remembers where it is and moves forward only.

next(it, default) returns default instead of raising StopIteration when the iterator is finished.

Build your own iterator

Using dunder methods from episode 23, we can build our own token machine:

class Tokens:
    def __init__(self, last):
        self.last = last
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.last:
            raise StopIteration
        self.current += 1
        return self.current

for t in Tokens(3):
    print("Token", t)

Output:

Token 1
Token 2
Token 3
  • __iter__ returns the iterator itself.
  • __next__ gives the next token, or raises StopIteration after the last one.
  • It only ever remembers one number, current, never a list.

It works, but it's twelve lines of class, state and StopIteration just to count to three. There must be a simpler way.

yield: the chai maker who pauses

A chai maker doesn't make a hundred cups in the morning and stack them. He makes one cup and hands it over, then waits right there, remembering how much milk is left, and when someone says "one more", carries on from exactly that spot. In Python, that word is yield.

def tokens(last):
    n = 1
    while n <= last:
        print("  (making token", n, ")")
        yield n
        n += 1

gen = tokens(3)
print(type(gen))
print(next(gen))
print(next(gen))
for t in gen:
    print("Token", t)

Output:

<class 'generator'>
  (making token 1 )
1
  (making token 2 )
2
  (making token 3 )
Token 3

What happened:

  1. Calling tokens(3) ran none of its code. It returned a generator object.
  2. Each next(gen) runs the function until the next yield, hands out that value, and pauses, keeping all its local variables (n).
  3. The for loop takes whatever is left: token 3. When the function ends, the generator raises StopIteration for you.

Any function containing yield is a generator function, and calling it gives a generator: an iterator that Python builds for you, with __iter__, __next__ and StopIteration all written automatically. Same token machine, six lines.

Generators save memory

Back to the ten million bills. Same function, no list:

import tracemalloc

def all_bills(n):
    for i in range(n):
        yield i * 2 + 10

tracemalloc.start()
total = sum(all_bills(1_000_000))
print("Total:", total)
peak = tracemalloc.get_traced_memory()[1]
print(f"Peak memory: {peak:,} bytes")

Output:

Total: 1000009000000
Peak memory: 336 bytes

Same total, and a peak of a few hundred bytes, which stays the same for ten million or a billion items. sum pulls one bill, adds it, asks for the next; the pile never exists. Values made only when asked for are called lazy. (range() itself is lazy too, which is why range(10_000_000) costs almost nothing.)

Generator expressions

Remember list comprehensions from episode 16? Use round brackets instead of square ones and you get a generator expression:

import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))

print("List:", sys.getsizeof(squares_list), "bytes")
print("Generator:", sys.getsizeof(squares_gen), "bytes")
print(sum(squares_gen))

Output:

List: 8448728 bytes
Generator: 200 bytes
333332833333500000

The generator is about 200 bytes whether it will produce a million items or a billion. (sys.getsizeof of a list counts only its slots, not the numbers inside, so the real list costs even more.)

When a generator expression is the only argument of a function, you can drop its extra brackets: sum(n * n for n in range(10)).

Lazy built-ins

You've been using iterators all along: zip, map, filter, enumerate and reversed return lazy iterators, not lists.

names = ["Asha", "Ravi", "Meena"]
bills = [120, 50, 0]

pairs = zip(names, bills)
print(pairs)
print(next(pairs))
for name, amount in pairs:
    print(name, amount)

print(list(map(str.upper, names)))

Output:

<zip object at 0x7f15e21abb00>
('Asha', 120)
Ravi 50
Meena 0
['ASHA', 'RAVI', 'MEENA']

zip follows the same rules as our generators: next() used up Asha, so the loop only saw Ravi and Meena. Stay lazy, and call list() only when you really need all the values.

The used-up trap

A generator makes each value once, hands it over, and forgets it. Like chai: once it's drunk, it's gone.

gen = (n * n for n in range(4))
print(list(gen))
print(list(gen))

Output:

[0, 1, 4, 9]
[]

The second list() gets nothing, with no error. This bug is silent. Also, because a generator doesn't know its items in advance:

gen = (n * n for n in range(4))
print(len(gen))

Output:

Traceback (most recent call last):
  File "/tmp/tmpwx11uzz1/main.py", line 2, in <module>
    print(len(gen))
          ~~~^^^^^
TypeError: object of type 'generator' has no len()
gen = (n * n for n in range(4))
print(gen[0])

Output:

Traceback (most recent call last):
  File "/tmp/tmpcr8j9tvx/main.py", line 2, in <module>
    print(gen[0])
          ~~~^^^
TypeError: 'generator' object is not subscriptable

If you need the values more than once, or need len() or indexing, use a list, or create a fresh generator each time.

Endless generators

Because a generator only works when asked, it can even be infinite:

def token_machine():
    n = 1
    while True:
        yield n
        n += 1

tm = token_machine()
print(next(tm), next(tm), next(tm))
print(next(tm))

Output:

1 2 3
4

It would go on forever, but it only does work when we call next(). A list could never be infinite. (Never call list() or sum() on an endless generator; take a few items with next() or itertools.islice.)

from itertools import islice

def token_machine():
    n = 1
    while True:
        yield n
        n += 1

print(list(islice(token_machine(), 5)))

Output:

[1, 2, 3, 4, 5]

Common mistakes

Mistake What you see Why Fix
next(it) after the last item StopIteration no items left use a for loop, or next(it, default)
next(["chai", "samosa"]) TypeError: 'list' object is not an iterator a list is iterable, not an iterator next(iter(orders))
using a generator twice no error: [] the second time a generator is used up after one pass a list, or a new generator
len(gen) TypeError: object of type 'generator' has no len() the items aren't made yet sum(1 for _ in gen) or a list
gen[0] TypeError: 'generator' object is not subscriptable no indexing next(gen)
expecting tokens(3) to print nothing happens calling a generator function runs no code loop over it or call next()
return a value instead of yield you get one value, not a sequence return ends the function yield each value
list(endless_generator()) the program never finishes / runs out of memory it never stops islice or next()

Mini project: a lazy file pipeline

A pipeline is like a conveyor belt: read lines → parse → keep only chai → sum. Each stage is a generator that pulls from the stage before, so only one line travels down the belt at a time. Even a 10 GB file needs the memory of a single line. (A file object is already an iterator over its lines, from episode 20.)

with open("sales.txt", "w") as f:
    f.write("chai,10\nsamosa,15\nchai,10\nvada pav,20\nchai,10\n")

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

def parse(lines):
    for line in lines:
        item, price = line.split(",")
        yield item, int(price)

def only(name, sales):
    for item, price in sales:
        if item == name:
            yield price

chai = sum(only("chai", parse(read_lines("sales.txt"))))
print("Chai sales: Rs", chai)

Output:

Chai sales: Rs 30

Nothing happens until sum starts pulling: sum asks only, only asks parse, parse asks read_lines, which reads one line from the file.

Try it: add a skip_blank stage (solution below).

Practice

1. Add a skip_blank(lines) stage to the pipeline, and test it with a file that has blank lines.

Solution
with open("sales.txt", "w") as f:
    f.write("chai,10\n\nsamosa,15\n\nchai,10\n")

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

def skip_blank(lines):
    for line in lines:
        if line:
            yield line

def parse(lines):
    for line in lines:
        item, price = line.split(",")
        yield item, int(price)

print(sum(price for item, price in parse(skip_blank(read_lines("sales.txt")))))

Output:

35

Without skip_blank, the empty line would crash parse with ValueError: not enough values to unpack (expected 2, got 1).

2. A generator evens(limit) that yields even numbers from 0 up to limit.

Solution
def evens(limit):
    n = 0
    while n <= limit:
        yield n
        n += 2

print(list(evens(10)))

Output:

[0, 2, 4, 6, 8, 10]

3. An endless fibonacci() generator; print the first 10 numbers.

Solution
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
print([next(fib) for _ in range(10)])

Output:

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

4. Rewrite the Tokens class from this episode as a class whose __iter__ is a generator (so the object can be looped over many times).

Solution
class Tokens:
    def __init__(self, last):
        self.last = last

    def __iter__(self):
        n = 1
        while n <= self.last:
            yield n
            n += 1

t = Tokens(3)
print(list(t))
print(list(t))

Output:

[1, 2, 3]
[1, 2, 3]

Because __iter__ is a generator function, every loop gets a fresh generator: Tokens is now a reusable iterable, not a one-time iterator.

Quick quiz

1. The video's quiz: what does this print?

def f():
    print("A")
    yield 1
    print("B")
    yield 2

g = f()
print("start")
print(next(g))
Answer

Output:

start
A
1

Calling f() runs nothing. The first next() runs up to the first yield, printing A, and hands out 1. B never runs, because nobody asked for a second value.

2. What does this print?

nums = [1, 2, 3]
it = iter(nums)
print(next(it), list(it), list(nums))
Answer

Output:

1 [2, 3] [1, 2, 3]

The iterator already handed out 1, so list(it) gets the rest. The list itself is untouched and can be looped over again.

3. Which uses less memory for a billion items: [x for x in range(10**9)] or (x for x in range(10**9))?

Answer

The generator expression: it stores only its current state (about 200 bytes). The list would need many gigabytes.

Interview questions

  1. Iterable vs iterator? An iterable can produce an iterator (__iter__); an iterator produces values one at a time (__next__), remembers its position, and is itself iterable (its __iter__ returns self). Lists are iterables, not iterators.
  2. How does a for loop work internally? It calls iter() on the object once, then calls next() repeatedly, assigning each value to the loop variable, until StopIteration is raised, which it catches silently.
  3. What is a generator? An iterator created by a function containing yield (or a generator expression). It produces values lazily and pauses between them, keeping its local state.
  4. yield vs return? return ends the function and sends back one value; yield sends back a value and pauses, resuming from the same point on the next next().
  5. Why use generators? Memory efficiency (one item at a time), working with huge or infinite sequences, and building lazy pipelines where stages are composed without intermediate lists.
  6. What happens when you iterate a generator twice? The second pass gets nothing: generators are single-use. Re-create it, or use a list if you need multiple passes.
  7. Generator expression vs list comprehension? ( ) creates a lazy generator; [ ] builds the full list in memory immediately. Use the generator when you only need to pass over the values once (e.g. into sum, max, any).
  8. What is StopIteration? The exception an iterator raises to signal it's exhausted. Inside a generator, just let the function end (or return); don't raise it yourself (that becomes a RuntimeError since Python 3.7).
  9. What does yield from do? It delegates to another iterable, yielding each of its values: yield from other_gen() (a preview; useful for nested generators).
  10. Name some lazy built-ins. range, map, filter, zip, enumerate, reversed, file objects, dict.items() views, and everything in itertools.

Glossary

Term Meaning Example
Iterable Anything you can loop over; iter(x) works a list, string, dict, file, range
Iterator An object that gives the next item on next() and remembers its position it = iter(orders)
iter() Gets an iterator from an iterable iter(["chai", "samosa"])
next() Gets the next item from an iterator next(it) → 'chai'
StopIteration The signal that an iterator has no more items after the last next()
Iteration protocol The rule: __iter__ returns an iterator, __next__ gives items, StopIteration ends how for works
__iter__ Dunder method called by iter() return self
__next__ Dunder method called by next() return self.current
Generator function A function containing yield def tokens(last): ... yield n
Generator The iterator object a generator function returns gen = tokens(3)
yield Hands out a value and pauses the function yield n
Lazy evaluation Making values only when they're asked for generators, range
Eager evaluation Making all values up front a list comprehension
Generator expression A comprehension with round brackets, giving a generator (n * n for n in range(4))
Exhausted (used up) An iterator that has no items left list(gen) → [] the second time
Infinite generator A generator that never stops (while True) token_machine()
itertools.islice Takes a slice of any iterator, even an endless one islice(gen, 5)
Pipeline Stages chained so each pulls from the one before only(parse(read_lines(...)))
Peak memory The most memory a program used at once tracemalloc.get_traced_memory()[1]
tracemalloc Standard-library module that measures memory Python allocates tracemalloc.get_traced_memory()[1]
zip, map, filter Built-ins that return lazy iterators zip(names, bills)
sys.getsizeof() The size of an object itself, in bytes sys.getsizeof(gen) → 200
Megabyte (MB) About a million bytes 40,444,808 bytes ≈ 40 MB
TypeError An operation not supported by that type len(gen), next([...])

Summary

  • A list keeps every item in memory at once; a generator keeps only its current state.
  • for = iter() once, then next() until StopIteration.
  • An iterable is the queue (can be looped many times); an iterator is the token machine (remembers its place, forward only).
  • An iterator class needs __iter__ and __next__, but yield writes all of that for you.
  • Calling a generator function runs no code; each next() runs to the next yield and pauses.
  • Generator expressions use ( ) instead of [ ] and cost about 200 bytes regardless of size.
  • A generator is used up after one pass, and has no len() or indexing.
  • Chained generators form lazy pipelines that process huge files one line at a time.

Next episode: Raju wants to time every function in his billing app without editing fifty functions by hand. We invent decorators, the @ sign you already met on @property.