SyntaxPilot
Courses/Go Programming/Lesson 32
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 32 of 89

Iterators and range over functions

range over func (Go 1.23)iter.Seqiter.Seq2yield functionearly breakpush vs pull iteratorsiter.Pullslices.Allslices.Valuesslices.Collectmaps.Keysmaps.Allwriting custom iterators

These are the notes for Go episode 32. Raju's app stores a day of one million orders, and he wants just the first three orders over 50 rupees. Returning a filtered slice works but checks every order and builds a huge slice. Passing a callback avoids the slice but can't break. So you invent a callback that returns a bool, the yield function, and then use Go 1.23's range over function types so a normal for ... range loop with a normal break works. You'll meet iter.Seq and iter.Seq2, see the run-time panic when an iterator ignores yield's false, use the iterators in slices and maps (All, Values, Collect, Keys, Sorted), write custom iterators (even infinite ones), and learn push vs pull iterators with iter.Pull. Every program here runs as-is in the SyntaxPilot compiler (language: Go) with go run main.go. All outputs and errors are from real runs (Go 1.24.7, linux/amd64).

Facts and sources

From the Go blog post "Range Over Function Types" by Ian Lance Taylor (20 August 2024):

  • "Range over function types is a new language feature in the Go 1.23 release."
  • The new iter package defines type Seq[V any] func(yield func(V) bool) and type Seq2[K, V any] func(yield func(K, V) bool).
  • "A push iterator pushes each value in a sequence to a yield function." A pull iterator works the other way: each call returns the next value. iter.Pull takes a push iterator and "returns a pair of functions": the next function and a stop function.
  • slices.All and slices.Values return iterators over a slice; slices.Collect gathers an iterator's values into a slice; maps.All, maps.Keys and maps.Values return iterators over a map.

The problem: a million orders

package main

import "fmt"

var checked int

// bigOrders returns every order over 50 rupees.
func bigOrders(orders []int) []int {
    var out []int
    for _, o := range orders {
        checked++
        if o > 50 {
            out = append(out, o)
        }
    }
    return out
}

func main() {
    orders := make([]int, 1_000_000)
    for i := range orders {
        orders[i] = i * 37 % 100 // a fake day of order amounts
    }
    big := bigOrders(orders)
    fmt.Println("first three:", big[:3])
    fmt.Println("checked", checked, "orders, built a slice of", len(big))
}

Output:

first three: [74 85 59]
checked 1000000 orders, built a slice of 490000

i * 37 % 100 just makes up a day of order amounts between 0 and 99. The answer is right, but we checked all 1,000,000 orders and built a slice of 490,000 numbers to show three. That's wasted time and memory. Worse, if the data were endless (a live stream of orders), this function would never return.

Callbacks can't break

A callback is a function you pass to another function, to be called back for each item. No slice is built:

package main

import "fmt"

// forEachBig calls f for every order over 50.
func forEachBig(orders []int, f func(int)) {
    for _, o := range orders {
        if o > 50 {
            f(o)
        }
    }
}

func main() {
    orders := make([]int, 1_000_000)
    for i := range orders {
        orders[i] = i * 37 % 100 // a fake day of order amounts
    }
    n := 0
    forEachBig(orders, func(o int) {
        fmt.Println(o)
        n++
        if n == 3 {
            break
        }
    })
}

Output:

# command-line-arguments
./main.go:24:13: break is not in a loop, switch, or select

Why: break sits inside the function literal, not inside a loop. The loop lives in forEachBig, a different function, and break can't cross a function boundary. A return would only end this one call of the callback; forEachBig would just call it again for the next order.

Inventing yield

Real-life example: the chai seller on the train.

On the train In Go
The seller hands you one cup yield(v): the iterator hands you one value
"One more!" yield returns true: keep going
"Bas, enough!" yield returns false: stop (and clean up)

The fix is one small change: the callback returns a bool. true means "send more", false means "stop". By convention this callback is called yield:

package main

import "fmt"

var checked int

// yield returns false when the caller wants to stop.
func forEachBig(orders []int, yield func(int) bool) {
    for _, o := range orders {
        checked++
        if o > 50 {
            if !yield(o) {
                return
            }
        }
    }
}

func main() {
    orders := make([]int, 1_000_000)
    for i := range orders {
        orders[i] = i * 37 % 100 // a fake day of order amounts
    }
    n := 0
    forEachBig(orders, func(o int) bool {
        fmt.Println(o)
        n++
        return n < 3 // false after the third: stop
    })
    fmt.Println("checked", checked, "orders")
}

Output:

74
85
59
checked 8 orders

Only 8 orders were checked (orders 0 to 7; the big ones are at positions 2, 5 and 7). No big slice, and the work stopped the moment the caller said enough. This is the iterator pattern: a function that produces values one at a time, on demand.

Range over functions (Go 1.23)

Writing return n < 3 is clumsy. Since Go 1.23 you can range over a function that takes a yield:

package main

import (
    "fmt"
    "iter"
)

var checked int

// bigOrders returns an iterator: a func that takes yield.
func bigOrders(orders []int) iter.Seq[int] {
    return func(yield func(int) bool) {
        for _, o := range orders {
            checked++
            if o > 50 && !yield(o) {
                return
            }
        }
    }
}

func main() {
    orders := make([]int, 1_000_000)
    for i := range orders {
        orders[i] = i * 37 % 100 // a fake day of order amounts
    }
    n := 0
    for o := range bigOrders(orders) {
        fmt.Println(o)
        n++
        if n == 3 {
            break
        }
    }
    fmt.Println("checked", checked, "orders")
}

Output:

74
85
59
checked 8 orders
  • bigOrders now returns an iterator: iter.Seq[int], which is just func(yield func(int) bool).
  • for o := range bigOrders(orders) calls that function. Go writes the yield callback for you from the loop body.
  • When the body finishes normally (or hits continue), yield returns true. When the body does break (or return, or goto out of the loop), yield returns false.
  • An iterator is any function that hands out a sequence of values like this. A sequence is just a series of values, one after another.

Which functions can you range over? Exactly three shapes:

Function type Loop
func(yield func() bool) for range f { ... }
func(yield func(V) bool) (iter.Seq[V]) for v := range f { ... }
func(yield func(K, V) bool) (iter.Seq2[K, V]) for k, v := range f { ... }

Anything else is rejected:

func notIter(f func(int)) {}

for v := range notIter { ... }
cannot range over notIter (value of type func(f func(int))): func must be func(yield func(...) bool): yield func does not return bool

What break really does: always check yield

package main

import (
    "fmt"
    "iter"
)

// tokens hands out token numbers 1, 2, 3, ... forever.
func tokens() iter.Seq[int] {
    return func(yield func(int) bool) {
        for t := 1; ; t++ {
            yield(t) // BUG: ignores what yield returns
        }
    }
}

func main() {
    for t := range tokens() {
        fmt.Println("token", t)
        if t == 2 {
            break
        }
    }
}

Output:

token 1
token 2
panic: runtime error: range function continued iteration after function for loop body returned false

goroutine 1 [running]:
...

After break, yield returned false, but the iterator called it again. Go detects this at run time and panics (crashes), so the bug can't hide. The rule: if !yield(v) { return }, every time.

package main

import (
    "fmt"
    "iter"
)

// tokens hands out token numbers 1, 2, 3, ... forever.
func tokens() iter.Seq[int] {
    return func(yield func(int) bool) {
        for t := 1; ; t++ {
            if !yield(t) {
                fmt.Println("(counter closed)")
                return
            }
        }
    }
}

func main() {
    for t := range tokens() {
        fmt.Println("token", t)
        if t == 2 {
            break
        }
    }
}

Output:

token 1
token 2
(counter closed)
  • The spot where the iterator returns after false is where it cleans up: closing a file, a network connection, and so on.
  • tokens is an infinite sequence: it never ends by itself. That's fine, because values are made only when the loop asks for them. This "only when needed" style is called lazy.

iter.Seq2: two values per step

package main

import (
    "fmt"
    "iter"
)

// numbered yields two values each time: a number and an item.
func numbered(menu []string) iter.Seq2[int, string] {
    return func(yield func(int, string) bool) {
        for i, item := range menu {
            if !yield(i+1, item) {
                return
            }
        }
    }
}

func main() {
    menu := []string{"chai", "samosa", "poha"}
    for n, item := range numbered(menu) {
        fmt.Println(n, item)
    }
}

Output:

1 chai
2 samosa
3 poha

Seq gives one value per step; Seq2 gives pairs, like index and value or key and value. Both are generic types (episode 30) defined in the iter package.

Iterators in slices and maps

package main

import (
    "fmt"
    "iter"
    "maps"
    "slices"
)

func cheap(prices map[string]int) iter.Seq[string] {
    return func(yield func(string) bool) {
        for item, p := range maps.All(prices) {
            if p <= 15 && !yield(item) {
                return
            }
        }
    }
}

func main() {
    menu := []string{"chai", "samosa", "poha"}
    for i, v := range slices.All(menu) {
        fmt.Print(i, "=", v, " ")
    }
    fmt.Println()
    again := slices.Collect(slices.Values(menu))
    fmt.Println(again, len(again))
    prices := map[string]int{"chai": 10, "samosa": 15, "thali": 120}
    fmt.Println(slices.Sorted(maps.Keys(prices)))
    fmt.Println(slices.Sorted(cheap(prices)))
}

Output:

0=chai 1=samosa 2=poha 
[chai samosa poha] 3
[chai samosa thali]
[chai samosa]
Function Gives Kind
slices.All(s) index and value iter.Seq2[int, E]
slices.Values(s) values only iter.Seq[E]
slices.Collect(seq) runs an iterator, returns a new slice []E
slices.Sorted(seq) collects and sorts (needs cmp.Ordered values) []E
maps.All(m) key and value iter.Seq2[K, V]
maps.Keys(m) keys iter.Seq[K]
maps.Values(m) values iter.Seq[V]

Map iteration order is random (episode 15), so slices.Sorted(maps.Keys(m)) is the standard way to get the keys in order. Our own cheap iterator, built on maps.All, plugs into slices.Sorted just like a library one: iterators compose like pipes.

Iterators plug together: a lazy pipeline

Filter takes any iterator and returns a new iterator. It builds nothing up front. orders is an infinite source that counts how many amounts it made:

package main

import (
    "fmt"
    "iter"
)

func Filter[V any](seq iter.Seq[V], keep func(V) bool) iter.Seq[V] {
    return func(yield func(V) bool) {
        for v := range seq {
            if keep(v) && !yield(v) {
                return
            }
        }
    }
}

var made int // how many orders were made

func orders() iter.Seq[int] {
    return func(yield func(int) bool) {
        for i := 0; ; i++ {
            made++
            if !yield(i * 37 % 100) {
                return
            }
        }
    }
}

func main() {
    big := Filter(orders(), func(o int) bool { return o > 50 })
    for o := range big {
        fmt.Println(o)
        if o == 59 { // the third big order
            break
        }
    }
    fmt.Println("orders made:", made)
}

Output:

74
85
59
orders made: 8

Only 8 orders were ever made. When the loop breaks, Filter's yield returns false, so Filter returns; that makes the range seq loop inside Filter stop, so orders's yield returns false too. The stop signal travels back down the whole chain, like kitchen stations passing plates one at a time and all stopping when the waiter says enough. Filter is a generic function (episode 30), so it works for an iterator of any type.

Push vs pull

Real-life example: two bank counters.

  • Push: the chai seller comes to you and decides the pace. That's every iterator so far: the iterator pushes each value into your yield. for ... range uses push iterators.
  • The problem: merge two counters' sorted token queues into one list, smallest first. You must walk both sequences side by side, and one range loop walks only one sequence.
  • Pull: you go to the counter and say "next!". You pull each value when you want it.

iter.Pull(seq) turns a push iterator into a pull iterator. It returns two functions:

  • next() returns the next value and ok (false when the sequence is finished);
  • stop() ends the iteration early and lets the iterator clean up. Always call it, usually with defer stop() (episode 10).
package main

import (
    "fmt"
    "iter"
    "slices"
)

// merge walks two sorted queues side by side, smallest first.
func merge(a, b iter.Seq[int]) []int {
    nextA, stopA := iter.Pull(a)
    defer stopA()
    nextB, stopB := iter.Pull(b)
    defer stopB()
    var out []int
    x, okA := nextA()
    y, okB := nextB()
    for okA || okB {
        if okA && (!okB || x <= y) {
            out = append(out, x)
            x, okA = nextA()
        } else {
            out = append(out, y)
            y, okB = nextB()
        }
    }
    return out
}

func main() {
    counter1 := slices.Values([]int{1, 4, 9})
    counter2 := slices.Values([]int{2, 3, 10})
    fmt.Println(merge(counter1, counter2))
}

Output:

[1 2 3 4 9 10]

A tiny pull example, step by step:

package main

import (
    "fmt"
    "iter"
)

func countdown(n int) iter.Seq[int] {
    return func(yield func(int) bool) {
        for i := n; i >= 1; i-- {
            if !yield(i) {
                return
            }
        }
    }
}

func main() {
    next, stop := iter.Pull(countdown(2))
    defer stop()
    v, ok := next()
    fmt.Println(v, ok)
    v, ok = next()
    fmt.Println(v, ok)
    v, ok = next()
    fmt.Println(v, ok)
}
2 true
1 true
0 false

After the sequence ends, next returns the zero value and false. Use push (a range loop) by default: it's simpler and faster. Use pull only when you must control the pace yourself, like walking two sequences at once.

Mini project: an order pipeline

Tasks: Filter (keep orders over 50, still lazy), Take (first n values), an infinite Fib, and Zip (pair two sequences with iter.Pull into an iter.Seq2).

Solution
package main

import (
    "fmt"
    "iter"
    "slices"
)

// Filter yields only the values for which keep returns true. Still lazy.
func Filter[V any](seq iter.Seq[V], keep func(V) bool) iter.Seq[V] {
    return func(yield func(V) bool) {
        for v := range seq {
            if keep(v) && !yield(v) {
                return
            }
        }
    }
}

// Take yields at most n values, then stops the source too.
func Take[V any](seq iter.Seq[V], n int) iter.Seq[V] {
    return func(yield func(V) bool) {
        if n <= 0 {
            return
        }
        i := 0
        for v := range seq {
            if !yield(v) {
                return
            }
            i++
            if i == n {
                return
            }
        }
    }
}

// Fib yields 0, 1, 1, 2, 3, 5, ... forever.
func Fib() iter.Seq[int] {
    return func(yield func(int) bool) {
        a, b := 0, 1
        for {
            if !yield(a) {
                return
            }
            a, b = b, a+b
        }
    }
}

// Zip pairs up two sequences until either one runs out.
func Zip[A, B any](as iter.Seq[A], bs iter.Seq[B]) iter.Seq2[A, B] {
    return func(yield func(A, B) bool) {
        nextA, stopA := iter.Pull(as)
        defer stopA()
        nextB, stopB := iter.Pull(bs)
        defer stopB()
        for {
            a, okA := nextA()
            b, okB := nextB()
            if !okA || !okB || !yield(a, b) {
                return
            }
        }
    }
}

func main() {
    orders := slices.Values([]int{74, 12, 85, 40, 59, 90})
    big := Filter(orders, func(o int) bool { return o > 50 })
    fmt.Println(slices.Collect(big))
    fmt.Println(slices.Collect(Take(big, 2)))
    fmt.Println(slices.Collect(Take(Fib(), 10)))

    tokens := slices.Values([]int{101, 102, 103})
    names := slices.Values([]string{"Asha", "Priya"})
    for t, name := range Zip(tokens, names) {
        fmt.Println(t, name)
    }
}

Output:

[74 85 59 90]
[74 85]
[0 1 1 2 3 5 8 13 21 34]
101 Asha
102 Priya

Notice that big can be used twice: an iter.Seq built from a slice starts over each time it is ranged. Take returns as soon as it has n values, which makes the inner range seq loop break, so Fib sees false and stops: infinite sequences are safe.

Common mistakes (and the real messages)

Mistake Real message / result Why Fix
returning a filtered slice to show 3 values checked 1000000 orders, built a slice of 490000 all the work and memory up front return an iter.Seq[int]
break inside a callback break is not in a loop, switch, or select the loop is in another function range over an iterator
calling yield without checking panic: runtime error: range function continued iteration after function for loop body returned false the iterator kept going after stop if !yield(v) { return }
ranging over a func whose callback returns nothing cannot range over notIter (...): func must be func(yield func(...) bool): yield func does not return bool only func(yield func(...) bool) shapes are allowed make the callback return bool
next, _ := iter.Pull(seq) (no message: the iterator never cleans up) stop was never called next, stop := iter.Pull(seq); defer stop()
expecting maps.Keys in a fixed order (different order on each run) map order is random slices.Sorted(maps.Keys(m))

Practice

  1. (Homework) Build Filter, Take and Fib (solution: the mini project).
  2. (Homework) Remove the if !yield check from an iterator and read the panic (shown above).
  3. (Homework) Print a map sorted by key with slices.Sorted(maps.Keys(m)).
Solution 3
prices := map[string]int{"chai": 10, "samosa": 15}
for _, k := range slices.Sorted(maps.Keys(prices)) {
    fmt.Println(k, prices[k])
}
chai 10
samosa 15
  1. (Homework) Zip two sequences with iter.Pull (solution: the mini project's Zip).
  2. Write countdown(n int) iter.Seq[int] that yields n, n-1, ..., 1, and print 3 2 1 go!.
Solution 5
func countdown(n int) iter.Seq[int] {
    return func(yield func(int) bool) {
        for i := n; i >= 1; i-- {
            if !yield(i) {
                return
            }
        }
    }
}

for i := range countdown(3) {
    fmt.Print(i, " ")
}
fmt.Println("go!")
3 2 1 go!
  1. Write Enumerate[V any](seq iter.Seq[V]) iter.Seq2[int, V] that numbers the values of any sequence from 0.
Solution 6
func Enumerate[V any](seq iter.Seq[V]) iter.Seq2[int, V] {
    return func(yield func(int, V) bool) {
        i := 0
        for v := range seq {
            if !yield(i, v) {
                return
            }
            i++
        }
    }
}
  1. Why is slices.Collect(Fib()) a bad idea?
Solution 7

Fib never stops and Collect never says false, so it runs forever, using more and more memory. Cut it first: slices.Collect(Take(Fib(), 10)).

Quick quiz

1. What does this print? (From the video.)

func evens(yield func(int) bool) {
    for i := 0; ; i += 2 {
        if !yield(i) {
            return
        }
    }
}

for v := range evens {
    if v > 4 { break }
    fmt.Print(v, " ")
}

A. 0 2 4 · B. 0 2 4 6 · C. it never stops · D. compile error

Answer

A. At v = 6 the loop breaks, yield returns false, and evens returns. Real output: 0 2 4.

2. What does yield return when the loop body runs continue?

Answer

true: continue just moves on to the next value.

3. Which function turns a push iterator into a pull one, and what two functions does it return?

Answer

iter.Pull; it returns next (value, ok) and stop.

Interview questions

  1. What is an iterator in Go? Since Go 1.23, a function of the form func(yield func(V) bool) (or with zero or two values) that produces values one at a time by calling yield. iter.Seq[V] and iter.Seq2[K, V] name these types.
  2. How does break work with range over func? The compiler turns the loop body into the yield function; on break (or return), yield returns false, and the iterator must stop and return.
  3. What happens if an iterator ignores false? The runtime panics: "range function continued iteration after function for loop body returned false".
  4. Why use an iterator instead of returning a slice? No up-front work or memory, early stopping, and it works for very large or infinite sequences. A slice is still better when you need random access or the data many times.
  5. Push vs pull? Push: the iterator calls your code for each value (for range). Pull: you call next() for each value (iter.Pull), useful for walking several sequences at once. Pull iterators must be stopped with stop().
  6. How do you get a map's keys in sorted order? slices.Sorted(maps.Keys(m)).
  7. Where does an iterator do cleanup? Right after yield returns false (or when its loop ends), before it returns, often with defer.
  8. Can an iterator be infinite? Yes. Values are produced lazily, so it's safe as long as the consumer stops (with break, Take, or stop()).

Glossary

Term Meaning Example
Iterator a function that produces a sequence of values one at a time bigOrders(orders)
Sequence a series of values, one after another 74, 85, 59, ...
Callback a function passed to another function, to be called for each item forEachBig(orders, func(o int) {...})
yield the callback an iterator calls with each value; returns true (more) or false (stop) if !yield(o) { return }
Range over func (Go 1.23) for ... range over a function of the form func(yield func(...) bool) for o := range bigOrders(orders)
iter package (Go 1.23) standard package defining Seq, Seq2 and Pull import "iter"
iter.Seq[V] func(yield func(V) bool): a sequence of single values iter.Seq[int]
iter.Seq2[K, V] func(yield func(K, V) bool): a sequence of pairs index and item
Early break leaving a range-over-func loop before the end; makes yield return false if n == 3 { break }
Infinite sequence a sequence that never ends on its own tokens(), Fib()
Lazy values are produced only when asked for Filter builds nothing up front
Pipeline iterators chained so each passes values to the next, one at a time Filter(orders(), keep)
Push iterator the iterator calls your code with each value for v := range seq
Pull iterator your code asks for each value with next() next, stop := iter.Pull(seq)
iter.Pull turns a push iterator into a pull iterator; returns next and stop merge two queues
next the pull function: returns the next value and ok v, ok := next()
stop ends a pull iteration and lets the iterator clean up defer stop()
Cleanup releasing resources (files, connections) when work is done after yield returns false
slices.All iterator of index and value over a slice for i, v := range slices.All(menu)
slices.Values iterator of the values of a slice slices.Values(menu)
slices.Collect runs an iterator and returns its values as a new slice slices.Collect(seq)
slices.Sorted collects an iterator's values and sorts them slices.Sorted(maps.Keys(m))
maps.All iterator of key and value over a map for k, v := range maps.All(m)
maps.Keys iterator of a map's keys maps.Keys(prices)
maps.Values iterator of a map's values maps.Values(prices)
Merge combining two sorted lists into one sorted list [1 2 3 4 9 10]
Function literal a function written inline without a name (episode 10) func(o int) bool { ... }
Panic a run-time crash that stops the program (episode 26) range function continued iteration ...
Generic type a type with type parameters (episode 30) iter.Seq[V any]

Summary

  • A filtered slice does all the work even to show three values; a plain callback can't break.
  • Make the callback return a bool: that's yield. true = more, false = stop.
  • Since Go 1.23 you can range over functions; break makes yield return false.
  • iter.Seq[V] and iter.Seq2[K, V] name the iterator function types.
  • Always if !yield(v) { return }, or Go panics; return is where you clean up.
  • Iterators are lazy and can be infinite.
  • slices.All, Values, Collect, Sorted and maps.All, Keys, Values are ready-made iterators and helpers.
  • Push (range) is the default; pull (iter.Pull → next, stop) is for when you set the pace, like merging two sequences. Always call stop.

Next episode: Standard library tour: io and bufio. One function that reads from a string, the keyboard, a file or the network, and the one forgotten line that makes your output vanish.