SyntaxPilot
Courses/Go Programming/Lesson 54
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 54 of 89

Concurrency bugs and testing concurrent code

goroutine leaksblocked senders and receiversdeadlocklivelockstarvationunbounded goroutine creationforgotten cancelgoleak (concept)testing/synctest (Go 1.25) and fake timestress testing with -count and -racemutex vs channel decision guide

These are the notes for Go episode 54. A function that asks two suppliers for a price leaves one goroutine leak behind on every call (a blocked sender), fixed with a buffer, and caught by a test that counts goroutines, the idea behind goleak. go vet catches a forgotten cancel. Two cooks taking a knife and a board in opposite orders deadlock; one lock order fixes it. Two polite waiters in a corridor show livelock, and a cook who always serves VIPs first shows starvation. A sale that starts a hundred thousand goroutines shows why goroutines need a bound. Then testing: a reminder test that takes real seconds becomes instant and exact with testing/synctest, its bubble and fake clock; a stress run with -count=300 passes despite a data race, and -race finds it at once. Last, a guide to choosing between a mutex and a channel.

All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Our compiler has Go 1.24, where testing/synctest is an experiment turned on with GOEXPERIMENT=synctest and the entry point is synctest.Run; from Go 1.25 it is a normal package and you call synctest.Test(t, ...) (both shown below).

Facts and sources

  • Go 1.25 release notes: "The new testing/synctest package provides support for testing concurrent code." Test "runs a test function in an isolated "bubble"." Wait "waits for all goroutines in the current bubble to block." "This package was first available in Go 1.24 under GOEXPERIMENT=synctest, with a slightly different API. The experiment has now graduated to general availability."
  • testing/synctest documentation: "Any goroutines started within the bubble are also part of the bubble." "The initial time is midnight UTC 2000-01-01." "Time in a bubble only advances when every goroutine in the bubble is durably blocked." A goroutine is durably blocked when it "can only be unblocked by another goroutine in the same bubble". "If the goroutines in the bubble become deadlocked, the test fails."
  • goleak README: "Goroutine leak detector to help avoid Goroutine leaks." Import path go.uber.org/goleak; defer goleak.VerifyNone(t) in a test, or goleak.VerifyTestMain(m) in TestMain.
  • Share Memory By Communicating (Andrew Gerrand, 13 July 2010): "Do not communicate by sharing memory; instead, share memory by communicating."
  • Go source, internal/sync/mutex.go: "If a waiter fails to acquire the mutex for more than 1ms, it switches mutex to the starvation mode. In starvation mode ownership of the mutex is directly handed off from the unlocking goroutine to the waiter at the front of the queue."
  • go help testflag: -count n "Run each test, benchmark, and fuzz seed n times (default 1)."
  • go vet's lostcancel check (Go source): "the cancel function returned by context.%s should be called, not discarded, to avoid a context leak".

Goroutine leaks

A goroutine leak is a goroutine that can never finish. Go never frees a blocked goroutine: it keeps its stack and everything it points to until the program exits. In a server that runs for weeks, leaks pile up until memory runs out.

The obvious way: keep the first answer

package main

import (
    "fmt"
    "runtime"
    "time"
)

// ask two suppliers for today's paneer price, keep the first answer
func firstPrice() int {
    ch := make(chan int) // unbuffered: a send waits for a receiver
    go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }() // supplier A
    go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }() // supplier B
    return <-ch // we receive ONE answer and leave
}

func main() {
    for i := 1; i <= 1000; i++ { // the app asks 1000 times a day
        firstPrice()
    }
    time.Sleep(10 * time.Millisecond)
    fmt.Println("price asked 1000 times")
    fmt.Println("goroutines still alive:", runtime.NumGoroutine())
}

Real output:

price asked 1000 times
goroutines still alive: 1001

Why: supplier B answers second. Its send on an unbuffered channel waits for a receiver (episode 47), but firstPrice has already returned, so nobody will ever receive. That goroutine waits forever: one leak per call.

The two kinds of stuck goroutine

Kind What it looks like Example
blocked sender ch <- v, but nobody will ever receive supplier B above
blocked receiver <-ch or for range ch, but nobody will ever send or close a worker whose job channel is never closed

Three fixes: give the channel room for every send (a buffer); make every send or receive also select on ctx.Done() (episode 52); make sure the sender closes a channel that someone ranges over.

The fix: a buffer for every answer

package main

import (
    "fmt"
    "runtime"
    "time"
)

// ask two suppliers for today's paneer price, keep the first answer
func firstPrice() int {
    ch := make(chan int, 2) // room for BOTH answers: no sender waits
    go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }() // supplier A
    go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }() // supplier B
    return <-ch // we receive ONE answer and leave
}

func main() {
    for i := 1; i <= 1000; i++ { // the app asks 1000 times a day
        firstPrice()
    }
    time.Sleep(10 * time.Millisecond)
    fmt.Println("price asked 1000 times")
    fmt.Println("goroutines still alive:", runtime.NumGoroutine())
}

Real output:

price asked 1000 times
goroutines still alive: 1

Catching leaks in tests (the goleak idea)

t.Cleanup(f) registers f to run after the test finishes (episode 40). Counting goroutines before and after turns a leak into a failing test:

shop.go:

package shop

import "time"

// FirstPrice asks two suppliers and keeps the first answer
func FirstPrice() int {
    ch := make(chan int)
    go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }()
    go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }()
    return <-ch
}

shop_test.go:

package shop

import (
    "runtime"
    "testing"
    "time"
)

// a tiny leak checker: the goroutine count must be back where it started
func checkLeaks(t *testing.T) {
    before := runtime.NumGoroutine()
    t.Cleanup(func() {
        time.Sleep(20 * time.Millisecond) // let finished goroutines exit
        if after := runtime.NumGoroutine(); after > before {
            t.Errorf("leaked %d goroutine(s)", after-before)
        }
    })
}

func TestFirstPrice(t *testing.T) {
    checkLeaks(t)
    if p := FirstPrice(); p != 410 {
        t.Fatalf("got %d, want 410", p)
    }
}

Commands:

$ go test

Real output:

$ go test
--- FAIL: TestFirstPrice (0.02s)
    shop_test.go:15: leaked 1 goroutine(s)
FAIL
exit status 1
FAIL    shop    0.024s

goleak (go.uber.org/goleak, from Uber, "Goroutine leak detector to help avoid Goroutine leaks") does this check for you. Usage from its README:

func TestA(t *testing.T) {
    defer goleak.VerifyNone(t)

    // test logic here.
}

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

The forgotten cancel

context.WithTimeout and WithCancel return a cancel function. Throwing it away keeps the context's timer and its link to the parent alive until the timeout fires, here an hour, even though the function finished long ago: a forgotten cancel, a kind of leak. go vet (episode 2) reports it:

main.go:

package main

import (
    "context"
    "fmt"
    "time"
)

func askSupplier(parent context.Context) error {
    ctx, _ := context.WithTimeout(parent, time.Hour) // the cancel func is thrown away
    select {
    case <-time.After(5 * time.Millisecond):
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func main() {
    fmt.Println(askSupplier(context.Background()))
}

Commands:

$ go vet
$ go run .

Real output (go vet fails, then go run still works, which is why this bug is easy to miss):

$ go vet
$ go run .
<nil>
# shop
# [shop]
./main.go:10:10: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

The fix is always: ctx, cancel := context.WithTimeout(parent, d) followed by defer cancel().

Deadlock

In the kitchen In Go
Asha takes the knife, then needs the board lock knife, then board
Ravi takes the board, then needs the knife lock board, then knife
each waits for the other, forever deadlock
rule: always the knife first lock ordering

A deadlock is a set of goroutines each waiting for something another one holds, so none can ever continue. Here it is a circular wait on two mutexes:

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var knife, board sync.Mutex
    var wg sync.WaitGroup
    wg.Add(2)
    go func() { // cook Asha: knife first, then board
        defer wg.Done()
        knife.Lock()
        time.Sleep(10 * time.Millisecond)
        board.Lock()
        fmt.Println("Asha chopped onions")
        board.Unlock()
        knife.Unlock()
    }()
    go func() { // cook Ravi: board first, then knife
        defer wg.Done()
        board.Lock()
        time.Sleep(10 * time.Millisecond)
        knife.Lock()
        fmt.Println("Ravi chopped tomatoes")
        knife.Unlock()
        board.Unlock()
    }()
    wg.Wait()
}

Real output (the message lists every goroutine and the line where it is blocked):

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0xc00007e040?)
    /usr/local/go1.24.7/src/runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0x0?)
    /usr/local/go1.24.7/src/sync/waitgroup.go:118 +0x48
main.main()
    main.go:31 +0x125

goroutine 6 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
    /usr/local/go1.24.7/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000010118)
    /usr/local/go1.24.7/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
    /usr/local/go1.24.7/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
    /usr/local/go1.24.7/src/sync/mutex.go:46
main.main.func1()
    main.go:17 +0x9b
created by main.main in goroutine 1
    main.go:13 +0xb9

goroutine 7 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
    /usr/local/go1.24.7/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000010110)
    /usr/local/go1.24.7/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
    /usr/local/go1.24.7/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
    /usr/local/go1.24.7/src/sync/mutex.go:46
main.main.func2()
    main.go:26 +0x9b
created by main.main in goroutine 1
    main.go:22 +0x119
exit status 2

The runtime reports all goroutines are asleep - deadlock! only when every goroutine is blocked. In a server, where other goroutines (timers, the HTTP listener) keep running, a deadlock simply hangs.

The fix: one lock order

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    var knife, board sync.Mutex
    var wg sync.WaitGroup
    wg.Add(2)
    go func() { // cook Asha: knife first, then board
        defer wg.Done()
        knife.Lock()
        time.Sleep(10 * time.Millisecond)
        board.Lock()
        fmt.Println("Asha chopped onions")
        board.Unlock()
        knife.Unlock()
    }()
    go func() { // cook Ravi: the SAME order, knife first, then board
        defer wg.Done()
        knife.Lock()
        time.Sleep(10 * time.Millisecond)
        board.Lock()
        fmt.Println("Ravi chopped tomatoes")
        board.Unlock()
        knife.Unlock()
    }()
    wg.Wait()
}

Real output (who chops first changes between runs):

Ravi chopped tomatoes
Asha chopped onions

Choose one global order for your locks, write it down in a comment, and use it everywhere. Holding a lock while calling unknown code (a callback, another package) is how hidden orderings sneak in.

Livelock

A livelock is when goroutines keep changing state in response to each other, so they are busy, not blocked, but make no progress, like two people in a corridor who both step aside the same way again and again. In code: each cook takes one tool, tries the other with TryLock, fails, puts his tool down and retries, in lockstep with the other cook. The runtime cannot detect it, because nothing is blocked; the program burns CPU. Fixes: random backoff (wait a random time before retrying) or avoid retrying at all with lock ordering.

// livelock-prone: both cooks retry at the same rhythm
for {
    knife.Lock()
    if board.TryLock() {
        chop()
        board.Unlock()
        knife.Unlock()
        return
    }
    knife.Unlock()
    time.Sleep(time.Millisecond) // fix: time.Duration(rand.IntN(5)) * time.Millisecond
}

Starvation

Starvation is when a goroutine or task is ready but never gets the resource, because others always win.

package main

import "fmt"

func main() {
    vip := make(chan int, 2000)
    normal := make(chan int, 2000)
    for i := 0; i < 2000; i++ { // a busy evening: both queues are full
        vip <- i
        normal <- i
    }
    servedVIP, servedNormal := 0, 0
    for n := 0; n < 1000; n++ { // the cook serves 1000 plates
        select {
        case <-vip: // ALWAYS check VIPs first...
            servedVIP++
        default:
            select { // ...normal customers only when no VIP waits
            case <-vip:
                servedVIP++
            case <-normal:
                servedNormal++
            }
        }
    }
    fmt.Println("VIP served:", servedVIP, " normal served:", servedNormal)
}

Real output:

VIP served: 1000  normal served: 0

The fix: one select

When several cases of one select are ready, Go picks one at random (uniformly), so no case can starve (episode 49):

package main

import "fmt"

func main() {
    vip := make(chan int, 2000)
    normal := make(chan int, 2000)
    for i := 0; i < 2000; i++ {
        vip <- i
        normal <- i
    }
    servedVIP, servedNormal := 0, 0
    for n := 0; n < 1000; n++ {
        select { // one select: when both are ready, Go picks one at random
        case <-vip:
            servedVIP++
        case <-normal:
            servedNormal++
        }
    }
    fmt.Println("VIP served:", servedVIP, " normal served:", servedNormal)
}

Real output (about half and half; the exact split changes every run):

VIP served: 519  normal served: 481

If you need priority, bound it (for example, at most 3 VIPs in a row). sync.Mutex protects itself against starvation: after a waiter has waited more than 1 ms, the mutex switches to starvation mode and hands the lock directly to the waiter at the front of the queue.

Unbounded goroutine creation

package main

import (
    "fmt"
    "runtime"
)

func main() {
    var before runtime.MemStats
    runtime.ReadMemStats(&before)
    wait := make(chan struct{})
    for i := 0; i < 100_000; i++ { // a sale! one goroutine per request, no limit
        go func() { <-wait }() // each one waits for a slow database
    }
    var after runtime.MemStats
    runtime.ReadMemStats(&after)
    fmt.Println("goroutines:", runtime.NumGoroutine())
    fmt.Println("extra memory (MB):", (after.Sys-before.Sys)/1_000_000)
    close(wait)
}

Real output:

goroutines: 100001
extra memory (MB): 267

Goroutines are cheap but not free: each has its own stack (episode 43). Starting one per request with no limit lets a big enough burst exhaust memory. Bound them with a worker pool or a semaphore (episode 52), errgroup.SetLimit (episode 53), or by refusing extra requests (load shedding).

Testing with fake time: testing/synctest

The obvious way: real time

shop.go:

package shop

import "time"

// Remind sends a message when the order has waited too long
func Remind(after time.Duration, out chan<- string) {
    time.Sleep(after)
    out <- "sorry for the wait, a free lassi is coming!"
}

shop_test.go:

package shop

import (
    "testing"
    "time"
)

func TestRemind(t *testing.T) {
    out := make(chan string, 1)
    go Remind(2*time.Second, out) // the real rule is 10 minutes!
    select {
    case msg := <-out:
        t.Log("got:", msg)
    case <-time.After(3 * time.Second):
        t.Fatal("no reminder")
    }
}

Commands:

$ go test -v

Real output:

$ go test -v
=== RUN   TestRemind
    shop_test.go:13: got: sorry for the wait, a free lassi is coming!
--- PASS: TestRemind (2.00s)
PASS
ok      shop    2.002s

Why it fails: two real seconds per test, and it doesn't even test the real 10-minute rule. Shorter sleeps make it flaky: a test that sometimes passes and sometimes fails without any code change, because a busy machine runs late.

The fix: a bubble with a fake clock

On a film set In testing/synctest
a closed set, its own world a bubble: the test and every goroutine it starts
"Ten minutes later..." a fake clock that jumps when everyone is blocked
wait until every actor is still synctest.Wait()

Inside a bubble, time starts at midnight UTC on 1 January 2000 and only moves when every goroutine in the bubble is durably blocked (blocked on something only another bubble goroutine can unblock: a channel made in the bubble, a timer, sync.WaitGroup...). Then the clock jumps straight to the next timer. synctest.Wait() waits until every other goroutine in the bubble is durably blocked, so you can check state at an exact moment.

Go 1.24 (our compiler), with the experiment turned on by the first command:

shop.go:

package shop

import "time"

func Remind(after time.Duration, out chan<- string) {
    time.Sleep(after)
    out <- "sorry for the wait, a free lassi is coming!"
}

shop_test.go:

package shop

import (
    "testing"
    "testing/synctest"
    "time"
)

func TestRemind(t *testing.T) {
    synctest.Run(func() { // a bubble with a fake clock (Go 1.25+: synctest.Test)
        start := time.Now()
        out := make(chan string, 1)
        go Remind(10*time.Minute, out) // the REAL rule: 10 minutes
        time.Sleep(10*time.Minute - time.Second)
        synctest.Wait() // wait until every goroutine in the bubble is blocked
        if len(out) != 0 {
            t.Fatal("reminder came too early")
        }
        msg := <-out
        t.Log("got:", msg, "after", time.Since(start))
    })
}

Commands:

$ go env -w GOEXPERIMENT=synctest
$ go test -v

Real output:

$ go env -w GOEXPERIMENT=synctest
$ go test -v
=== RUN   TestRemind
    shop_test.go:20: got: sorry for the wait, a free lassi is coming! after 10m0s
--- PASS: TestRemind (0.00s)
PASS
ok      shop    0.003s

Ten minutes of fake time, 0.00s of real time, exact on every run.

Go 1.25 and later (no GOEXPERIMENT):

func TestRemind(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        start := time.Now()
        out := make(chan string, 1)
        go Remind(10*time.Minute, out)
        time.Sleep(10*time.Minute - time.Second)
        synctest.Wait()
        if len(out) != 0 {
            t.Fatal("reminder came too early")
        }
        msg := <-out
        t.Log("got:", msg, "after", time.Since(start))
    })
}

A bubble catches deadlocks

shop.go:

package shop

func Remind(out chan<- string) {
    // bug: forgot to send anything
}

shop_test.go:

package shop

import (
    "testing"
    "testing/synctest"
)

func TestRemind(t *testing.T) {
    synctest.Run(func() {
        out := make(chan string)
        go Remind(out)
        <-out // waits for a message that never comes
    })
}

Commands:

$ go env -w GOEXPERIMENT=synctest
$ go test

Real output (shortened):

$ go env -w GOEXPERIMENT=synctest
$ go test
--- FAIL: TestRemind (0.00s)
panic: deadlock: all goroutines in bubble are blocked [recovered]
    panic: deadlock: all goroutines in bubble are blocked

goroutine 6 [running]:
testing.tRunner.func1.2({0x5477e0, 0x5afcc0})
    /usr/local/go1.24.7/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
    /usr/local/go1.24.7/src/testing/testing.go:1737 +0x35e
panic({0x5477e0?, 0x5afcc0?})
    /usr/local/go1.24.7/src/runtime/panic.go:792 +0x132
internal/synctest.Run(0x584628)
    /usr/local/go1.24.7/src/runtime/synctest.go:209 +0x245
testing/synctest.Run(...)
    /usr/local/go1.24.7/src/testing/synctest/synctest.go:38
shop.TestRemind(0xc0000de000?)
    shop_test.go:9 +0x1b
testing.tRunner(0xc0000de000, 0x584540)
    /usr/local/go1.24.7/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
    /usr/local/go1.24.7/src/testing/testing.go:1851 +0x413
exit status 2
FAIL    shop    0.004s

Without a bubble, this test would hang until go test's timeout (10 minutes by default).

Stress testing: -count and -race

Stress testing means running a test many times, often under load, to shake out rare failures. -count=n runs each test n times (and also bypasses the test cache). It raises the odds of seeing a timing bug, but passing many times proves nothing:

shop.go:

package shop

import "sync"

// Tally adds up orders from 2 counters at the same time
func Tally(n int) int {
    total := 0
    var wg sync.WaitGroup
    for c := 0; c < 2; c++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < n; i++ {
                total++ // two goroutines, no lock: a data race
            }
        }()
    }
    wg.Wait()
    return total
}

shop_test.go:

package shop

import "testing"

func TestTally(t *testing.T) {
    if got := Tally(1000); got != 2000 {
        t.Errorf("Tally(1000) = %d, want 2000", got)
    }
}

Commands:

$ go test
$ go test -count=300

Real output:

$ go test
PASS
ok      shop    0.007s
$ go test -count=300
PASS
ok      shop    0.004s

The race detector (-race, episode 46) watches every memory access while the test runs and reports any unsynchronised access from two goroutines, even if the result happened to be right:

shop.go:

package shop

import "sync"

// Tally adds up orders from 2 counters at the same time
func Tally(n int) int {
    total := 0
    var wg sync.WaitGroup
    for c := 0; c < 2; c++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < n; i++ {
                total++ // two goroutines, no lock: a data race
            }
        }()
    }
    wg.Wait()
    return total
}

shop_test.go:

package shop

import "testing"

func TestTally(t *testing.T) {
    if got := Tally(1000); got != 2000 {
        t.Errorf("Tally(1000) = %d, want 2000", got)
    }
}

Commands:

$ go test
$ go test -race

Real output:

$ go test
PASS
ok      shop    0.002s
$ go test -race
==================
WARNING: DATA RACE
Read at 0x00c00011a228 by goroutine 9:
  shop.Tally.func1()
      shop.go:14 +0xa4

Previous write at 0x00c00011a228 by goroutine 8:
  shop.Tally.func1()
      shop.go:14 +0xb6

Goroutine 9 (running) created at:
  shop.Tally()
      shop.go:11 +0x84
  shop.TestTally()
      shop_test.go:6 +0x2e
  testing.tRunner()
      /usr/local/go1.24.7/src/testing/testing.go:1792 +0x225
  testing.(*T).Run.gowrap1()
      /usr/local/go1.24.7/src/testing/testing.go:1851 +0x44

Goroutine 8 (finished) created at:
  shop.Tally()
      shop.go:11 +0x84
  shop.TestTally()
      shop_test.go:6 +0x2e
  testing.tRunner()
      /usr/local/go1.24.7/src/testing/testing.go:1792 +0x225
  testing.(*T).Run.gowrap1()
      /usr/local/go1.24.7/src/testing/testing.go:1851 +0x44
==================
--- FAIL: TestTally (0.00s)
    testing.go:1490: race detected during execution of test
FAIL
exit status 1
FAIL    shop    0.012s

In CI, run both together, for example go test -race -count=20 ./....

Mutex or channel?

Use a mutex when... Use a channel when...
several goroutines share one piece of state you hand data, or ownership of it, to another goroutine
each access is a short update work flows through stages: pipelines, worker pools
caches, counters, maps, stock levels results, events, signals (done, cancel)

The Go slogan: "Do not communicate by sharing memory; instead, share memory by communicating." It's a guide, not a law: pick whichever makes the code simplest, and test it with -race.

Mini project: the bug hunt

The kitchen service below has five planted bugs. Find and fix each one, writing a test that fails before the fix.

// ---- kitchen.go ----
package kitchen

import (
    "context"
    "sync"
    "time"
)

var knife, board sync.Mutex
var served int // bug 5: written by many goroutines, no lock

// bug 1: the slower supplier leaks
func SupplierPrice() int {
    ch := make(chan int)
    go func() { time.Sleep(time.Millisecond); ch <- 410 }()
    go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }()
    return <-ch
}

// bug 2: the cancel function is discarded
func Ask(parent context.Context) error {
    ctx, _ := context.WithTimeout(parent, time.Hour)
    <-time.After(time.Millisecond)
    return ctx.Err()
}

// bug 3: opposite lock orders
func ChopOnions()   { knife.Lock(); board.Lock(); board.Unlock(); knife.Unlock() }
func ChopTomatoes() { board.Lock(); knife.Lock(); knife.Unlock(); board.Unlock() }

// bug 4 (to test): Late reports on out if a delivery takes longer than 15 minutes
func Late(delivered <-chan struct{}, out chan<- string) {
    select {
    case <-delivered:
    case <-time.After(15 * time.Minute):
        out <- "late delivery: refund"
    }
}

func Serve() { served++ }
Solution
  1. make(chan int, 2), and a test with the goroutine-count check (or goleak.VerifyNone).
  2. ctx, cancel := context.WithTimeout(parent, time.Hour) + defer cancel(); go vet stops complaining.
  3. Make ChopTomatoes take knife first, then board.
  4. A synctest test (Go 1.24 experiment form; use synctest.Test(t, ...) from Go 1.25):

kitchen.go:

package kitchen

import "time"

func Late(delivered <-chan struct{}, out chan<- string) {
    select {
    case <-delivered:
    case <-time.After(15 * time.Minute):
        out <- "late delivery: refund"
    }
}

kitchen_test.go:

package kitchen

import (
    "testing"
    "testing/synctest"
    "time"
)

func TestLate(t *testing.T) {
    synctest.Run(func() {
        delivered := make(chan struct{})
        out := make(chan string, 1)
        go Late(delivered, out)
        time.Sleep(15*time.Minute - time.Second)
        synctest.Wait()
        if len(out) != 0 {
            t.Fatal("refund too early")
        }
        time.Sleep(time.Second)
        synctest.Wait()
        t.Log(<-out, "at", time.Now().UTC().Format("15:04:05"))
    })
}

func TestOnTime(t *testing.T) {
    synctest.Run(func() {
        delivered := make(chan struct{})
        out := make(chan string, 1)
        go Late(delivered, out)
        time.Sleep(10 * time.Minute)
        close(delivered) // delivered after 10 minutes
        synctest.Wait()
        if len(out) != 0 {
            t.Fatal("refund for an on-time delivery")
        }
    })
}

go.mod:

module kitchen

go 1.24

Commands:

$ go env -w GOEXPERIMENT=synctest
$ go test -v

Real output:

$ go env -w GOEXPERIMENT=synctest
$ go test -v
=== RUN   TestLate
    kitchen_test.go:21: late delivery: refund at 00:15:00
--- PASS: TestLate (0.00s)
=== RUN   TestOnTime
--- PASS: TestOnTime (0.00s)
PASS
ok      kitchen 0.002s
  1. Protect served with a mutex or use atomic.Int64; go test -race -count=20 shows the race before and nothing after.

Common mistakes (and the real messages)

Mistake Real message / result Why Fix
send on an unbuffered channel after the reader left goroutines still alive: 1001 blocked sender a buffer, or select with ctx.Done()
ranging over a channel nobody closes the goroutine never ends blocked receiver the sender closes it
ctx, _ := context.WithTimeout(...) the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak timer kept alive defer cancel()
locks taken in different orders fatal error: all goroutines are asleep - deadlock! circular wait one lock order
retrying in lockstep busy, no progress livelock random backoff
a nested select that prefers one case normal served: 0 starvation one select, or a bounded priority
one goroutine per request, no limit goroutines: 100001, 267 MB unbounded pool / semaphore / SetLimit
time.Sleep in tests PASS (2.00s), flaky real clock testing/synctest
a bug that blocks forever inside a bubble panic: deadlock: all goroutines in bubble are blocked nothing can ever happen fix the missing send
trusting -count=300 passes PASS with a race inside races are rare go test -race

Practice

  1. Fix the quiz's fastest function in two different ways.
  2. Write a test with synctest for a function that retries a request 3 times, waiting 1 minute between tries.
  3. Change the starvation program so VIPs get priority but normal customers get at least one plate in every 4.
  4. Write the livelock loop above, count the attempts over 100 ms, then add random backoff and count again.
Solutions

1. (a) ch := make(chan string, 3), so every sender finishes; (b) pass a context, make each sender select { case ch <- ask(s): case <-ctx.Done(): }, and defer cancel() in fastest after receiving the first answer.

2.

func Retry(f func() error, tries int, wait time.Duration) error {
    var err error
    for i := 0; i < tries; i++ {
        if err = f(); err == nil {
            return nil
        }
        if i < tries-1 {
            time.Sleep(wait)
        }
    }
    return err
}

func TestRetry(t *testing.T) {
    synctest.Run(func() { // Go 1.25+: synctest.Test(t, func(t *testing.T) { ... })
        start := time.Now()
        tries := 0
        Retry(func() error { tries++; return errors.New("down") }, 3, time.Minute)
        if tries != 3 || time.Since(start) != 2*time.Minute {
            t.Fatalf("tries=%d after %v", tries, time.Since(start))
        }
    })
}

(Retry sleeps between tries, not after the last one, so 3 tries take 2 minutes of fake time.)

3. Count VIPs in a row; when the count reaches 3, use a plain select on normal (or a fair select) and reset the count.

4. The count of attempts depends on the machine; with backoff the cooks get through within a few attempts instead of retrying for the whole 100 ms.

Quick quiz

1. fastest starts 3 goroutines that send on an unbuffered channel and returns the first value. How many goroutines leak per call? (From the video.)

A. none · B. 1 · C. 2 · D. 3

Answer

C. Three senders, one receive: the other two block forever. make(chan string, 3) fixes it.

2. When does the runtime print all goroutines are asleep - deadlock!?

Answer

Only when every goroutine in the program is blocked. A partial deadlock in a running server just hangs.

3. In a synctest bubble, when does the fake clock move?

Answer

Only when every goroutine in the bubble is durably blocked; then it jumps to the next timer.

4. A test passes with -count=1000. Is the code race-free?

Answer

No. Run it with -race; passing many times only shows the race didn't change the result in those runs.

Interview questions

  1. What is a goroutine leak, and how do you find one? A goroutine that never finishes, usually blocked on a channel; find it with runtime.NumGoroutine, a goroutine profile (pprof, episode 57), or goleak in tests.
  2. Deadlock vs livelock vs starvation? Deadlock: all blocked waiting on each other. Livelock: all busy reacting to each other, no progress. Starvation: some are ready but never get the resource.
  3. How do you prevent lock-order deadlocks? A single global lock order; don't hold locks while calling unknown code; prefer smaller critical sections.
  4. Why is the cancel function from context.WithTimeout important? It releases the timer and the parent's reference immediately; go vet reports a discarded one.
  5. How do you test code that uses time.Sleep or timers? testing/synctest (Go 1.25+, synctest.Test): a bubble with a fake clock, plus synctest.Wait.
  6. What does -count do, and why isn't it enough? Runs each test n times and skips the cache; it raises the chance of hitting a timing bug but proves nothing. Combine it with -race.
  7. Mutex or channel? Mutex for shared state with short updates; channels for handing off data and ownership, and for signalling. Choose the simplest.
  8. Why limit goroutines if they're cheap? Each has a stack and holds whatever it references; an unbounded burst can exhaust memory and overload what they call.

Glossary

Term Meaning Example
Goroutine leak a goroutine that can never finish supplier B blocked on ch <- 395
Blocked sender a goroutine stuck on a send nobody will receive ch <- v after the reader returned
Blocked receiver a goroutine stuck on a receive nobody will satisfy range jobs with no close(jobs)
Unbuffered channel a channel where a send waits for a receiver make(chan int)
runtime.NumGoroutine how many goroutines exist now 1001
t.Cleanup registers a function to run after the test the leak check
goleak Uber's goroutine leak detector for tests defer goleak.VerifyNone(t)
Cancel function the func that releases a derived context ctx, cancel := context.WithTimeout(...)
Forgotten cancel a cancel function that's never called ctx, _ := ...
go vet lostcancel the vet check for a forgotten cancel "should be called, not discarded"
Deadlock goroutines each waiting for another, forever knife and board
Circular wait A waits for B while B waits for A opposite lock orders
Lock ordering always taking locks in the same order knife, then board
Livelock busy reacting to each other, no progress the corridor dance
TryLock lock only if free now, else return false board.TryLock()
Random backoff wait a random time before retrying rand.IntN(5) ms
Starvation ready but never chosen normal served: 0
Starvation mode sync.Mutex handing the lock to the longest waiter after 1 ms inside sync.Mutex
Unbounded goroutine creation starting goroutines with no limit one per request
Flaky test passes or fails without code changes a test with short sleeps
testing/synctest testing with a bubble and fake time synctest.Test (Go 1.25)
Bubble a test and the goroutines it starts, isolated inside synctest.Run
Fake clock time that jumps instead of passing 10m0s in 0.00s
Durably blocked blocked on something only the bubble can unblock waiting on a bubble channel
synctest.Wait wait until all other bubble goroutines are blocked before checking len(out)
GOEXPERIMENT turns on an experimental Go feature GOEXPERIMENT=synctest
Stress testing running tests many times to shake out rare bugs go test -count=300
-count n run each test n times -count=300
Race detector finds unsynchronised memory access at run time go test -race
Data race two goroutines access the same memory, one writes, no sync total++
CI continuous integration: tests run on every change go test -race -count=20 ./...

Summary

  • Leaks: a goroutine blocked forever as a sender or receiver. Fix with a buffer, ctx.Done(), or close. Catch with a goroutine count in tests or goleak.
  • Forgotten cancel: always defer cancel(); go vet reports it.
  • Deadlock: circular waiting; fix with one lock order. Livelock: busy, no progress; random backoff. Starvation: ready, never chosen; one select, bounded priority.
  • Bound your goroutines.
  • testing/synctest: a bubble with a fake clock; synctest.Wait; deadlocks inside a bubble fail at once. Go 1.25+: synctest.Test(t, ...).
  • -count raises the odds; -race finds races. Use both in CI.
  • Mutex for shared state, channel for handing things over; the simplest wins.

Next episode: memory: the stack and the heap, escape analysis with -gcflags=-m, what makes values escape, the cost of allocations, values vs pointers, and struct alignment and padding with unsafe.Sizeof, Alignof, Offsetof and fieldalignment.