These are the notes for Go episode 54. A function that asks two suppliers for a price leaves one goroutine leak behind on every call (a blocked sender), fixed with a buffer, and caught by a test that counts goroutines, the idea behind goleak. go vet catches a forgotten cancel. Two cooks taking a knife and a board in opposite orders deadlock; one lock order fixes it. Two polite waiters in a corridor show livelock, and a cook who always serves VIPs first shows starvation. A sale that starts a hundred thousand goroutines shows why goroutines need a bound. Then testing: a reminder test that takes real seconds becomes instant and exact with testing/synctest, its bubble and fake clock; a stress run with -count=300 passes despite a data race, and -race finds it at once. Last, a guide to choosing between a mutex and a channel.
All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Our compiler has Go 1.24, where testing/synctest is an experiment turned on with GOEXPERIMENT=synctest and the entry point is synctest.Run; from Go 1.25 it is a normal package and you call synctest.Test(t, ...) (both shown below).
Facts and sources
- Go 1.25 release notes: "The new testing/synctest package provides support for testing concurrent code."
Test"runs a test function in an isolated "bubble"."Wait"waits for all goroutines in the current bubble to block." "This package was first available in Go 1.24 underGOEXPERIMENT=synctest, with a slightly different API. The experiment has now graduated to general availability." - testing/synctest documentation: "Any goroutines started within the bubble are also part of the bubble." "The initial time is midnight UTC 2000-01-01." "Time in a bubble only advances when every goroutine in the bubble is durably blocked." A goroutine is durably blocked when it "can only be unblocked by another goroutine in the same bubble". "If the goroutines in the bubble become deadlocked, the test fails."
- goleak README: "Goroutine leak detector to help avoid Goroutine leaks." Import path
go.uber.org/goleak;defer goleak.VerifyNone(t)in a test, orgoleak.VerifyTestMain(m)inTestMain. - Share Memory By Communicating (Andrew Gerrand, 13 July 2010): "Do not communicate by sharing memory; instead, share memory by communicating."
- Go source,
internal/sync/mutex.go: "If a waiter fails to acquire the mutex for more than 1ms, it switches mutex to the starvation mode. In starvation mode ownership of the mutex is directly handed off from the unlocking goroutine to the waiter at the front of the queue." go help testflag:-count n"Run each test, benchmark, and fuzz seed n times (default 1)."go vet's lostcancel check (Go source): "the cancel function returned by context.%s should be called, not discarded, to avoid a context leak".
Goroutine leaks
A goroutine leak is a goroutine that can never finish. Go never frees a blocked goroutine: it keeps its stack and everything it points to until the program exits. In a server that runs for weeks, leaks pile up until memory runs out.
The obvious way: keep the first answer
package main
import (
"fmt"
"runtime"
"time"
)
// ask two suppliers for today's paneer price, keep the first answer
func firstPrice() int {
ch := make(chan int) // unbuffered: a send waits for a receiver
go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }() // supplier A
go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }() // supplier B
return <-ch // we receive ONE answer and leave
}
func main() {
for i := 1; i <= 1000; i++ { // the app asks 1000 times a day
firstPrice()
}
time.Sleep(10 * time.Millisecond)
fmt.Println("price asked 1000 times")
fmt.Println("goroutines still alive:", runtime.NumGoroutine())
}
Real output:
price asked 1000 times
goroutines still alive: 1001
Why: supplier B answers second. Its send on an unbuffered channel waits for a receiver (episode 47), but firstPrice has already returned, so nobody will ever receive. That goroutine waits forever: one leak per call.
The two kinds of stuck goroutine
| Kind | What it looks like | Example |
|---|---|---|
| blocked sender | ch <- v, but nobody will ever receive |
supplier B above |
| blocked receiver | <-ch or for range ch, but nobody will ever send or close |
a worker whose job channel is never closed |
Three fixes: give the channel room for every send (a buffer); make every send or receive also select on ctx.Done() (episode 52); make sure the sender closes a channel that someone ranges over.
The fix: a buffer for every answer
package main
import (
"fmt"
"runtime"
"time"
)
// ask two suppliers for today's paneer price, keep the first answer
func firstPrice() int {
ch := make(chan int, 2) // room for BOTH answers: no sender waits
go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }() // supplier A
go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }() // supplier B
return <-ch // we receive ONE answer and leave
}
func main() {
for i := 1; i <= 1000; i++ { // the app asks 1000 times a day
firstPrice()
}
time.Sleep(10 * time.Millisecond)
fmt.Println("price asked 1000 times")
fmt.Println("goroutines still alive:", runtime.NumGoroutine())
}
Real output:
price asked 1000 times
goroutines still alive: 1
Catching leaks in tests (the goleak idea)
t.Cleanup(f) registers f to run after the test finishes (episode 40). Counting goroutines before and after turns a leak into a failing test:
shop.go:
package shop
import "time"
// FirstPrice asks two suppliers and keeps the first answer
func FirstPrice() int {
ch := make(chan int)
go func() { time.Sleep(1 * time.Millisecond); ch <- 410 }()
go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }()
return <-ch
}
shop_test.go:
package shop
import (
"runtime"
"testing"
"time"
)
// a tiny leak checker: the goroutine count must be back where it started
func checkLeaks(t *testing.T) {
before := runtime.NumGoroutine()
t.Cleanup(func() {
time.Sleep(20 * time.Millisecond) // let finished goroutines exit
if after := runtime.NumGoroutine(); after > before {
t.Errorf("leaked %d goroutine(s)", after-before)
}
})
}
func TestFirstPrice(t *testing.T) {
checkLeaks(t)
if p := FirstPrice(); p != 410 {
t.Fatalf("got %d, want 410", p)
}
}
Commands:
$ go test
Real output:
$ go test
--- FAIL: TestFirstPrice (0.02s)
shop_test.go:15: leaked 1 goroutine(s)
FAIL
exit status 1
FAIL shop 0.024s
goleak (go.uber.org/goleak, from Uber, "Goroutine leak detector to help avoid Goroutine leaks") does this check for you. Usage from its README:
func TestA(t *testing.T) {
defer goleak.VerifyNone(t)
// test logic here.
}
func TestMain(m *testing.M) {
goleak.VerifyTestMain(m)
}
The forgotten cancel
context.WithTimeout and WithCancel return a cancel function. Throwing it away keeps the context's timer and its link to the parent alive until the timeout fires, here an hour, even though the function finished long ago: a forgotten cancel, a kind of leak. go vet (episode 2) reports it:
main.go:
package main
import (
"context"
"fmt"
"time"
)
func askSupplier(parent context.Context) error {
ctx, _ := context.WithTimeout(parent, time.Hour) // the cancel func is thrown away
select {
case <-time.After(5 * time.Millisecond):
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func main() {
fmt.Println(askSupplier(context.Background()))
}
Commands:
$ go vet
$ go run .
Real output (go vet fails, then go run still works, which is why this bug is easy to miss):
$ go vet
$ go run .
<nil>
# shop
# [shop]
./main.go:10:10: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak
The fix is always: ctx, cancel := context.WithTimeout(parent, d) followed by defer cancel().
Deadlock
| In the kitchen | In Go |
|---|---|
| Asha takes the knife, then needs the board | lock knife, then board |
| Ravi takes the board, then needs the knife | lock board, then knife |
| each waits for the other, forever | deadlock |
| rule: always the knife first | lock ordering |
A deadlock is a set of goroutines each waiting for something another one holds, so none can ever continue. Here it is a circular wait on two mutexes:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var knife, board sync.Mutex
var wg sync.WaitGroup
wg.Add(2)
go func() { // cook Asha: knife first, then board
defer wg.Done()
knife.Lock()
time.Sleep(10 * time.Millisecond)
board.Lock()
fmt.Println("Asha chopped onions")
board.Unlock()
knife.Unlock()
}()
go func() { // cook Ravi: board first, then knife
defer wg.Done()
board.Lock()
time.Sleep(10 * time.Millisecond)
knife.Lock()
fmt.Println("Ravi chopped tomatoes")
knife.Unlock()
board.Unlock()
}()
wg.Wait()
}
Real output (the message lists every goroutine and the line where it is blocked):
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0xc00007e040?)
/usr/local/go1.24.7/src/runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0x0?)
/usr/local/go1.24.7/src/sync/waitgroup.go:118 +0x48
main.main()
main.go:31 +0x125
goroutine 6 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
/usr/local/go1.24.7/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000010118)
/usr/local/go1.24.7/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/usr/local/go1.24.7/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/usr/local/go1.24.7/src/sync/mutex.go:46
main.main.func1()
main.go:17 +0x9b
created by main.main in goroutine 1
main.go:13 +0xb9
goroutine 7 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
/usr/local/go1.24.7/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000010110)
/usr/local/go1.24.7/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
/usr/local/go1.24.7/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
/usr/local/go1.24.7/src/sync/mutex.go:46
main.main.func2()
main.go:26 +0x9b
created by main.main in goroutine 1
main.go:22 +0x119
exit status 2
The runtime reports all goroutines are asleep - deadlock! only when every goroutine is blocked. In a server, where other goroutines (timers, the HTTP listener) keep running, a deadlock simply hangs.
The fix: one lock order
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var knife, board sync.Mutex
var wg sync.WaitGroup
wg.Add(2)
go func() { // cook Asha: knife first, then board
defer wg.Done()
knife.Lock()
time.Sleep(10 * time.Millisecond)
board.Lock()
fmt.Println("Asha chopped onions")
board.Unlock()
knife.Unlock()
}()
go func() { // cook Ravi: the SAME order, knife first, then board
defer wg.Done()
knife.Lock()
time.Sleep(10 * time.Millisecond)
board.Lock()
fmt.Println("Ravi chopped tomatoes")
board.Unlock()
knife.Unlock()
}()
wg.Wait()
}
Real output (who chops first changes between runs):
Ravi chopped tomatoes
Asha chopped onions
Choose one global order for your locks, write it down in a comment, and use it everywhere. Holding a lock while calling unknown code (a callback, another package) is how hidden orderings sneak in.
Livelock
A livelock is when goroutines keep changing state in response to each other, so they are busy, not blocked, but make no progress, like two people in a corridor who both step aside the same way again and again. In code: each cook takes one tool, tries the other with TryLock, fails, puts his tool down and retries, in lockstep with the other cook. The runtime cannot detect it, because nothing is blocked; the program burns CPU. Fixes: random backoff (wait a random time before retrying) or avoid retrying at all with lock ordering.
// livelock-prone: both cooks retry at the same rhythm
for {
knife.Lock()
if board.TryLock() {
chop()
board.Unlock()
knife.Unlock()
return
}
knife.Unlock()
time.Sleep(time.Millisecond) // fix: time.Duration(rand.IntN(5)) * time.Millisecond
}
Starvation
Starvation is when a goroutine or task is ready but never gets the resource, because others always win.
package main
import "fmt"
func main() {
vip := make(chan int, 2000)
normal := make(chan int, 2000)
for i := 0; i < 2000; i++ { // a busy evening: both queues are full
vip <- i
normal <- i
}
servedVIP, servedNormal := 0, 0
for n := 0; n < 1000; n++ { // the cook serves 1000 plates
select {
case <-vip: // ALWAYS check VIPs first...
servedVIP++
default:
select { // ...normal customers only when no VIP waits
case <-vip:
servedVIP++
case <-normal:
servedNormal++
}
}
}
fmt.Println("VIP served:", servedVIP, " normal served:", servedNormal)
}
Real output:
VIP served: 1000 normal served: 0
The fix: one select
When several cases of one select are ready, Go picks one at random (uniformly), so no case can starve (episode 49):
package main
import "fmt"
func main() {
vip := make(chan int, 2000)
normal := make(chan int, 2000)
for i := 0; i < 2000; i++ {
vip <- i
normal <- i
}
servedVIP, servedNormal := 0, 0
for n := 0; n < 1000; n++ {
select { // one select: when both are ready, Go picks one at random
case <-vip:
servedVIP++
case <-normal:
servedNormal++
}
}
fmt.Println("VIP served:", servedVIP, " normal served:", servedNormal)
}
Real output (about half and half; the exact split changes every run):
VIP served: 519 normal served: 481
If you need priority, bound it (for example, at most 3 VIPs in a row). sync.Mutex protects itself against starvation: after a waiter has waited more than 1 ms, the mutex switches to starvation mode and hands the lock directly to the waiter at the front of the queue.
Unbounded goroutine creation
package main
import (
"fmt"
"runtime"
)
func main() {
var before runtime.MemStats
runtime.ReadMemStats(&before)
wait := make(chan struct{})
for i := 0; i < 100_000; i++ { // a sale! one goroutine per request, no limit
go func() { <-wait }() // each one waits for a slow database
}
var after runtime.MemStats
runtime.ReadMemStats(&after)
fmt.Println("goroutines:", runtime.NumGoroutine())
fmt.Println("extra memory (MB):", (after.Sys-before.Sys)/1_000_000)
close(wait)
}
Real output:
goroutines: 100001
extra memory (MB): 267
Goroutines are cheap but not free: each has its own stack (episode 43). Starting one per request with no limit lets a big enough burst exhaust memory. Bound them with a worker pool or a semaphore (episode 52), errgroup.SetLimit (episode 53), or by refusing extra requests (load shedding).
Testing with fake time: testing/synctest
The obvious way: real time
shop.go:
package shop
import "time"
// Remind sends a message when the order has waited too long
func Remind(after time.Duration, out chan<- string) {
time.Sleep(after)
out <- "sorry for the wait, a free lassi is coming!"
}
shop_test.go:
package shop
import (
"testing"
"time"
)
func TestRemind(t *testing.T) {
out := make(chan string, 1)
go Remind(2*time.Second, out) // the real rule is 10 minutes!
select {
case msg := <-out:
t.Log("got:", msg)
case <-time.After(3 * time.Second):
t.Fatal("no reminder")
}
}
Commands:
$ go test -v
Real output:
$ go test -v
=== RUN TestRemind
shop_test.go:13: got: sorry for the wait, a free lassi is coming!
--- PASS: TestRemind (2.00s)
PASS
ok shop 2.002s
Why it fails: two real seconds per test, and it doesn't even test the real 10-minute rule. Shorter sleeps make it flaky: a test that sometimes passes and sometimes fails without any code change, because a busy machine runs late.
The fix: a bubble with a fake clock
| On a film set | In testing/synctest |
|---|---|
| a closed set, its own world | a bubble: the test and every goroutine it starts |
| "Ten minutes later..." | a fake clock that jumps when everyone is blocked |
| wait until every actor is still | synctest.Wait() |
Inside a bubble, time starts at midnight UTC on 1 January 2000 and only moves when every goroutine in the bubble is durably blocked (blocked on something only another bubble goroutine can unblock: a channel made in the bubble, a timer, sync.WaitGroup...). Then the clock jumps straight to the next timer. synctest.Wait() waits until every other goroutine in the bubble is durably blocked, so you can check state at an exact moment.
Go 1.24 (our compiler), with the experiment turned on by the first command:
shop.go:
package shop
import "time"
func Remind(after time.Duration, out chan<- string) {
time.Sleep(after)
out <- "sorry for the wait, a free lassi is coming!"
}
shop_test.go:
package shop
import (
"testing"
"testing/synctest"
"time"
)
func TestRemind(t *testing.T) {
synctest.Run(func() { // a bubble with a fake clock (Go 1.25+: synctest.Test)
start := time.Now()
out := make(chan string, 1)
go Remind(10*time.Minute, out) // the REAL rule: 10 minutes
time.Sleep(10*time.Minute - time.Second)
synctest.Wait() // wait until every goroutine in the bubble is blocked
if len(out) != 0 {
t.Fatal("reminder came too early")
}
msg := <-out
t.Log("got:", msg, "after", time.Since(start))
})
}
Commands:
$ go env -w GOEXPERIMENT=synctest
$ go test -v
Real output:
$ go env -w GOEXPERIMENT=synctest
$ go test -v
=== RUN TestRemind
shop_test.go:20: got: sorry for the wait, a free lassi is coming! after 10m0s
--- PASS: TestRemind (0.00s)
PASS
ok shop 0.003s
Ten minutes of fake time, 0.00s of real time, exact on every run.
Go 1.25 and later (no GOEXPERIMENT):
func TestRemind(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
start := time.Now()
out := make(chan string, 1)
go Remind(10*time.Minute, out)
time.Sleep(10*time.Minute - time.Second)
synctest.Wait()
if len(out) != 0 {
t.Fatal("reminder came too early")
}
msg := <-out
t.Log("got:", msg, "after", time.Since(start))
})
}
A bubble catches deadlocks
shop.go:
package shop
func Remind(out chan<- string) {
// bug: forgot to send anything
}
shop_test.go:
package shop
import (
"testing"
"testing/synctest"
)
func TestRemind(t *testing.T) {
synctest.Run(func() {
out := make(chan string)
go Remind(out)
<-out // waits for a message that never comes
})
}
Commands:
$ go env -w GOEXPERIMENT=synctest
$ go test
Real output (shortened):
$ go env -w GOEXPERIMENT=synctest
$ go test
--- FAIL: TestRemind (0.00s)
panic: deadlock: all goroutines in bubble are blocked [recovered]
panic: deadlock: all goroutines in bubble are blocked
goroutine 6 [running]:
testing.tRunner.func1.2({0x5477e0, 0x5afcc0})
/usr/local/go1.24.7/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
/usr/local/go1.24.7/src/testing/testing.go:1737 +0x35e
panic({0x5477e0?, 0x5afcc0?})
/usr/local/go1.24.7/src/runtime/panic.go:792 +0x132
internal/synctest.Run(0x584628)
/usr/local/go1.24.7/src/runtime/synctest.go:209 +0x245
testing/synctest.Run(...)
/usr/local/go1.24.7/src/testing/synctest/synctest.go:38
shop.TestRemind(0xc0000de000?)
shop_test.go:9 +0x1b
testing.tRunner(0xc0000de000, 0x584540)
/usr/local/go1.24.7/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
/usr/local/go1.24.7/src/testing/testing.go:1851 +0x413
exit status 2
FAIL shop 0.004s
Without a bubble, this test would hang until go test's timeout (10 minutes by default).
Stress testing: -count and -race
Stress testing means running a test many times, often under load, to shake out rare failures. -count=n runs each test n times (and also bypasses the test cache). It raises the odds of seeing a timing bug, but passing many times proves nothing:
shop.go:
package shop
import "sync"
// Tally adds up orders from 2 counters at the same time
func Tally(n int) int {
total := 0
var wg sync.WaitGroup
for c := 0; c < 2; c++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < n; i++ {
total++ // two goroutines, no lock: a data race
}
}()
}
wg.Wait()
return total
}
shop_test.go:
package shop
import "testing"
func TestTally(t *testing.T) {
if got := Tally(1000); got != 2000 {
t.Errorf("Tally(1000) = %d, want 2000", got)
}
}
Commands:
$ go test
$ go test -count=300
Real output:
$ go test
PASS
ok shop 0.007s
$ go test -count=300
PASS
ok shop 0.004s
The race detector (-race, episode 46) watches every memory access while the test runs and reports any unsynchronised access from two goroutines, even if the result happened to be right:
shop.go:
package shop
import "sync"
// Tally adds up orders from 2 counters at the same time
func Tally(n int) int {
total := 0
var wg sync.WaitGroup
for c := 0; c < 2; c++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < n; i++ {
total++ // two goroutines, no lock: a data race
}
}()
}
wg.Wait()
return total
}
shop_test.go:
package shop
import "testing"
func TestTally(t *testing.T) {
if got := Tally(1000); got != 2000 {
t.Errorf("Tally(1000) = %d, want 2000", got)
}
}
Commands:
$ go test
$ go test -race
Real output:
$ go test
PASS
ok shop 0.002s
$ go test -race
==================
WARNING: DATA RACE
Read at 0x00c00011a228 by goroutine 9:
shop.Tally.func1()
shop.go:14 +0xa4
Previous write at 0x00c00011a228 by goroutine 8:
shop.Tally.func1()
shop.go:14 +0xb6
Goroutine 9 (running) created at:
shop.Tally()
shop.go:11 +0x84
shop.TestTally()
shop_test.go:6 +0x2e
testing.tRunner()
/usr/local/go1.24.7/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go1.24.7/src/testing/testing.go:1851 +0x44
Goroutine 8 (finished) created at:
shop.Tally()
shop.go:11 +0x84
shop.TestTally()
shop_test.go:6 +0x2e
testing.tRunner()
/usr/local/go1.24.7/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go1.24.7/src/testing/testing.go:1851 +0x44
==================
--- FAIL: TestTally (0.00s)
testing.go:1490: race detected during execution of test
FAIL
exit status 1
FAIL shop 0.012s
In CI, run both together, for example go test -race -count=20 ./....
Mutex or channel?
| Use a mutex when... | Use a channel when... |
|---|---|
| several goroutines share one piece of state | you hand data, or ownership of it, to another goroutine |
| each access is a short update | work flows through stages: pipelines, worker pools |
| caches, counters, maps, stock levels | results, events, signals (done, cancel) |
The Go slogan: "Do not communicate by sharing memory; instead, share memory by communicating." It's a guide, not a law: pick whichever makes the code simplest, and test it with -race.
Mini project: the bug hunt
The kitchen service below has five planted bugs. Find and fix each one, writing a test that fails before the fix.
// ---- kitchen.go ----
package kitchen
import (
"context"
"sync"
"time"
)
var knife, board sync.Mutex
var served int // bug 5: written by many goroutines, no lock
// bug 1: the slower supplier leaks
func SupplierPrice() int {
ch := make(chan int)
go func() { time.Sleep(time.Millisecond); ch <- 410 }()
go func() { time.Sleep(2 * time.Millisecond); ch <- 395 }()
return <-ch
}
// bug 2: the cancel function is discarded
func Ask(parent context.Context) error {
ctx, _ := context.WithTimeout(parent, time.Hour)
<-time.After(time.Millisecond)
return ctx.Err()
}
// bug 3: opposite lock orders
func ChopOnions() { knife.Lock(); board.Lock(); board.Unlock(); knife.Unlock() }
func ChopTomatoes() { board.Lock(); knife.Lock(); knife.Unlock(); board.Unlock() }
// bug 4 (to test): Late reports on out if a delivery takes longer than 15 minutes
func Late(delivered <-chan struct{}, out chan<- string) {
select {
case <-delivered:
case <-time.After(15 * time.Minute):
out <- "late delivery: refund"
}
}
func Serve() { served++ }
Solution
make(chan int, 2), and a test with the goroutine-count check (orgoleak.VerifyNone).ctx, cancel := context.WithTimeout(parent, time.Hour)+defer cancel();go vetstops complaining.- Make
ChopTomatoestakeknifefirst, thenboard. - A synctest test (Go 1.24 experiment form; use
synctest.Test(t, ...)from Go 1.25):
kitchen.go:
package kitchen
import "time"
func Late(delivered <-chan struct{}, out chan<- string) {
select {
case <-delivered:
case <-time.After(15 * time.Minute):
out <- "late delivery: refund"
}
}
kitchen_test.go:
package kitchen
import (
"testing"
"testing/synctest"
"time"
)
func TestLate(t *testing.T) {
synctest.Run(func() {
delivered := make(chan struct{})
out := make(chan string, 1)
go Late(delivered, out)
time.Sleep(15*time.Minute - time.Second)
synctest.Wait()
if len(out) != 0 {
t.Fatal("refund too early")
}
time.Sleep(time.Second)
synctest.Wait()
t.Log(<-out, "at", time.Now().UTC().Format("15:04:05"))
})
}
func TestOnTime(t *testing.T) {
synctest.Run(func() {
delivered := make(chan struct{})
out := make(chan string, 1)
go Late(delivered, out)
time.Sleep(10 * time.Minute)
close(delivered) // delivered after 10 minutes
synctest.Wait()
if len(out) != 0 {
t.Fatal("refund for an on-time delivery")
}
})
}
go.mod:
module kitchen
go 1.24
Commands:
$ go env -w GOEXPERIMENT=synctest
$ go test -v
Real output:
$ go env -w GOEXPERIMENT=synctest
$ go test -v
=== RUN TestLate
kitchen_test.go:21: late delivery: refund at 00:15:00
--- PASS: TestLate (0.00s)
=== RUN TestOnTime
--- PASS: TestOnTime (0.00s)
PASS
ok kitchen 0.002s
- Protect
servedwith a mutex or useatomic.Int64;go test -race -count=20shows the race before and nothing after.
Common mistakes (and the real messages)
| Mistake | Real message / result | Why | Fix |
|---|---|---|---|
| send on an unbuffered channel after the reader left | goroutines still alive: 1001 |
blocked sender | a buffer, or select with ctx.Done() |
| ranging over a channel nobody closes | the goroutine never ends | blocked receiver | the sender closes it |
ctx, _ := context.WithTimeout(...) |
the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak |
timer kept alive | defer cancel() |
| locks taken in different orders | fatal error: all goroutines are asleep - deadlock! |
circular wait | one lock order |
| retrying in lockstep | busy, no progress | livelock | random backoff |
a nested select that prefers one case |
normal served: 0 |
starvation | one select, or a bounded priority |
| one goroutine per request, no limit | goroutines: 100001, 267 MB |
unbounded | pool / semaphore / SetLimit |
time.Sleep in tests |
PASS (2.00s), flaky |
real clock | testing/synctest |
| a bug that blocks forever inside a bubble | panic: deadlock: all goroutines in bubble are blocked |
nothing can ever happen | fix the missing send |
trusting -count=300 passes |
PASS with a race inside |
races are rare | go test -race |
Practice
- Fix the quiz's
fastestfunction in two different ways. - Write a test with
synctestfor a function that retries a request 3 times, waiting 1 minute between tries. - Change the starvation program so VIPs get priority but normal customers get at least one plate in every 4.
- Write the livelock loop above, count the attempts over 100 ms, then add random backoff and count again.
Solutions
1. (a) ch := make(chan string, 3), so every sender finishes; (b) pass a context, make each sender select { case ch <- ask(s): case <-ctx.Done(): }, and defer cancel() in fastest after receiving the first answer.
2.
func Retry(f func() error, tries int, wait time.Duration) error {
var err error
for i := 0; i < tries; i++ {
if err = f(); err == nil {
return nil
}
if i < tries-1 {
time.Sleep(wait)
}
}
return err
}
func TestRetry(t *testing.T) {
synctest.Run(func() { // Go 1.25+: synctest.Test(t, func(t *testing.T) { ... })
start := time.Now()
tries := 0
Retry(func() error { tries++; return errors.New("down") }, 3, time.Minute)
if tries != 3 || time.Since(start) != 2*time.Minute {
t.Fatalf("tries=%d after %v", tries, time.Since(start))
}
})
}
(Retry sleeps between tries, not after the last one, so 3 tries take 2 minutes of fake time.)
3. Count VIPs in a row; when the count reaches 3, use a plain select on normal (or a fair select) and reset the count.
4. The count of attempts depends on the machine; with backoff the cooks get through within a few attempts instead of retrying for the whole 100 ms.
Quick quiz
1. fastest starts 3 goroutines that send on an unbuffered channel and returns the first value. How many goroutines leak per call? (From the video.)
A. none · B. 1 · C. 2 · D. 3
Answer
C. Three senders, one receive: the other two block forever. make(chan string, 3) fixes it.
2. When does the runtime print all goroutines are asleep - deadlock!?
Answer
Only when every goroutine in the program is blocked. A partial deadlock in a running server just hangs.
3. In a synctest bubble, when does the fake clock move?
Answer
Only when every goroutine in the bubble is durably blocked; then it jumps to the next timer.
4. A test passes with -count=1000. Is the code race-free?
Answer
No. Run it with -race; passing many times only shows the race didn't change the result in those runs.
Interview questions
- What is a goroutine leak, and how do you find one? A goroutine that never finishes, usually blocked on a channel; find it with
runtime.NumGoroutine, a goroutine profile (pprof, episode 57), or goleak in tests. - Deadlock vs livelock vs starvation? Deadlock: all blocked waiting on each other. Livelock: all busy reacting to each other, no progress. Starvation: some are ready but never get the resource.
- How do you prevent lock-order deadlocks? A single global lock order; don't hold locks while calling unknown code; prefer smaller critical sections.
- Why is the cancel function from
context.WithTimeoutimportant? It releases the timer and the parent's reference immediately;go vetreports a discarded one. - How do you test code that uses
time.Sleepor timers?testing/synctest(Go 1.25+,synctest.Test): a bubble with a fake clock, plussynctest.Wait. - What does
-countdo, and why isn't it enough? Runs each test n times and skips the cache; it raises the chance of hitting a timing bug but proves nothing. Combine it with-race. - Mutex or channel? Mutex for shared state with short updates; channels for handing off data and ownership, and for signalling. Choose the simplest.
- Why limit goroutines if they're cheap? Each has a stack and holds whatever it references; an unbounded burst can exhaust memory and overload what they call.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Goroutine leak | a goroutine that can never finish | supplier B blocked on ch <- 395 |
| Blocked sender | a goroutine stuck on a send nobody will receive | ch <- v after the reader returned |
| Blocked receiver | a goroutine stuck on a receive nobody will satisfy | range jobs with no close(jobs) |
| Unbuffered channel | a channel where a send waits for a receiver | make(chan int) |
runtime.NumGoroutine |
how many goroutines exist now | 1001 |
t.Cleanup |
registers a function to run after the test | the leak check |
| goleak | Uber's goroutine leak detector for tests | defer goleak.VerifyNone(t) |
| Cancel function | the func that releases a derived context | ctx, cancel := context.WithTimeout(...) |
| Forgotten cancel | a cancel function that's never called | ctx, _ := ... |
go vet lostcancel |
the vet check for a forgotten cancel | "should be called, not discarded" |
| Deadlock | goroutines each waiting for another, forever | knife and board |
| Circular wait | A waits for B while B waits for A | opposite lock orders |
| Lock ordering | always taking locks in the same order | knife, then board |
| Livelock | busy reacting to each other, no progress | the corridor dance |
TryLock |
lock only if free now, else return false | board.TryLock() |
| Random backoff | wait a random time before retrying | rand.IntN(5) ms |
| Starvation | ready but never chosen | normal served: 0 |
| Starvation mode | sync.Mutex handing the lock to the longest waiter after 1 ms |
inside sync.Mutex |
| Unbounded goroutine creation | starting goroutines with no limit | one per request |
| Flaky test | passes or fails without code changes | a test with short sleeps |
testing/synctest |
testing with a bubble and fake time | synctest.Test (Go 1.25) |
| Bubble | a test and the goroutines it starts, isolated | inside synctest.Run |
| Fake clock | time that jumps instead of passing | 10m0s in 0.00s |
| Durably blocked | blocked on something only the bubble can unblock | waiting on a bubble channel |
synctest.Wait |
wait until all other bubble goroutines are blocked | before checking len(out) |
GOEXPERIMENT |
turns on an experimental Go feature | GOEXPERIMENT=synctest |
| Stress testing | running tests many times to shake out rare bugs | go test -count=300 |
-count n |
run each test n times | -count=300 |
| Race detector | finds unsynchronised memory access at run time | go test -race |
| Data race | two goroutines access the same memory, one writes, no sync | total++ |
| CI | continuous integration: tests run on every change | go test -race -count=20 ./... |
Summary
- Leaks: a goroutine blocked forever as a sender or receiver. Fix with a buffer,
ctx.Done(), orclose. Catch with a goroutine count in tests or goleak. - Forgotten cancel: always
defer cancel();go vetreports it. - Deadlock: circular waiting; fix with one lock order. Livelock: busy, no progress; random backoff. Starvation: ready, never chosen; one
select, bounded priority. - Bound your goroutines.
testing/synctest: a bubble with a fake clock;synctest.Wait; deadlocks inside a bubble fail at once. Go 1.25+:synctest.Test(t, ...).-countraises the odds;-racefinds races. Use both in CI.- Mutex for shared state, channel for handing things over; the simplest wins.
Next episode: memory: the stack and the heap, escape analysis with -gcflags=-m, what makes values escape, the cost of allocations, values vs pointers, and struct alignment and padding with unsafe.Sizeof, Alignof, Offsetof and fieldalignment.