These are the notes for Go episode 58, the last of the performance block. Eight orders take seven times longer than they should, yet the CPU profile is empty: "Total samples = 0". Waiting uses no CPU, so a profiler can't see it. The execution tracer records every event instead; with runtime/trace tasks and regions and go tool trace we find the queue for one stove. Then the flight recorder (Go 1.25), latency vs throughput, and four tuning techniques, each measured: preallocating slices and maps, sync.Pool, inlining and its budget, and bounds-check elimination. The fifth technique: measure first.
All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Timings change between runs; allocation counts and compiler messages don't.
Facts and sources
- Go 1.25 release notes: "The new
runtime/trace.FlightRecorderAPI provides a lightweight way to capture a runtime execution trace by continuously recording the trace into an in-memory ring buffer. When a significant event occurs, a program can callFlightRecorder.WriteToto snapshot the last few seconds of the trace to a file." The time and amount of data "may be configured within theFlightRecorderConfig". go tool trace -h(Go 1.24.7): views in a web browser;-pprof=TYPEfornet,sync,syscall,schedprofiles.- The inlining budget of 80 and the costs are the compiler's own
-gcflags=-m=2messages, shown below.
What profiles can't see
// $ go test -run Rush -v -cpuprofile cpu.prof
// $ go tool pprof -top -nodecount=3 cpu.prof
// ---- kitchen.go ----
package shop
import (
"sync"
"time"
)
var stove sync.Mutex // one stove for the whole kitchen
func Cook(order int) {
time.Sleep(time.Millisecond) // chop
stove.Lock()
time.Sleep(3 * time.Millisecond) // fry
stove.Unlock()
}
// ---- kitchen_test.go ----
package shop
import (
"sync"
"testing"
"time"
)
func TestRush(t *testing.T) {
start := time.Now()
var wg sync.WaitGroup
for order := 1; order <= 8; order++ {
wg.Add(1)
go func() { defer wg.Done(); Cook(order) }()
}
wg.Wait()
t.Log("8 orders in", time.Since(start).Round(time.Millisecond))
}
Real output:
=== RUN TestRush
kitchen_test.go:17: 8 orders in 28ms
--- PASS: TestRush (0.03s)
PASS
ok shop 0.204s
$ go tool pprof -top -nodecount=3 cpu.prof
File: shop.test
Type: cpu
Duration: 201.16ms, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total
Expected about 4 ms (1 chop + 3 fry, all in parallel); got 28 ms. A CPU profile samples only running code; sleeping and waiting for a lock use no CPU.
The execution tracer
| Watching the kitchen | Watching a Go program |
|---|---|
| photos every few seconds show only busy cooks | CPU profile: samples running code |
| a CCTV recording: every moment, every cook, even standing still | execution trace: every runtime event with a timestamp |
| labels on the recording: "order 4: waiting for the stove" | tasks and regions: your own names on the timeline |
- The execution tracer records events: a goroutine is created, starts running, stops, blocks (on a lock, channel, I/O, sleep), is unblocked; GC phases; system calls; processors starting and stopping.
runtime/trace:trace.Start(w)/trace.Stop()record into anyio.Writer;go test -trace trace.outdoes it without code changes.- Task: one logical piece of work (
trace.NewTask(ctx, "order"), ended withtask.End()); region: a named step inside a goroutine (trace.WithRegion(ctx, "fry", f), ortrace.StartRegion).
// $ go run .
// ---- main.go ----
package main
import (
"context"
"fmt"
"os"
"runtime/trace"
"sync"
"time"
)
var stove sync.Mutex // one stove for the whole kitchen
func cook(ctx context.Context, order int) {
ctx, task := trace.NewTask(ctx, "order") // one task per order
defer task.End()
trace.WithRegion(ctx, "chop", func() { time.Sleep(time.Millisecond) })
trace.WithRegion(ctx, "wait for stove", func() { stove.Lock() })
trace.WithRegion(ctx, "fry", func() { time.Sleep(3 * time.Millisecond) })
stove.Unlock()
}
func main() {
f, _ := os.Create("trace.out")
defer f.Close()
trace.Start(f) // record everything the runtime does from here
defer trace.Stop()
start := time.Now()
var wg sync.WaitGroup
for order := 1; order <= 8; order++ {
wg.Add(1)
go func() { defer wg.Done(); cook(context.Background(), order) }()
}
wg.Wait()
fmt.Println("8 orders in", time.Since(start).Round(time.Millisecond))
fmt.Println("cooking time per order: 1 ms chop + 3 ms fry")
}
Real output:
8 orders in 27ms
cooking time per order: 1 ms chop + 3 ms fry
Reading it with go tool trace
go tool trace trace.out starts a local web server. Its pages include View trace by proc (a timeline per processor), Goroutine analysis, User-defined tasks, User-defined regions, and blocking profiles (network, synchronization, syscall, scheduler latency).
- View trace by proc (real screenshot in the video): Proc 0 and Proc 1 show only thin slivers of running code over ~27 ms. The processors are idle: the program is waiting, not computing.
- User-defined regions (real data from the same trace):
| Region | Count | Durations |
|---|---|---|
"chop" |
8 | all in the 1 ms bucket |
"fry" |
8 | all in the 2.5–4 ms bucket |
"wait for stove" |
8 | 1 at 251 ns, 1 at ~2.5 ms, 2 at ~6.3 ms, 1 at ~10 ms, 3 at ~15.8 ms+ |
One stove = one mutex, held while frying: 8 × 3 ms = 24 ms, plus the chop. The fix is more stoves or a shorter critical section (the code that runs while holding the lock), not faster code.
The flight recorder (Go 1.25)
- A full trace of a busy server is big and costs time to write; the rare slow request comes at an unknown moment.
runtime/trace.FlightRecorderkeeps the trace in an in-memory ring buffer (a fixed-size buffer where new data overwrites the oldest), so it always holds the last few seconds.- When something bad happens,
WriteTosaves that window to a file;FlightRecorderConfigsets how much time and data to keep.
The pattern, in words (the channel's compiler is Go 1.24.7, so this isn't run here): create a FlightRecorder with a FlightRecorderConfig, start it when the server starts, and in a request handler, if the request took longer than your limit, open a file and call the recorder's WriteTo with it. Then open that file with go tool trace. See the runtime/trace package documentation for Go 1.25+ for the exact constructor and config fields.
Latency vs throughput
| A chai stall | A server |
|---|---|
| how long one customer waits for the first sip | latency: time for one request |
| cups sold per hour | throughput: requests per second |
| a huge pot: many cups, but a long first wait | the trade-off: batching raises throughput, hurts latency |
Decide which number your users feel (often a latency percentile, like "99% of requests under 200 ms"), and measure that.
Technique 1: fewer allocations — preallocate
// $ go test -bench . -benchmem
// ---- orders.go ----
package shop
func Grow(n int) []int {
var ids []int // starts empty: append must grow it again and again
for i := 0; i < n; i++ {
ids = append(ids, i)
}
return ids
}
func Prealloc(n int) []int {
ids := make([]int, 0, n) // room for n from the start
for i := 0; i < n; i++ {
ids = append(ids, i)
}
return ids
}
func MapGrow(n int) map[int]bool {
m := map[int]bool{}
for i := 0; i < n; i++ {
m[i] = true
}
return m
}
func MapPrealloc(n int) map[int]bool {
m := make(map[int]bool, n) // size hint
for i := 0; i < n; i++ {
m[i] = true
}
return m
}
// ---- orders_test.go ----
package shop
import "testing"
func BenchmarkGrow(b *testing.B) { for i := 0; i < b.N; i++ { Grow(10_000) } }
func BenchmarkPrealloc(b *testing.B) { for i := 0; i < b.N; i++ { Prealloc(10_000) } }
func BenchmarkMapGrow(b *testing.B) { for i := 0; i < b.N; i++ { MapGrow(10_000) } }
func BenchmarkMapPrealloc(b *testing.B) { for i := 0; i < b.N; i++ { MapPrealloc(10_000) } }
Real output:
BenchmarkGrow-2 13965 185700 ns/op 357624 B/op 19 allocs/op
BenchmarkPrealloc-2 37429 40037 ns/op 81920 B/op 1 allocs/op
BenchmarkMapGrow-2 1676 724763 ns/op 591480 B/op 79 allocs/op
BenchmarkMapPrealloc-2 4108 246690 ns/op 295552 B/op 33 allocs/op
make([]T, 0, n): length 0, capacity n: no regrowing.make(map[K]V, n): a size hint.- Text:
strings.Builderandbytes.Buffer(a growable byte buffer that also implementsio.Reader/io.Writer) both haveGrow(n).
Technique 2: reuse — sync.Pool
// $ go test -bench . -benchmem
// ---- receipt.go ----
package shop
import (
"bytes"
"strconv"
"sync"
)
func Render(items []int) int { // a new 128 KB buffer for every receipt
buf := bytes.NewBuffer(make([]byte, 0, 128<<10))
for _, p := range items {
buf.WriteString("item ₹" + strconv.Itoa(p) + "\n")
}
return buf.Len()
}
var bufPool = sync.Pool{ // a shelf of used buffers
New: func() any { return bytes.NewBuffer(make([]byte, 0, 128<<10)) },
}
func RenderPooled(items []int) int {
buf := bufPool.Get().(*bytes.Buffer) // take one from the shelf
buf.Reset() // wipe the old receipt!
for _, p := range items {
buf.WriteString("item ₹" + strconv.Itoa(p) + "\n")
}
n := buf.Len()
bufPool.Put(buf) // put it back for the next receipt
return n
}
// ---- receipt_test.go ----
package shop
import "testing"
var items = []int{120, 80, 45, 300, 60}
func BenchmarkRender(b *testing.B) { for i := 0; i < b.N; i++ { Render(items) } }
func BenchmarkRenderPooled(b *testing.B) { for i := 0; i < b.N; i++ { RenderPooled(items) } }
Real output (two runs):
BenchmarkRender-2 13885 107620 ns/op 131078 B/op 3 allocs/op
BenchmarkRenderPooled-2 5087568 262.5 ns/op 6 B/op 2 allocs/op
BenchmarkRender-2 20929 59808 ns/op 131078 B/op 3 allocs/op
BenchmarkRenderPooled-2 6078786 448.4 ns/op 6 B/op 2 allocs/op
Getreturns a pooled object, or callsNewif the pool is empty;Putreturns it.- Rules: always reset what you
Get; never use an object afterPut; the GC may empty the pool at any time, so it's a cache, never storage. Use it for big, short-lived, frequently made objects (buffers, encoders). - (A 64 KB buffer made with a constant size stayed on the stack in a first try, so there was nothing to pool: measure before pooling.)
Technique 3: inlining and the budget
- Inlining: the compiler pastes a small function's body into its caller: no call overhead, and the pasted code can be optimised with the caller.
- Every function gets a cost (roughly its number of operations); functions over the inlining budget (80) are not inlined. PGO raises the budget for hot functions.
// $ go build -gcflags=-m=2 .
// ---- bill.go ----
package shop
func GST(price int) int { return price * 18 / 100 } // small
func Receipt(items []int) (total, bad int) { // bigger
for _, p := range items {
switch {
case p < 0:
bad++
case p == 0:
continue // a free item
case p > 10_000:
total += p + GST(p) - p/10 // discount on big items
default:
total += p + GST(p)
}
if total > 1_000_000 {
total = 1_000_000
break
}
}
for i := 0; i < bad; i++ {
total -= 5 // fine per bad item
}
if bad > 3 {
total = 0 // too many bad items: reject the bill
}
return total, bad
}
Real output:
# shop
./bill.go:3:6: can inline GST with cost 6 as: func(int) int { return price * 18 / 100 }
./bill.go:5:6: cannot inline Receipt: function too complex: cost 87 exceeds budget 80
./bill.go:13:29: inlining call to GST
./bill.go:15:29: inlining call to GST
./bill.go:5:14: items does not escape
Without the case p == 0 and the final if, the same function had can inline Receipt with cost 75 (real run). Small functions are free: write them for clarity.
Technique 4: bounds-check elimination
- Every index
d[i]is bounds-checked: ifiis outside0..len(d)-1, Go panics. This makes Go memory-safe, but each check is a compare and a branch. - Bounds-check elimination (BCE): the compiler removes checks it can prove can't fail.
-gcflags=-d=ssa/check_bce/debug=1prints each check that remains (Found IsInBounds).
// $ go build -gcflags=-d=ssa/check_bce/debug=1 .
// ---- code.go ----
package shop
// SumCode adds the 4 digits of an order code, checked one by one.
func SumCode(d []int) int {
return d[0] + d[1] + d[2] + d[3] // 4 bounds checks
}
// SumCodeFast tells the compiler the length once.
func SumCodeFast(d []int) int {
_ = d[3] // one check: if this works, 0..3 are all fine
return d[0] + d[1] + d[2] + d[3]
}
// Total ranges over the slice: the compiler knows i is in range.
func Total(prices []int) int {
t := 0
for i := range prices {
t += prices[i]
}
return t
}
Real output:
# shop
./code.go:5:13: Found IsInBounds
./code.go:5:20: Found IsInBounds
./code.go:5:27: Found IsInBounds
./code.go:5:34: Found IsInBounds
./code.go:10:10: Found IsInBounds
Four checks in SumCode, one in SumCodeFast (the _ = d[3] hint), none in Total. Only worth it in the hottest loops, after a profile points there.
Technique 5: measure first
- Decide the goal: latency or throughput, and a number.
- Benchmark it (
go test -bench -benchmem). - Find the cause: CPU profile (busy), heap profile (allocations), trace (waiting).
- Fix the biggest cause only; measure again. Compare runs with
benchstat(episode 42).
Tuning code nobody measured is premature optimisation: harder to read, usually no gain. Keep code clear; tune the hot path (the code that runs most).
Mini project: the fast kitchen
- Give the kitchen N stoves with a buffered channel as a semaphore (episode 52) and compare 1, 2, 4 and 8 stoves.
- Trace it and look at the "wait for stove" regions.
- Pool each order's ticket buffer; benchmark with
-benchmem. - Find a function just over budget 80 and split it.
Solution (step 1)
package main
import (
"fmt"
"sync"
"time"
)
func rush(stoves int) time.Duration {
stove := make(chan struct{}, stoves) // a semaphore: N free stoves
start := time.Now()
var wg sync.WaitGroup
for order := 1; order <= 8; order++ {
wg.Add(1)
go func() {
defer wg.Done()
time.Sleep(time.Millisecond) // chop
stove <- struct{}{} // take a stove
time.Sleep(3 * time.Millisecond) // fry
<-stove // give it back
}()
}
wg.Wait()
return time.Since(start).Round(time.Millisecond)
}
func main() {
for _, n := range []int{1, 2, 4, 8} {
fmt.Printf("%d stove(s): 8 orders in %v\n", n, rush(n))
}
}
Real output:
1 stove(s): 8 orders in 27ms
2 stove(s): 8 orders in 14ms
4 stove(s): 8 orders in 8ms
8 stove(s): 8 orders in 4ms
Steps 2–4 use exactly the patterns shown above: wrap the stove wait in trace.WithRegion, use the bufPool pattern, and check -gcflags=-m=2 before and after splitting.
Common mistakes (and the real messages)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| CPU profile for a waiting problem | Total samples = 0 |
waiting uses no CPU | go tool trace, block profile |
| slow work while holding a lock | wait for stove up to ~16 ms+ |
everyone queues | shorter critical section, more capacity |
| pooled object not reset | the previous receipt's lines appear | Get returns used objects |
buf.Reset() |
sync.Pool as storage |
objects disappear | the GC may empty it | use it only as a cache |
| huge function on the hot path | cost 87 exceeds budget 80 |
not inlined | split out small helpers |
| tuning without a benchmark | no measurable gain | not the hot path | measure first |
Practice
- A request takes 2 s; CPU usage is 3%. Which tool first?
- What's the difference between a task and a region in
runtime/trace? - Why does
make([]int, 0, n)give 1 allocation wherevar ids []intgives 19? - When would you not use
sync.Pool?
Solutions
1. An execution trace (or block profile): the time is spent waiting.
2. A task is a logical operation that can span goroutines (one order); a region is a named time interval within one goroutine (one step).
3. With capacity n, append never has to grow; without it, append repeatedly allocates a bigger array and copies.
4. For small, cheap objects (the pool's overhead is larger than the allocation), for anything that must be kept, and when no benchmark shows allocation cost.
Quick quiz
1. 200 requests/s, each takes 900 ms, CPU 5%, pprof has almost no samples. First tool? (From the video.)
A. a CPU profile again · B. an execution trace · C. raise GOGC · D. inline everything
Answer
B. Low CPU and slow requests means waiting; a trace shows what each goroutine waits for.
2. What is Go's inlining budget?
Answer
80 (cost 87 exceeds budget 80).
3. What does _ = d[3] do before using d[0]..d[3]?
Answer
One bounds check up front, which lets the compiler remove the checks for indexes 0–3.
Interview questions
- Profile vs trace? A profile samples where CPU time (or memory) goes; a trace records every event over time, including blocking and scheduling.
- How do you take a trace?
go test -trace,runtime/trace.Start/Stop, or/debug/pprof/tracevianet/http/pprof; view withgo tool trace. - What are tasks and regions? User annotations in traces: a task = a logical operation, a region = a named interval in a goroutine.
- What is the flight recorder? Go 1.25's
runtime/trace.FlightRecorder: keeps the last seconds of trace in a ring buffer and writes them on demand. - Latency vs throughput? Time per request vs requests per time; batching trades one for the other.
- How do you reduce allocations? Preallocate (
makewith capacity, size hints), reuse buffers (sync.Pool), avoid+=on strings, keep values on the stack. - When does Go inline a function? When its cost is within the budget (80), or higher for hot functions with PGO; see
-gcflags=-m. - What is bounds-check elimination? The compiler removing index checks it can prove are safe; see
-d=ssa/check_bce/debug=1.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Execution tracer | records runtime events with timestamps | runtime/trace |
| Execution trace | the recorded events | trace.out |
| Event | one thing the runtime did | goroutine blocks on a lock |
runtime/trace |
the package that writes traces | trace.Start(f) |
go test -trace |
records a trace of a test run | -trace trace.out |
go tool trace |
web viewer for traces | View trace by proc |
| Task | a logical operation in a trace | trace.NewTask(ctx, "order") |
| Region | a named interval inside a goroutine | trace.WithRegion(ctx, "fry", f) |
| Proc (P) | a logical processor running goroutines | Proc 0, Proc 1 |
| Critical section | code that runs while holding a lock | frying with the stove locked |
| Flight recorder | keeps the last seconds of trace in memory (Go 1.25) | FlightRecorder.WriteTo |
| Ring buffer | fixed-size buffer where new data overwrites the oldest | the flight recorder's memory |
| Latency | time for one request | 900 ms |
| Throughput | work per unit of time | 200 requests/s |
| Percentile | the value under which a share of measurements fall | p99 latency |
| Preallocate | reserve room up front | make([]int, 0, n) |
| Capacity | how many elements fit before a slice must grow | cap(ids) |
| Size hint | expected number of map entries | make(map[int]bool, n) |
bytes.Buffer |
a growable byte buffer | buf.WriteString |
sync.Pool |
a cache of reusable objects | bufPool.Get() |
| Inlining | pasting a function's body into its caller | inlining call to GST |
| Inlining cost | the compiler's size estimate of a function | cost 6 |
| Inlining budget | the maximum cost that is inlined: 80 | exceeds budget 80 |
-gcflags=-m=2 |
explain inlining and escape decisions in detail | go build -gcflags=-m=2 . |
| Bounds check | the run-time check that an index is in range | d[3] |
| Bounds-check elimination (BCE) | removing checks proven unnecessary | _ = d[3] |
check_bce |
debug flag listing remaining checks | Found IsInBounds |
| Hot path | the code that runs most often | the inner loop |
| Premature optimisation | tuning before measuring | rewriting cold code |
benchstat |
compares benchmark runs statistically | episode 42 |
| Semaphore | a counter limiting how many may proceed | make(chan struct{}, 2) |
Summary
- A CPU profile can't see waiting; the execution tracer records every event. Use
runtime/trace(tasks, regions) orgo test -trace, and read it withgo tool trace. - The flight recorder (Go 1.25) keeps the last seconds for rare slow moments.
- Decide latency or throughput first.
- Tune, measured: preallocate (
makewith capacity, size hints,Grow),sync.Poolfor big short-lived buffers, small functions get inlined (budget 80), BCE for hot loops. - Measure first, fix the biggest cause, measure again.
Next episode: building command-line tools: os.Args, the flag package, subcommands, stdin/stdout/stderr, exit codes, os/exec, signals.