SyntaxPilot
Courses/Go Programming/Lesson 57
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 57 of 89

Profiling with pprof

profilingruntime/pprofnet/http/pprofCPU profileheap profile (inuse vs alloc)goroutine profileblock profilemutex profilego test -cpuprofile and -memprofilego tool pprof (top, list, web)flame graphsprofile-guided optimisation (PGO, default.pgo)

These are the notes for Go episode 57. Raju's closing report is slow. The obvious guess (the sort) is wrong, and timing every line by hand doesn't scale. So we invent a sampling profiler, then use Go's real one, pprof: a CPU profile read with go tool pprof -top and -list, flat vs cum, the flame graph, then a fix that makes the report about 50 times faster. Then the other profiles: heap (alloc_space vs inuse_space), goroutine, block and mutex, profiling a live server with net/http/pprof, and profile-guided optimisation with default.pgo.

All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Timings and sample counts change between runs; the shape stays the same.

Facts and sources

  • Go 1.24.7 source, runtime/pprof/pprof.go: StartCPUProfile samples at const hz = 100 (100 samples per second).
  • go help build (Go 1.24.7): -pgo file — "When the special name "auto" is specified, for each main package in the build, the go command selects a file named "default.pgo" in the package's directory if that file exists ... The default is "auto"."
  • Profile-guided optimization (Go documentation): PGO supported "Starting in Go 1.20"; "Before Go 1.21, the default is -pgo=off"; "As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%"; the compiler "may decide to more aggressively inline functions which the profile indicates are called frequently"; collecting profiles "directly from the production environment is recommended".
  • Go 1.26 release notes: "The pprof tool web UI, enabled with the -http flag, now defaults to the flame graph view"; an experimental goroutineleak profile (GOEXPERIMENT=goroutineleakprofile).

Why guessing fails

package main

import (
    "fmt"
    "regexp"
    "sort"
    "strconv"
    "time"
)

func Report(prices []int) string {
    t := time.Now()
    sort.Ints(prices) // slow part?
    fmt.Println("sort took:  ", time.Since(t))
    out := ""
    for _, p := range prices {
        line := "item ₹" + strconv.Itoa(p) + "\n"
        if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
            out += line
        }
    }
    return out
}

func main() {
    prices := make([]int, 2000)
    for i := range prices {
        prices[i] = (i * 7919) % 1000
    }
    t := time.Now()
    Report(prices)
    fmt.Println("Report took:", time.Since(t))
}

Real output:

sort took:   103.966µs
Report took: 27.348371ms

The sort is about 0.4% of the time. Timing every line by hand doesn't scale, and the timers change the timing.

The profiler idea: sampling

Finding the slow step in a kitchen A CPU profiler
take a photo every few seconds of what each cook is doing sampling: interrupt the program regularly and record where it is (Go: 100 times a second)
count the photos per task: 60 of 100 show chopping a profile: samples counted per function
note who asked for the onions the call stack: which function called which

A profile is a statistical picture, not a stopwatch, so it is cheap enough to run on real programs.

CPU profile: go test -cpuprofile and go tool pprof -top

The program under test (package shop):

// ---- bill.go ----
package shop

import (
    "regexp"
    "sort"
    "strconv"
)

func Report(prices []int) string {
    sort.Ints(prices) // slow part?
    out := ""
    for _, p := range prices {
        line := "item ₹" + strconv.Itoa(p) + "\n"
        if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
            out += line
        }
    }
    return out
}

// ---- bill_test.go ----
package shop

import "testing"

func BenchmarkReport(b *testing.B) {
    prices := make([]int, 2000)
    for i := range prices {
        prices[i] = (i * 7919) % 1000
    }
    for i := 0; i < b.N; i++ {
        Report(prices)
    }
}
$ go test -bench . -cpuprofile cpu.prof
$ go tool pprof -top -cum -nodecount=9 cpu.prof

Real output of the second command (one run):

File: shop.test
Type: cpu
Duration: 2.13s, Total samples = 1.84s (86.24%)
Showing nodes accounting for 0.01s, 0.54% of 1.84s total
Showing top 9 nodes out of 170
      flat  flat%   sum%        cum   cum%
         0     0%     0%      1.08s 58.70%  shop.BenchmarkReport
         0     0%     0%      1.08s 58.70%  shop.Report
         0     0%     0%      1.08s 58.70%  testing.(*B).launch
         0     0%     0%      1.08s 58.70%  testing.(*B).runN
         0     0%     0%      0.69s 37.50%  regexp.Compile (inline)
         0     0%     0%      0.69s 37.50%  regexp.MustCompile
         0     0%     0%      0.69s 37.50%  regexp.compile
         0     0%     0%      0.59s 32.07%  runtime.bgsweep
     0.01s  0.54%  0.54%      0.41s 22.28%  runtime.mallocgc
Column Meaning
flat time spent in the function itself
cum (cumulative) time in the function plus everything it called
flat% / cum% the same as a share of all samples
sum% running total of flat% down the table

-top prints the table; -cum sorts by cum; -nodecount=9 keeps nine rows. runtime.mallocgc hands out heap memory and runtime.bgsweep is the GC's background sweep (episode 56): lots of garbage.

CPU profile: -list

$ go test -bench . -cpuprofile cpu.prof
$ go tool pprof -list=shop.Report cpu.prof

Real output (the -list part):

ROUTINE ======================== shop.Report in bill.go
         0      980ms (flat, cum) 52.97% of Total
         .          .      9:func Report(prices []int) string {
         .          .     10:    sort.Ints(prices) // slow part?
         .          .     11:    out := ""
         .          .     12:    for _, p := range prices {
         .       20ms     13:        line := "item ₹" + strconv.Itoa(p) + "\n"
         .      660ms     14:        if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
         .      300ms     15:            out += line
         .          .     16:        }
         .          .     17:    }
         .          .     18:    return out
         .          .     19:}

Flame graphs

  • Each box is a function; a box sits under (pprof draws root on top) the function that called it.
  • Width = share of samples. Left-to-right order means nothing.
  • Look for the widest boxes deep in the chain: that's where time burns. Thin boxes aren't worth optimising.
  • go tool pprof -http=127.0.0.1:8090 cpu.prof opens the web UI; since Go 1.26 it opens on the flame graph.

The video shows a real screenshot of the flame graph view for this benchmark: shop.Report → regexp.MustCompile (wide) and runtime.concatstrings (the +=), plus a wide runtime.bgsweep (the GC).

The fix, measured

// $ go test -bench . -benchmem
// ---- bill.go ----
package shop

import (
    "regexp"
    "sort"
    "strconv"
    "strings"
)

func Report(prices []int) string { // the original
    sort.Ints(prices)
    out := ""
    for _, p := range prices {
        line := "item ₹" + strconv.Itoa(p) + "\n"
        if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
            out += line
        }
    }
    return out
}

var lineFormat = regexp.MustCompile(`^item ₹\d+\n$`) // fix 1: compile once

func ReportFast(prices []int) string {
    sort.Ints(prices)
    var sb strings.Builder // fix 2: one growing buffer, no copies
    for _, p := range prices {
        line := "item ₹" + strconv.Itoa(p) + "\n"
        if lineFormat.MatchString(line) {
            sb.WriteString(line)
        }
    }
    return sb.String()
}

// ---- bill_test.go ----
package shop

import "testing"

func prices() []int {
    p := make([]int, 2000)
    for i := range p {
        p[i] = (i * 7919) % 1000
    }
    return p
}

func BenchmarkReport(b *testing.B) {
    p := prices()
    for i := 0; i < b.N; i++ {
        Report(p)
    }
}

func BenchmarkReportFast(b *testing.B) {
    p := prices()
    for i := 0; i < b.N; i++ {
        ReportFast(p)
    }
}

Real output:

BenchmarkReport-2                 67      23896532 ns/op    33848040 B/op     119826 allocs/op
BenchmarkReportFast-2           2738        442877 ns/op      145406 B/op       3818 allocs/op
PASS

About 54× faster. The loop: measure → read the profile → fix the widest box → measure again.

Heap profiles: alloc_space vs inuse_space

  • -memprofile mem.prof writes a heap profile: where memory was allocated (sampled, about one sample per 512 KB allocated by default).
  • -sample_index=alloc_space: every byte allocated since the start, garbage included → finds GC pressure.
  • -sample_index=inuse_space (the default): bytes still in use when the profile was written → finds leaks.
  • Also alloc_objects / inuse_objects count objects instead of bytes.
$ go test -bench . -memprofile mem.prof
$ go tool pprof -sample_index=alloc_space -top -nodecount=5 mem.prof
$ go tool pprof -sample_index=inuse_space -top -nodecount=3 mem.prof

Real output (for the original Report benchmark):

Type: alloc_space
Showing nodes accounting for 1893.41MB, 93.67% of 2021.42MB total
Dropped 16 nodes (cum <= 10.11MB)
Showing top 5 nodes out of 26
      flat  flat%   sum%        cum   cum%
 1500.23MB 74.22% 74.22%  2021.42MB   100%  shop.Report
  162.57MB  8.04% 82.26%   162.57MB  8.04%  regexp/syntax.(*compiler).inst (inline)
  104.59MB  5.17% 87.43%   104.59MB  5.17%  regexp.onePassCopy
   93.51MB  4.63% 92.06%    93.51MB  4.63%  regexp/syntax.(*parser).newRegexp (inline)
   32.51MB  1.61% 93.67%    85.01MB  4.21%  regexp.makeOnePass

Type: inuse_space
Showing nodes accounting for 0, 0% of 0 total
      flat  flat%   sum%        cum   cum%

About 2 GB allocated, nothing still in use at the end.

Goroutine profiles

// $ go run .
// ---- main.go ----
package main

import (
    "fmt"
    "os"
    "runtime/pprof"
    "time"
)

func askKitchen(order int, reply chan string) {
    reply <- fmt.Sprint("order ", order, " ready") // nobody reads this
}

func main() {
    for i := 0; i < 100; i++ {
        go askKitchen(i, make(chan string)) // forgot to receive
    }
    time.Sleep(100 * time.Millisecond)
    pprof.Lookup("goroutine").WriteTo(os.Stdout, 1) // 1 = short text
}

Real output (shortened; the program's own stack lines are kept):

goroutine profile: total 101
100 @ 0x46bf0e 0x40b3e5 0x40b037 0x4cf72f 0x472a01
#   0x4cf72e    main.askKitchen+0xae    main.go:11

1 @ 0x432bb1 0x46b33d 0x4bbd51 0x4bbb85 0x4b89ab 0x4cf65d 0x43cc0b 0x472a01
#   0x4bbd50    runtime/pprof.writeRuntimeProfile+0xb0  .../src/runtime/pprof/pprof.go:796
...
#   0x4cf65c    main.main+0xbc              main.go:19
  • pprof.Lookup(name) returns a built-in profile: goroutine, heap, allocs, threadcreate, block, mutex.
  • Identical stacks are grouped and counted, so a goroutine leak (episode 54) shows as one big number.

Block and mutex profiles

// $ go test -run Rush -blockprofile block.prof -mutexprofile mutex.prof
// $ go tool pprof -top -nodecount=4 block.prof
// $ go tool pprof -top -nodecount=4 mutex.prof
// ---- kitchen.go ----
package shop

import (
    "sync"
    "time"
)

type Stove struct{ mu sync.Mutex } // one stove, many cooks

func (s *Stove) Cook() {
    s.mu.Lock()
    time.Sleep(2 * time.Millisecond) // cooking while holding the lock
    s.mu.Unlock()
}

// ---- kitchen_test.go ----
package shop

import (
    "sync"
    "testing"
)

func TestRush(t *testing.T) {
    var s Stove
    var wg sync.WaitGroup
    for cook := 0; cook < 8; cook++ {
        wg.Add(1)
        go func() { defer wg.Done(); for i := 0; i < 10; i++ { s.Cook() } }()
    }
    wg.Wait()
}

Real output:

Type: delay
Showing nodes accounting for 1521.81ms, 100% of 1521.81ms total
Showing top 4 nodes out of 14
      flat  flat%   sum%        cum   cum%
 1166.05ms 76.62% 76.62%  1166.05ms 76.62%  sync.(*Mutex).Lock
  177.95ms 11.69% 88.32%   177.95ms 11.69%  runtime.chanrecv1
  177.81ms 11.68%   100%   177.81ms 11.68%  sync.(*WaitGroup).Wait
         0     0%   100%   177.95ms 11.69%  main.main

Type: delay
Showing nodes accounting for 1.17s, 100% of 1.17s total
      flat  flat%   sum%        cum   cum%
     1.17s   100%   100%      1.17s   100%  sync.(*Mutex).Unlock (inline)
         0     0%   100%      1.17s   100%  shop.(*Stove).Cook
         0     0%   100%      1.17s   100%  shop.TestRush.func1
  • Block profile: where goroutines waited (locks, channels, WaitGroup, select).
  • Mutex profile: which lock holders made others wait (charged where the lock is released).
  • Outside tests: runtime.SetBlockProfileRate(rate) and runtime.SetMutexProfileFraction(rate); rate 1 records every event.

Profiling a live server: net/http/pprof

// $ go run .
// ---- main.go ----
package main

import (
    "bufio"
    "fmt"
    "net/http"
    _ "net/http/pprof" // blank import: adds the /debug/pprof/ pages
    "time"
)

func main() {
    go http.ListenAndServe("localhost:6060", nil) // your real server
    time.Sleep(200 * time.Millisecond)

    // what you'd open in a browser, or fetch with go tool pprof:
    for _, page := range []string{"goroutine?debug=1", "heap?debug=1"} {
        resp, err := http.Get("http://localhost:6060/debug/pprof/" + page)
        if err != nil {
            fmt.Println(err)
            return
        }
        first, _ := bufio.NewReader(resp.Body).ReadString('\n')
        resp.Body.Close()
        fmt.Printf("%-18s -> %s", page, first)
    }
}

Real output:

goroutine?debug=1  -> goroutine profile: total 6
heap?debug=1       -> heap profile: 2: 65632 [2: 65633] @ heap/1048576
  • A blank import (_ "net/http/pprof") runs the package's init, which registers handlers on http.DefaultServeMux.
  • From your machine: go tool pprof http://host:6060/debug/pprof/profile?seconds=30 (30-second CPU profile), .../heap, .../goroutine.
  • Never expose these pages publicly: serve them on a private port or behind authentication.

Profile-guided optimisation (PGO)

// $ go test -bench . -cpuprofile default.pgo
// $ go build -o app .
// $ go version -m app
// ---- main.go ----
package main

import (
    "fmt"
    "strconv"
    "strings"
)

func Bill(n int) string {
    var sb strings.Builder
    for i := 1; i <= n; i++ {
        sb.WriteString("item ₹" + strconv.Itoa(i*10) + "\n")
    }
    return sb.String()
}

func main() { fmt.Print(Bill(2)) }

// ---- main_test.go ----
package main

import "testing"

func BenchmarkBill(b *testing.B) {
    for i := 0; i < b.N; i++ {
        Bill(1000)
    }
}

Real output (shortened):

BenchmarkBill-2        14086         78190 ns/op
PASS
$ go build -o app .
$ go version -m app
app: go1.24.7
    path    shop
    build   -buildmode=exe
    build   -compiler=gc
    build   -pgo=default.pgo
    ...
  • PGO: the compiler uses a CPU profile to optimise hot code, mainly by inlining hot functions more aggressively. Since Go 1.20; -pgo=auto (use default.pgo in the main package directory) is the default since Go 1.21; around 2–14% faster as of Go 1.22.
  • Use a profile from production, commit it as default.pgo, refresh it now and then.

Which profile answers which question

Question Profile How
Slow, CPU busy? CPU go test -cpuprofile, pprof.StartCPUProfile, /debug/pprof/profile
GC busy, too much garbage? heap, alloc_space -memprofile, /debug/pprof/allocs
Memory keeps growing? heap, inuse_space -memprofile, /debug/pprof/heap
Stuck, too many goroutines? goroutine pprof.Lookup("goroutine"), /debug/pprof/goroutine
Waiting on locks/channels? block -blockprofile, SetBlockProfileRate
Lock held too long? mutex -mutexprofile, SetMutexProfileFraction

Mini project: the slow menu

Write Menu(dishes []string) string using +=, fmt.Sprintf and regexp.MustCompile in the loop; benchmark with 5000 dishes; profile; fix one thing at a time.

Solution
// $ go test -bench . -benchmem
// ---- menu.go ----
package shop

import (
    "fmt"
    "regexp"
    "strings"
)

func Menu(dishes []string) string {
    out := ""
    for i, d := range dishes {
        if regexp.MustCompile(`^[a-z ]+$`).MatchString(d) {
            out += fmt.Sprintf("%d. %s\n", i+1, d)
        }
    }
    return out
}

var dishName = regexp.MustCompile(`^[a-z ]+$`)

func MenuFast(dishes []string) string {
    var sb strings.Builder
    sb.Grow(len(dishes) * 24)
    for i, d := range dishes {
        if dishName.MatchString(d) {
            fmt.Fprintf(&sb, "%d. %s\n", i+1, d)
        }
    }
    return sb.String()
}

// ---- menu_test.go ----
package shop

import "testing"

func dishes() []string {
    d := make([]string, 5000)
    for i := range d {
        d[i] = "masala dosa"
    }
    return d
}

func BenchmarkMenu(b *testing.B) {
    d := dishes()
    for i := 0; i < b.N; i++ {
        Menu(d)
    }
}

func BenchmarkMenuFast(b *testing.B) {
    d := dishes()
    for i := 0; i < b.N; i++ {
        MenuFast(d)
    }
}

Real output:

BenchmarkMenu-2                9     115730039 ns/op    247025728 B/op    195272 allocs/op
BenchmarkMenuFast-2          542       2231287 ns/op      241074 B/op       9747 allocs/op

About 50× faster and 1000× less memory. The CPU profile's -list=shop.Menu points at the MustCompile line first; the heap profile (alloc_space) at the += line.

Common mistakes (and the real messages)

Mistake What you see Why Fix
guessing the slow part sort took: 103.966µs of 27.348371ms intuition is often wrong profile first
reading only flat shop.Report flat 0 the time is in callees look at cum and -list
inuse_space to find GC pressure 0, 0% of 0 total the garbage is gone -sample_index=alloc_space
regexp.MustCompile in a loop regexp.MustCompile ~37% cum compiles every call a package-level variable
/debug/pprof on the internet internals exposed anyone can read and trigger profiles private port only
optimising thin boxes no measurable change not where time goes fix the widest box

Practice

  1. A function has flat 2% and cum 70%. Where should you look?
  2. Your service's memory grows all day. Which profile and sample index?
  3. Why can't a CPU profile show time spent waiting for a mutex?
  4. Where must default.pgo be for go build to use it?
Solutions

1. In its callees: run -list on it, or look at the boxes above/below it in the flame graph.

2. The heap profile with inuse_space (the default), compared between two moments.

3. It only samples goroutines running on a CPU; a goroutine blocked on a mutex isn't running. Use the block or mutex profile, or a trace (episode 58).

4. In the main package's directory.

Quick quiz

1. shop.Load flat 1% cum 75%; json.Unmarshal flat 45% cum 45%; strings.Split flat 15%. Which first? (From the video.)

A. shop.Load · B. json.Unmarshal · C. strings.Split · D. none

Answer

B. Load's cum is high only because it calls Unmarshal, which does 45% of the work itself.

2. How many samples per second does Go's CPU profiler take?

Answer

100.

3. What does a box's width mean in a flame graph?

Answer

Its share of the samples (time), including its callees.

Interview questions

  1. What is a sampling profiler? It interrupts the program at a fixed rate and records the current call stack; counts per function give the profile.
  2. flat vs cum? flat: time in the function itself; cum: including callees.
  3. How do you take a CPU profile in Go? go test -cpuprofile, pprof.StartCPUProfile/StopCPUProfile, or /debug/pprof/profile via net/http/pprof.
  4. inuse_space vs alloc_space? Live memory at profile time vs everything allocated since start.
  5. How do you find a goroutine leak? Goroutine profile: many goroutines with the same stack.
  6. Block vs mutex profile? Block: where goroutines waited; mutex: which lock holders caused waiting.
  7. How do you profile production safely? net/http/pprof on a private port, short CPU profiles; profiles are low overhead.
  8. What is PGO in Go? Building with a CPU profile (default.pgo) so the compiler inlines hot code more; around 2–14% faster.

Glossary

Term Meaning Example
Profiler a tool that measures where a program spends time or memory pprof
Profiling running a program under a profiler go test -cpuprofile
Sampling recording the program's state at regular moments 100 times a second
Profile the collected samples, counted per function cpu.prof
Call stack the chain of functions that led to the current point main → Report → MustCompile
pprof Go's profiling package and tool go tool pprof
CPU profile samples of code running on the CPU -cpuprofile cpu.prof
go tool pprof -top the busiest functions as a table -top -cum -nodecount=9
-list source lines with samples -list=shop.Report
flat time in the function itself 0 for Report
cum (cumulative) time in the function and its callees 58.70%
Flame graph boxes stacked by call chain, width = time pprof -http view
-http opens pprof's web UI -http=127.0.0.1:8090
Immutable can't be changed after creation; += makes a new string Go strings
strings.Builder a growable buffer for building strings sb.WriteString(line)
Heap profile where memory was allocated -memprofile mem.prof
alloc_space all bytes allocated since start 2021.42MB total
inuse_space bytes still live at profile time 0 total
GC pressure lots of allocation that keeps the GC busy out += line
Goroutine profile stacks of all live goroutines goroutine profile: total 101
pprof.Lookup get a named profile pprof.Lookup("goroutine")
Goroutine leak goroutines blocked forever 100 stuck senders
Block profile where goroutines waited sync.(*Mutex).Lock
Mutex profile which lock holders made others wait sync.(*Mutex).Unlock
SetBlockProfileRate, SetMutexProfileFraction turn those profiles on in a program rate 1 = every event
net/http/pprof HTTP handlers serving profiles /debug/pprof/
Blank import importing only for side effects _ "net/http/pprof"
PGO profile-guided optimisation default.pgo
default.pgo the CPU profile go build uses automatically in the main package dir
go version -m prints a binary's build information build -pgo=default.pgo
Inlining pasting a small function's body into its caller more aggressive with PGO

Summary

  • Don't guess: measure. A sampling profiler records the call stack 100 times a second.
  • CPU profile: go test -cpuprofile, go tool pprof -top (flat vs cum) and -list for lines; flame graph: width = time.
  • Heap profile: alloc_space for GC pressure, inuse_space for leaks.
  • Goroutine profile finds stuck goroutines; block and mutex profiles find waiting.
  • net/http/pprof profiles live servers (private port only).
  • PGO: commit a production CPU profile as default.pgo for around 2–14% speed.

Next episode: execution tracing and performance tuning: runtime/trace, go tool trace, the flight recorder, latency vs throughput, fewer allocations, sync.Pool, inlining, bounds-check elimination.