These are the notes for Go episode 42. Raju's receipt joins items with s += item, and a friend says strings.Builder is faster. Timing one call with time.Now gives numbers that jump around, so you invent the benchmark: run the code b.N times and report the average, ns/op. Then you meet the lies benchmarks can tell (setup counted, work removed by the compiler) and their fixes (b.ResetTimer, b.Loop), memory columns (-benchmem, B/op, allocs/op), sub-benchmarks, noise and benchstat. Finally a unit test passes but fuzzing finds a name that Short cuts in the middle of a letter, saves it in testdata/fuzz, and every later go test re-runs it.
All outputs and errors here are from real runs (Go 1.24.7, linux/amd64, 2 CPUs at 2.10 GHz). Timings change on every run and on every machine: your ns/op numbers will differ; allocation counts, the b.N sequence and error messages are stable. In the SyntaxPilot compiler, a program whose first lines are // $ go test ... is run as a package: each // ---- name ---- line starts a new file (a go.mod for module shop is added), and the commands run in order (episode 40 explains this). On your computer: go mod init shop, create the files, run the commands.
Facts and sources
go doc testing.B.Loop(Go 1.24.7): "Loop returns true as long as the benchmark should continue running." "Loop resets the benchmark timer the first time it is called in a benchmark, so any setup performed prior to starting the benchmark loop does not count toward the benchmark measurement. Likewise, when it returns false, it stops the timer so cleanup code is not measured." "The compiler never optimizes away calls to functions within the body of a "for b.Loop() { ... }" loop." "Benchmarks should either use Loop or contain a loop to b.N, but not both."- Go 1.24 release notes (tip.golang.org/doc/go1.24): Go 1.24 arrived in February 2025. "Benchmarks may now use the faster and less error-prone testing.B.Loop method to perform benchmark iterations ... in place of the typical loop structures involving b.N": "The benchmark function will execute exactly once per -count, so expensive setup and cleanup steps execute only once." "Function call parameters and results are kept alive, preventing the compiler from fully optimizing away the loop body."
go help testflag:-benchtime t"Run enough iterations of each benchmark to take t ... The default is 1 second (1s). The special syntax Nx means to run the benchmark N times (for example, -benchtime 100x)."-count n"Run each test, benchmark, and fuzz seed n times (default 1)."-fuzz regexp"the command line argument must match exactly one package within the main module, and regexp must match exactly one fuzz test within that package."-fuzztime t"The default is to run forever."- Go Fuzzing docs (golang.google.cn/doc/fuzz): fuzzing is in the standard toolchain "beginning in Go 1.18". Allowed argument types:
string,[]byte,int,int8,int16,int32/rune,int64,uint,uint8/byte,uint16,uint32,uint64,float32,float64,bool. The seed corpus comes fromf.Addcalls and files intestdata/fuzz/{FuzzTestName}; these run by default withgo test. A failing input is written totestdata/fuzz/{FuzzTestName}and the nextgo testuses it as a regression test. A fuzz test is theFuzzXxx(f *testing.F)function; the fuzz target is the function passed tof.Fuzz. - benchstat doc comment (go.googlesource.com/perf cmd/benchstat): "Benchstat computes statistical summaries and A/B comparisons of Go benchmarks." "Each benchmark should be run at least 10 times to gather a statistically significant sample of results." "The p-value measures how likely it is that any differences were due to random chance." "the '~' means benchstat did not detect a statistically significant difference between the two inputs." The example table below is from that page.
The problem: timing one call
Raju's shop prints a line of items on every receipt. Two ways to join them: += and strings.Builder. The obvious way to compare them: time each one with time.Now() and time.Since().
package main
import (
"fmt"
"strings"
"time"
)
func joinPlus(items []string) string {
s := ""
for _, it := range items {
s += it + ","
}
return s
}
func joinBuilder(items []string) string {
var b strings.Builder
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
func main() {
items := []string{"chai", "samosa", "vada pav", "lassi"}
for i := 0; i < 3; i++ {
t := time.Now()
joinPlus(items)
fmt.Println("+= took", time.Since(t))
t = time.Now()
joinBuilder(items)
fmt.Println("Builder took", time.Since(t))
}
}
Output of two runs (go run main.go):
+= took 4.653µs
Builder took 456ns
+= took 430ns
Builder took 180ns
+= took 321ns
Builder took 159ns
+= took 4.236µs
Builder took 561ns
+= took 436ns
Builder took 227ns
+= took 260ns
Builder took 268ns
The numbers jump around. The first call is about ten times slower (the CPU caches are cold), and in the second run the last pair even flips: Builder looks slower. µs is a microsecond (a millionth of a second); ns is a nanosecond (a billionth of a second, a thousand times smaller).
Why it fails: one call takes a few hundred nanoseconds. In that time the noise (cold caches, another program taking the CPU for a moment, the cost of reading the clock itself) is bigger than the thing being measured. One reading tells you nothing.
Benchmarks: run it many times (timing one ball vs a whole spell)
Timing one cricket ball with a phone is mostly your thumb. Time 1,000 balls and divide by 1,000 and the thumb error almost vanishes. That is a benchmark: run the code many times and report the average time per run. How many times? Go decides, and the count is b.N.
The rules:
- it lives in a _test.go file;
- its name starts with Benchmark (BenchmarkXxx);
- it takes one parameter b *testing.B (a pointer to testing.B, the benchmark's handle);
- it runs the code b.N times.
// $ go test
// $ go test -bench=.
// ---- bill.go ----
package shop
import "strings"
func JoinPlus(items []string) string {
s := ""
for _, it := range items {
s += it + ","
}
return s
}
func JoinBuilder(items []string) string {
var b strings.Builder
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
// ---- bill_test.go ----
package shop
import "testing"
var items = []string{"chai", "samosa", "vada pav", "lassi"}
func BenchmarkJoinPlus(b *testing.B) {
for i := 0; i < b.N; i++ {
JoinPlus(items)
}
}
func BenchmarkJoinBuilder(b *testing.B) {
for i := 0; i < b.N; i++ {
JoinBuilder(items)
}
}
Output (real run; your timings will differ):
$ go test
testing: warning: no tests to run
PASS
ok shop 0.002s
$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinPlus-2 6605364 177.6 ns/op
BenchmarkJoinBuilder-2 10385799 113.1 ns/op
PASS
ok shop 2.662s
(The first part, plain go test printing "no tests to run", is from a run of the same kind of package; benchmarks never run without -bench.)
How to read a line:
| Part | Meaning |
|---|---|
BenchmarkJoinPlus |
the benchmark's name |
-2 |
GOMAXPROCS: how many CPUs Go used (2 here) |
6605364 |
the final b.N: how many iterations ran |
177.6 ns/op |
average nanoseconds per operation (one trip round the loop) |
-bench=regexpruns benchmarks whose name matches the pattern;.matches all. It works like-runfor tests.- Builder wins, but with only four items, not by much.
How b.N is chosen
// $ go test -bench=Builder
// ---- bill_test.go ----
package shop
import (
"fmt"
"strings"
"testing"
)
func JoinBuilder(items []string) string {
var b strings.Builder
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
var items = []string{"chai", "samosa", "vada pav", "lassi"}
func BenchmarkJoinBuilder(b *testing.B) {
fmt.Println("called with b.N =", b.N)
for i := 0; i < b.N; i++ {
JoinBuilder(items)
}
}
Output (real run; your timings will differ):
$ go test -bench=Builder
called with b.N = 1
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinBuilder-2 called with b.N = 100
called with b.N = 10000
called with b.N = 1000000
called with b.N = 11315289
11315289 104.2 ns/op
PASS
ok shop 1.293s
Go calls the benchmark function several times with a growing b.N: 1, 100, 10,000, 1,000,000, then a final count it predicts will fill the benchtime (default 1 second, -benchtime=2s or -benchtime=100x to change it). The final b.N is the number printed in the result line.
Setup cost and b.ResetTimer
The menu must be loaded (100 ms, like reading a file) before it can be sorted. That load is setup, not what we want to measure. -benchtime=100x runs exactly 100 iterations.
// $ go test -bench=. -benchtime=100x
// ---- menu_test.go ----
package shop
import (
"slices"
"testing"
"time"
)
func loadMenu() []string {
time.Sleep(100 * time.Millisecond) // like reading a big file
return []string{"vada pav", "chai", "samosa", "lassi", "poha"}
}
func BenchmarkSortNoReset(b *testing.B) {
menu := loadMenu() // setup
for i := 0; i < b.N; i++ {
slices.Sorted(slices.Values(menu))
}
}
func BenchmarkSortReset(b *testing.B) {
menu := loadMenu() // setup
b.ResetTimer() // start the stopwatch here
for i := 0; i < b.N; i++ {
slices.Sorted(slices.Values(menu))
}
}
Output (real run; your timings will differ):
$ go test -bench=. -benchtime=100x
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkSortNoReset-2 100 1005746 ns/op
BenchmarkSortReset-2 100 783.1 ns/op
PASS
ok shop 0.414s
Without the reset, the 100 ms setup is spread over 100 sorts: about 1,000,000 ns (1 ms) per "sort", a lie. b.ResetTimer() sets the elapsed time (and allocation counters) back to zero, so only the loop is measured: well under a microsecond per sort. Related: b.StopTimer() and b.StartTimer() pause and resume the stopwatch inside the loop.
b.Loop and the vanishing loop
withGST adds 18% GST (Goods and Services Tax) to a price in paise (1 rupee = 100 paise), rounding to the nearest paisa.
// $ go test -bench=.
// ---- gst_test.go ----
package shop
import "testing"
// GST: price in paise plus 18%, rounded
func withGST(paise int) int {
return (paise*118 + 50) / 100
}
func BenchmarkGSTOld(b *testing.B) {
for i := 0; i < b.N; i++ {
withGST(4000) // result thrown away
}
}
func BenchmarkGSTLoop(b *testing.B) {
for b.Loop() { // Go 1.24
withGST(4000)
}
}
Output (real run; your timings will differ):
$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkGSTOld-2 1000000000 0.4033 ns/op
BenchmarkGSTLoop-2 603246115 1.903 ns/op
PASS
ok shop 1.603s
0.4 ns per call is impossible. This CPU runs at 2.1 GHz (2.1 billion clock ticks a second), so one tick is about 0.48 ns. A function call, a multiply, an add and a divide can't happen in less than one tick. What happened: the compiler inlined withGST (copied its body into the loop) and then saw that the result is never used, so it deleted the work. That is dead code elimination. The benchmark measured an empty loop.
for b.Loop() { ... } (Go 1.24, February 2025) fixes this: calls written directly inside the loop body are never optimized away, so we get a real ~1.9 ns. b.Loop also:
- resets the timer the first time it is called, so setup before the loop isn't counted (no b.ResetTimer needed);
- stops the timer when it returns false, so cleanup after the loop isn't counted;
- runs the benchmark function once per -count, instead of once per growing b.N, so expensive setup runs once.
Use b.Loop for new code. Never mix it with a b.N loop in the same benchmark (see Common mistakes). With the old b.N style, the usual workaround was storing the result in a package-level variable (a "sink") so the compiler couldn't remove it.
Memory: -benchmem and allocs/op (new tiffin box for every item)
Imagine buying a new, bigger tiffin box for every item you pack, moving everything over and throwing the old box away. Go strings can't be changed (they are immutable), so s += item makes a brand-new string each time and copies all the old text into it. Each new block of memory taken from the heap (the memory area for values that outlive a function call) is an allocation. Allocations cost time, and the garbage collector (the part of the Go runtime that finds and frees memory nobody uses any more) must clean each one up later.
strings.Builder keeps one buffer with spare room and only moves to a bigger one now and then; b.Grow(n) reserves the room up front.
// $ go test -bench=. -benchmem
// ---- bill.go ----
package shop
import "strings"
func JoinPlus(items []string) string {
s := ""
for _, it := range items {
s += it + ","
}
return s
}
func JoinBuilder(items []string) string {
var b strings.Builder
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
// ---- bill_test.go ----
package shop
import (
"slices"
"testing"
)
var items = slices.Repeat([]string{"chai"}, 1000) // 1000 lines
func BenchmarkJoinPlus(b *testing.B) {
for b.Loop() {
JoinPlus(items)
}
}
func BenchmarkJoinBuilder(b *testing.B) {
for b.Loop() {
JoinBuilder(items)
}
}
Output (real run; your timings will differ):
$ go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinPlus-2 1222 965959 ns/op 2684239 B/op 1000 allocs/op
BenchmarkJoinBuilder-2 94279 11772 ns/op 17912 B/op 13 allocs/op
PASS
ok shop 2.295s
slices.Repeat(s, n)(Go 1.23) returns a new slice withsrepeatedntimes: here 1,000"chai"s.- With 1,000 lines,
+=is about 80 times slower in this run. - B/op: bytes allocated per operation: 2.6 MB for
+=(every step copies the whole receipt so far), 17.9 KB for Builder. - allocs/op: allocations per operation: exactly 1,000 (one per line) vs 13.
Sub-benchmarks and b.ReportAllocs
b.Run(name, func(b *testing.B)) creates a sub-benchmark with its own name, like t.Run creates subtests. b.ReportAllocs() turns on the memory columns for that benchmark even without -benchmem.
// $ go test -bench=.
// ---- bill_test.go ----
package shop
import (
"fmt"
"slices"
"strings"
"testing"
)
func JoinPlus(items []string) string {
s := ""
for _, it := range items {
s += it + ","
}
return s
}
func JoinGrow(items []string) string {
var b strings.Builder
b.Grow(len(items) * 5) // reserve room once
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
func BenchmarkJoin(b *testing.B) {
for _, n := range []int{10, 100, 1000} {
items := slices.Repeat([]string{"chai"}, n)
b.Run(fmt.Sprintf("plus/n=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
JoinPlus(items)
}
})
b.Run(fmt.Sprintf("grow/n=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
JoinGrow(items)
}
})
}
}
Output (real run; your timings will differ):
$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoin/plus/n=10-2 2304675 487.6 ns/op 328 B/op 10 allocs/op
BenchmarkJoin/grow/n=10-2 11403918 102.7 ns/op 64 B/op 1 allocs/op
BenchmarkJoin/plus/n=100-2 99112 12766 ns/op 26376 B/op 100 allocs/op
BenchmarkJoin/grow/n=100-2 1732596 669.6 ns/op 512 B/op 1 allocs/op
BenchmarkJoin/plus/n=1000-2 1446 813052 ns/op 2684233 B/op 1000 allocs/op
BenchmarkJoin/grow/n=1000-2 185520 6013 ns/op 5376 B/op 1 allocs/op
PASS
ok shop 7.025s
| n | += ns/op |
growth | Grow ns/op | growth |
|---|---|---|---|---|
| 10 | 487.6 | 102.7 | ||
| 100 | 12,766 | ×26 | 669.6 | ×6.5 |
| 1000 | 813,052 | ×64 | 6,013 | ×9 |
Ten times more items costs += far more than ten times the time: each step copies everything before it, so the total work is about 1 + 2 + … + n, which grows like n². That is quadratic growth. JoinGrow grows roughly in step with n (linear), with exactly 1 allocation at every size.
-bench=Join/growruns only the grow sub-benchmarks (each/-separated part of the pattern matches one level of names).-run='^$'is a test pattern that matches no test name, so only benchmarks run: use it when benchmarking.
Noise, -count and benchstat
// $ go test -bench=JoinBuilder -count=5
// ---- bill_test.go ----
package shop
import (
"strings"
"testing"
)
func JoinBuilder(items []string) string {
var b strings.Builder
for _, it := range items {
b.WriteString(it)
b.WriteString(",")
}
return b.String()
}
var items = []string{"chai", "samosa", "vada pav", "lassi"}
func BenchmarkJoinBuilder(b *testing.B) {
for b.Loop() {
JoinBuilder(items)
}
}
Output (real run; your timings will differ):
$ go test -bench=JoinBuilder -count=5
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinBuilder-2 9751136 117.5 ns/op
BenchmarkJoinBuilder-2 12359983 102.8 ns/op
BenchmarkJoinBuilder-2 11107392 122.2 ns/op
BenchmarkJoinBuilder-2 12428426 104.5 ns/op
BenchmarkJoinBuilder-2 11530644 117.8 ns/op
PASS
ok shop 6.440s
Same code, five runs: 102.8 to 122.2 ns/op, a spread of almost 20%. So "5% faster" from one run before and one after proves nothing. You need many runs of each version and statistics that say whether the difference is bigger than the wobble. That is benchstat, from the Go team's golang.org/x/perf repository:
$ go install golang.org/x/perf/cmd/benchstat@latest
$ go test -run='^$' -bench=. -count=10 > old.txt
... change the code ...
$ go test -run='^$' -bench=. -count=10 > new.txt
$ benchstat old.txt new.txt
Example output from the benchstat documentation (not from our machine):
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Encode/format=json 1.718µ ± 1% 1.423µ ± 1% -17.20% (p=0.000 n=10)
Encode/format=gob 3.066µ ± 0% 3.070µ ± 2% ~ (p=0.446 n=10)
geomean 2.295µ 2.090µ -8.94%
| Part | Meaning |
|---|---|
1.718µ |
the median (middle value) of the 10 runs, in seconds per op |
± 1% |
the spread: a 95% confidence interval around the median |
-17.20% |
the change from old to new (vs base) |
p=0.000 |
the p-value: how likely a difference this big is pure chance; small (below 0.05) means real |
n=10 |
samples used from each file |
~ |
no statistically significant difference: don't celebrate |
geomean |
geometric mean over all benchmarks: the overall change |
Rules: at least 10 runs (ideally 20) per version, the same number every time, on a quiet machine.
Fuzzing: when your examples aren't enough
Short fits a customer's name on the receipt (at most 6 letters). Raju's table test:
// $ go test -v
// ---- name.go ----
package shop
// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
if len(name) <= 6 {
return name
}
return name[:6]
}
// ---- name_test.go ----
package shop
import "testing"
func TestShort(t *testing.T) {
cases := map[string]string{
"Raju": "Raju",
"Priyanka": "Priyan",
"Lakshmipati": "Lakshm",
}
for in, want := range cases {
if got := Short(in); got != want {
t.Errorf("Short(%q) = %q, want %q", in, got, want)
}
}
}
Output (real run; your timings will differ):
$ go test -v
=== RUN TestShort
--- PASS: TestShort (0.00s)
PASS
ok shop 0.003s
It passes, but only for the names Raju thought of.
The phone drop test
| In a phone factory | In Go fuzzing |
|---|---|
| a machine drops phones at random angles, thousands of times | fuzzing: the computer invents inputs and runs your code with each |
| start from known hard drops (corner, edge) | seed corpus: starting inputs added with f.Add |
| rule: the screen must not crack | property: a rule true for every input |
| save the angle that cracked it, re-test every new model | the failing input is saved in testdata/fuzz/FuzzXxx and re-run by every go test |
Go's fuzzer is coverage-guided: it watches which parts of your code each input reaches and keeps mutating ("changing a little") the inputs that reach new code. Built into the toolchain since Go 1.18.
Writing a fuzz test
- name starts with
Fuzz(FuzzXxx), takesf *testing.F, returns nothing, in a_test.gofile; f.Add(...)adds seeds; their types must match the target's parameters, in order;f.Fuzz(func(t *testing.T, ...))gives the fuzz target, exactly one per fuzz test;- with random input you can't write the expected answer, so check a property. Here: if the name was valid UTF-8, the result must be valid UTF-8 (
utf8.ValidString).
// $ go test -fuzz=FuzzShort -fuzztime=10s
// $ go test -v
// ---- name.go ----
package shop
// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
if len(name) <= 6 {
return name
}
return name[:6]
}
// ---- name_test.go ----
package shop
import (
"testing"
"unicode/utf8"
)
func FuzzShort(f *testing.F) {
f.Add("Raju") // seed corpus
f.Add("Priyanka")
f.Fuzz(func(t *testing.T, name string) {
got := Short(name)
if utf8.ValidString(name) && !utf8.ValidString(got) {
t.Errorf("Short(%q) = %q: broken letters", name, got)
}
})
}
Output (real run; the fuzzer is random, so your failing input and its file name will differ):
$ go test -fuzz=FuzzShort -fuzztime=10s
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 2 workers
fuzz: elapsed: 0s, execs: 3574 (40978/sec), new interesting: 7 (total: 9)
--- FAIL: FuzzShort (0.09s)
--- FAIL: FuzzShort (0.00s)
name_test.go:14: Short("00000ݕ") = "00000\xdd": broken letters
Failing input written to testdata/fuzz/FuzzShort/d2b01acf94f5cb32
To re-run:
go test -run=FuzzShort/d2b01acf94f5cb32
FAIL
exit status 1
FAIL shop 0.090s
$ go test -v
=== RUN FuzzShort
=== RUN FuzzShort/seed#0
=== RUN FuzzShort/seed#1
=== RUN FuzzShort/d2b01acf94f5cb32
name_test.go:14: Short("00000ݕ") = "00000\xdd": broken letters
--- FAIL: FuzzShort (0.00s)
--- PASS: FuzzShort/seed#0 (0.00s)
--- PASS: FuzzShort/seed#1 (0.00s)
--- FAIL: FuzzShort/d2b01acf94f5cb32 (0.00s)
FAIL
exit status 1
FAIL shop 0.003s
Reading the fuzz lines: - gathering baseline coverage: first it runs the seeds (and any saved inputs) to learn which code they reach; - workers: copies of the test binary running inputs in parallel (default: GOMAXPROCS, 2 here); - execs: inputs tried so far, and per second; - new interesting: inputs that reached new code and were added to the generated corpus.
In 0.09 s it found "00000ݕ": five zeros and a letter outside English. ݕ is 2 bytes in UTF-8, so name[:6] cut it in half and left the broken byte \xdd.
Why: len counts bytes, not letters (episode 17). In UTF-8, English letters are 1 byte, most others 2, 3 or 4 bytes (Devanagari letters are 3). name[:6] cuts at byte 6, which can be the middle of a letter, so a customer named in Hindi or Tamil gets a broken receipt.
Crash reproduction: the failing input was written to testdata/fuzz/FuzzShort/<hash>. The plain go test -v after it ran seed#0, seed#1 and the saved file as subtests, and failed on it: a regression test for free. Re-run just that one with go test -run=FuzzShort/<hash>. Commit testdata/fuzz with your tests. A crash file looks like:
go test fuzz v1
string("00000ݕ")
Fixing it, and fuzzing again
Convert to []rune (letters, not bytes) and cut that. A second property: the result has at most 6 letters (utf8.RuneCountInString).
// $ go test -fuzz=FuzzShort -fuzztime=5s
// ---- name.go ----
package shop
// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
r := []rune(name) // letters, not bytes
if len(r) <= 6 {
return name
}
return string(r[:6])
}
// ---- name_test.go ----
package shop
import (
"testing"
"unicode/utf8"
)
func FuzzShort(f *testing.F) {
f.Add("Raju") // seed corpus
f.Add("Priyanka")
f.Fuzz(func(t *testing.T, name string) {
got := Short(name)
if utf8.ValidString(name) && !utf8.ValidString(got) {
t.Errorf("Short(%q) = %q: broken letters", name, got)
}
if utf8.RuneCountInString(got) > 6 {
t.Errorf("Short(%q) = %q: too long", name, got)
}
})
}
Output (real run; your timings will differ):
$ go test -fuzz=FuzzShort -fuzztime=5s
fuzz: elapsed: 0s, gathering baseline coverage: 0/9 completed
fuzz: elapsed: 0s, gathering baseline coverage: 9/9 completed, now fuzzing with 2 workers
fuzz: elapsed: 3s, execs: 107799 (35927/sec), new interesting: 19 (total: 28)
fuzz: elapsed: 6s, execs: 120746 (4316/sec), new interesting: 21 (total: 30)
fuzz: elapsed: 6s, execs: 120746 (0/sec), new interesting: 21 (total: 30)
PASS
ok shop 6.024s
(The baseline here is 9 inputs because the earlier run's interesting inputs were kept in the Go build cache, $GOCACHE/fuzz; yours may differ.)
- Over 100,000 names in about 5 seconds, no failure: PASS.
- Without
-fuzztime, fuzzing runs until it finds a failure or you press Ctrl+C. - Plain
go testnever fuzzes: it only runs the seeds and the saved files intestdata/fuzz, in milliseconds. - Fuzz arguments can only be
string,[]byte, integers, floats andbool. -fuzzmust match exactly one fuzz test, in one package.- Fuzz anything that reads text from outside: parsers, decoders, validators.
Benchmarks and fuzzing, written down
func BenchmarkX(b *testing.B) {
// setup (not timed)
for b.Loop() {
X()
}
}
func FuzzX(f *testing.F) {
f.Add("seed")
f.Fuzz(func(t *testing.T, s string) {
// check a property of X(s)
})
}
| Command | What it does |
|---|---|
go test -bench=. |
run all benchmarks (tests too) |
go test -run='^$' -bench=Join -benchmem |
only benchmarks matching Join, with memory columns |
go test -bench=. -benchtime=100x |
exactly 100 iterations each |
go test -bench=. -count=10 > new.txt |
10 samples for benchstat |
go test -fuzz=FuzzX -fuzztime=30s |
fuzz FuzzX for 30 seconds |
go test -run=FuzzX/<hash> |
re-run one saved failing input |
Mini project: a faster, safer receipt
Receipt(items []Item) stringprintsname xQty Rs totalper line. Write it with+=(ReceiptPlus), then withstrings.Builder+Grow.- Sub-benchmarks for 10, 100 and 1,000 items with
b.ReportAllocs. FuzzReceipt: the receipt is valid UTF-8, has one line per item, never panics, and both versions agree.- Compare the two versions with
-count=10and benchstat.
Solution
receipt.go:
package shop
import (
"fmt"
"strings"
)
// Item is one line on the receipt.
type Item struct {
Name string
Qty int
Price int // rupees per piece
}
// ReceiptPlus builds the receipt with += (slow for long receipts).
func ReceiptPlus(items []Item) string {
s := ""
for _, it := range items {
s += fmt.Sprintf("%s x%d Rs %d\n", it.Name, it.Qty, it.Qty*it.Price)
}
return s
}
// Receipt builds the same text with a strings.Builder sized up front.
func Receipt(items []Item) string {
var b strings.Builder
b.Grow(len(items) * 24) // a typical line is under 24 bytes
for _, it := range items {
fmt.Fprintf(&b, "%s x%d Rs %d\n", it.Name, it.Qty, it.Qty*it.Price)
}
return b.String()
}
receipt_test.go:
package shop
import (
"fmt"
"strings"
"testing"
"unicode/utf8"
)
func makeItems(n int) []Item {
items := make([]Item, n)
for i := range items {
items[i] = Item{Name: "chai", Qty: 2, Price: 20}
}
return items
}
func BenchmarkReceipt(b *testing.B) {
for _, n := range []int{10, 100, 1000} {
items := makeItems(n)
b.Run(fmt.Sprintf("plus/n=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
ReceiptPlus(items)
}
})
b.Run(fmt.Sprintf("builder/n=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
Receipt(items)
}
})
}
}
func FuzzReceipt(f *testing.F) {
f.Add("chai", 2, 20)
f.Add("समोसा", 1, 15)
f.Fuzz(func(t *testing.T, name string, qty, price int) {
if strings.Contains(name, "\n") {
t.Skip("a name with a newline would add a line")
}
got := Receipt([]Item{{name, qty, price}})
if utf8.ValidString(name) && !utf8.ValidString(got) {
t.Errorf("receipt is not valid UTF-8: %q", got)
}
if n := strings.Count(got, "\n"); n != 1 {
t.Errorf("got %d lines, want 1: %q", n, got)
}
if got != ReceiptPlus([]Item{{name, qty, price}}) {
t.Errorf("Receipt and ReceiptPlus disagree")
}
})
}
Run:
go test -run='^$' -bench=Receipt -benchmem
go test -fuzz=FuzzReceipt -fuzztime=30s
To compare with benchstat, benchmark only plus into old.txt and only builder into new.txt (rename the sub-benchmarks to the same name, e.g. n=1000, in each run so benchstat pairs them), with -count=10 each, then benchstat old.txt new.txt. The seeds pass with go test; expect the Builder version to need far fewer allocations as n grows. The t.Skip handles a name containing a newline, which would legitimately add a line: a fuzz test must state its property precisely.
Common mistakes (and the real messages)
| Mistake | Real message / result | Why | Fix |
|---|---|---|---|
plain go test, expecting benchmarks |
testing: warning: no tests to run |
benchmarks need -bench |
go test -bench=. |
func BenchmarkA(b testing.B) |
wrong signature for BenchmarkA, must be: func BenchmarkA(b *testing.B) |
not a pointer (found by go vet, which go test runs) |
b *testing.B |
result unused in a b.N loop |
0.4033 ns/op |
the compiler removed the call | for b.Loop() |
| setup inside the timed part | 1005746 ns/op for a sub-microsecond sort |
setup counted | b.ResetTimer() or b.Loop |
b.Loop inside a b.N loop |
B.Loop called with timer stopped |
use one style, not both | only for b.Loop() |
f.Add(4) for a string target |
mismatched type in call to (*testing.F).Add: int, fuzz target expects string |
seed types must match | f.Add("4") |
f.Fuzz(func(t *testing.T, s []string)) |
fuzzing arguments can only have the following types: string, bool, float32, float64, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, []byte |
unsupported type | string or []byte |
go test -fuzz=. with two fuzz tests |
testing: will not fuzz, -fuzz matches more than one fuzz test: [FuzzA FuzzB] |
one fuzz test at a time | -fuzz=FuzzA |
| comparing one run before and after | "5% faster" that is just noise | runs wobble | -count=10 + benchstat |
not committing testdata/fuzz |
the bug can return unnoticed | the regression input is lost | commit it |
Practice
- (Homework) Build the receipt mini project (solution above).
- Benchmark
strconv.Itoa(n)againstfmt.Sprint(n)with-benchmem. Which allocates less?
Solution
func BenchmarkItoa(b *testing.B) {
for b.Loop() {
strconv.Itoa(123456)
}
}
func BenchmarkSprint(b *testing.B) {
for b.Loop() {
fmt.Sprint(123456)
}
}
Run go test -run='^$' -bench=. -benchmem. strconv.Itoa is faster and makes fewer allocations: fmt.Sprint takes ...any, so the number is boxed into an interface and formatted with reflection-like machinery. Check your own numbers.
- A benchmark reports
0.25 ns/opon a 3 GHz CPU. What do you suspect, and how do you fix it?
Solution
One tick at 3 GHz is about 0.33 ns, so less than a tick per call means the work was removed (dead code elimination). Rewrite it with for b.Loop() { ... }.
- Write a fuzz test for
func Reverse(s string) stringwith the property: reversing twice gives the original, for valid UTF-8 input.
Solution
func FuzzReverse(f *testing.F) {
f.Add("chai")
f.Add("नमस्ते")
f.Fuzz(func(t *testing.T, s string) {
if !utf8.ValidString(s) {
t.Skip()
}
if got := Reverse(Reverse(s)); got != s {
t.Errorf("Reverse(Reverse(%q)) = %q", s, got)
}
})
}
A byte-by-byte Reverse fails; reversing a []rune passes.
- Your benchmark loads a 50 MB file before measuring. Old
b.Nstyle: where doesb.ResetTimer()go? Withb.Loop?
Solution
Right after loading, before the loop. With b.Loop you need nothing: it resets the timer the first time it is called, and the function (so the load) runs once.
Quick quiz
1. What does this benchmark line tell you? (From the video.)
BenchmarkJoinPlus-2 1222 965959 ns/op 2684239 B/op 1000 allocs/op
A. It ran 1222 times; each call took ~0.97 ms and made 1000 allocations · B. It ran 2 times on 1222 CPUs · C. Each call took 965959 seconds · D. It allocated 1000 bytes in total
Answer
A. 1222 is the final b.N; 965959 ns/op ≈ 0.97 ms per call; 2684239 B/op (2.6 MB) and 1000 allocs/op are per call; -2 is GOMAXPROCS.
2. Which values of b.N does Go try first?
Answer
1, 100, 10000, 1000000, ... until a run fills the benchtime (1 s by default).
3. Does plain go test fuzz?
Answer
No. It runs the seed corpus and the saved inputs in testdata/fuzz as ordinary subtests. Fuzzing needs -fuzz.
4. benchstat shows ~ (p=0.446 n=10). Is the new version faster?
Answer
No detectable difference: ~ means not statistically significant.
Interview questions
- How do you write and run a benchmark in Go?
func BenchmarkX(b *testing.B)in a_test.gofile, loop withfor b.Loop()(orb.Ntimes), run withgo test -bench=X -benchmem. - What is b.N and who sets it? The iteration count; the testing package grows it (1, 100, 10000, …) until the run fills
-benchtime. - What does b.Loop (Go 1.24) improve? It times only the loop (setup and cleanup excluded), runs the function once per
-count, and keeps calls in the loop body from being optimized away. - Why can a benchmark report an impossibly small time? The compiler inlined the function and eliminated it because the result was unused (dead code elimination).
- What do B/op and allocs/op mean, and why care? Bytes and heap allocations per operation; allocations cost time now and garbage-collector work later.
- Why is building a string with += in a loop slow? Strings are immutable; each
+=allocates a new string and copies the old content: quadratic time. - How do you decide if an optimization really helped? Run both versions at least 10 times (
-count=10) and compare with benchstat; trust a small p-value, not one run. - What is fuzzing, and how is Go's fuzzer guided? Automatic generation of inputs from a seed corpus; coverage-guided mutation keeps inputs that reach new code.
- Fuzz test vs fuzz target? The
FuzzX(f *testing.F)function vs the function passed tof.Fuzz. - What happens when the fuzzer finds a failure? It minimizes the input, writes it to
testdata/fuzz/FuzzX/<hash>, and every latergo testruns it as a regression test. - What makes a good fuzz property? Something true for every input without knowing the answer: no panic, valid output, round-trip (decode(encode(x)) == x), two implementations agree.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Benchmark | code that runs a function many times to measure its average speed | BenchmarkJoinPlus |
*testing.B |
the handle Go passes to a benchmark: b.N, b.Loop, b.Run, timers |
func BenchmarkX(b *testing.B) |
BenchmarkXxx |
the naming rule: starts with Benchmark, in a _test.go file |
BenchmarkSortReset |
b.N |
the number of iterations Go asks for; grows until the run fills the benchtime | 1, 100, 10000, 1000000, 11315289 |
b.Loop() |
Go 1.24 loop condition: times only the loop, keeps the work from being removed | for b.Loop() { withGST(4000) } |
b.ResetTimer() |
sets the benchmark's elapsed time and counters back to zero | after loadMenu() |
b.ReportAllocs() |
shows B/op and allocs/op for this benchmark, even without -benchmem |
inside each b.Run |
b.Run |
creates a sub-benchmark with its own name | b.Run("grow/n=100", ...) |
| Sub-benchmark | a named benchmark inside another, often one per input size | BenchmarkJoin/grow/n=10 |
-bench=regexp |
flag: run benchmarks matching the pattern | -bench=., -bench=Join/grow |
-benchtime |
flag: how long (default 1s) or how many times (100x) each benchmark runs |
-benchtime=100x |
-benchmem |
flag: add B/op and allocs/op columns | go test -bench=. -benchmem |
-count n |
flag: run each test/benchmark n times | -count=10 |
-run='^$' |
a test pattern matching no name, so only benchmarks run | go test -run='^$' -bench=. |
| ns/op | average nanoseconds per operation (one loop iteration) | 113.1 ns/op |
| B/op | bytes allocated per operation | 2684239 B/op |
| allocs/op | heap allocations per operation | 1000 allocs/op |
| GOMAXPROCS suffix | the -2 after a benchmark name: CPUs Go used |
BenchmarkJoinPlus-2 |
| Allocation | taking a new block of memory from the heap | each s += it |
| Heap | memory for values that must outlive the function that made them | a new string |
| Garbage collector | the runtime part that frees heap memory nobody uses | cleans up old receipt strings |
| Immutable | cannot be changed after creation; Go strings are immutable | += makes a new string |
strings.Builder |
a growable byte buffer for building strings efficiently | b.WriteString("chai") |
Grow(n) |
reserves room for n more bytes up front | b.Grow(len(items) * 5) |
slices.Repeat |
returns a slice repeated n times (Go 1.23) | slices.Repeat([]string{"chai"}, 1000) |
| Dead code elimination | the compiler deleting work whose result is never used | 0.4033 ns/op |
| Inlining | the compiler copying a small function's body into its caller | withGST in the loop |
| Clock tick / GHz | one step of the CPU clock; 2.1 GHz = 2.1 billion ticks a second (~0.48 ns each) | |
| Microsecond (µs) | a millionth of a second | 4.653µs |
| Nanosecond (ns) | a billionth of a second; 1 µs = 1000 ns | 456ns |
| Setup | preparation a benchmark needs that should not be timed | loadMenu() |
| Noise | random variation in measurements unrelated to your code | 102.8–122.2 ns/op |
| Quadratic growth | time grows with the square of the input: 10× input ≈ 100× time | += joining |
| Linear growth | time grows in step with the input | JoinGrow |
| Median | the middle value of sorted measurements | benchstat's 1.718µ |
| p-value | how likely a difference this large is pure chance | p=0.000 |
| Statistically significant | unlikely to be chance (benchstat: small p; ~ means not) |
-17.20% |
| benchstat | Go team tool that summarizes and compares benchmark results | benchstat old.txt new.txt |
| Fuzzing | automatically generating many inputs to find ones that break code | go test -fuzz=FuzzShort |
| Coverage-guided | the fuzzer keeps inputs that reach new code and mutates them | "new interesting" |
| Fuzz test | FuzzXxx(f *testing.F) function in a test file |
FuzzShort |
| Fuzz target | the function passed to f.Fuzz; runs on each input |
func(t *testing.T, name string) |
*testing.F |
the handle passed to a fuzz test: f.Add, f.Fuzz |
func FuzzShort(f *testing.F) |
f.Add |
adds a seed input; types must match the target | f.Add("Raju") |
f.Fuzz |
registers the fuzz target (exactly one per fuzz test) | f.Fuzz(func(t *testing.T, s string) {...}) |
| Corpus | a collection of inputs | seeds plus saved inputs |
| Seed corpus | the starting inputs: f.Add calls and files in testdata/fuzz/FuzzX |
"Raju", "Priyanka" |
| Property | a rule that must hold for every input | output is valid UTF-8 |
| Worker | a process running fuzz inputs in parallel | "fuzzing with 2 workers" |
| execs | number of inputs tried | execs: 3574 (40978/sec) |
| New interesting | inputs that reached new code, added to the corpus | new interesting: 7 |
-fuzz=regexp |
flag: fuzz the one fuzz test matching the pattern | -fuzz=FuzzShort |
-fuzztime |
flag: how long to fuzz (default: forever) | -fuzztime=10s |
testdata/fuzz |
folder where failing inputs are saved, one folder per fuzz test | testdata/fuzz/FuzzShort/d2b01acf94f5cb32 |
| Crash file | a saved failing input | go test fuzz v1 + string("00000ݕ") |
| Regression test | a test that makes sure a fixed bug never returns | the saved input re-run by go test |
| UTF-8 | the text encoding Go uses: 1 to 4 bytes per letter | ݕ = 2 bytes |
| Byte | 8 bits; len and s[:6] count bytes |
"Raju" = 4 bytes |
| Rune | one Unicode letter (code point), type int32 |
[]rune(name) |
utf8.ValidString |
reports whether a string is valid UTF-8 | false for "00000\xdd" |
utf8.RuneCountInString |
counts letters (runes), not bytes | "राजू" → 4 |
| GST | Goods and Services Tax, here 18% | withGST(4000) → 4720 |
| Paise | 1/100 of a rupee; prices kept in paise avoid fractions | 4000 paise = Rs 40 |
Summary
- One
time.Nowreading is noise; a benchmark runs codeb.Ntimes and reports ns/op. Go growsb.N(1, 100, 10000, …) until a run fills-benchtime(1 s). - Plain
go testruns no benchmarks: use-bench. The-2suffix is GOMAXPROCS. - Setup counted makes a benchmark lie:
b.ResetTimer()inb.Nstyle. Impossibly fast means the compiler removed the work. - In Go 1.24+, write
for b.Loop(): setup and cleanup excluded, work never removed, function run once. -benchmem/b.ReportAllocsshow B/op and allocs/op;+=made 1000 allocations vs 1 withGrow, and grows quadratically.b.Runmakes sub-benchmarks per size. Results wobble:-count=10and benchstat (median, ±, p-value,~).- A fuzz test (
FuzzXxx,*testing.F) adds seeds withf.Addand checks a property inf.Fuzz.go test -fuzz=FuzzX -fuzztime=…fuzzes; failures go totestdata/fuzzand everygo testre-runs them. Commit them. - Next episode: concurrency vs parallelism and goroutines.