SyntaxPilot
Courses/Go Programming/Lesson 44
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 44 of 89

The Go scheduler

G-M-P modelgoroutine (G)machine/OS thread (M)processor (P)local and global run queueswork stealingcooperative and asynchronous preemptionblocking system calls and handoffnetwork pollergoroutine stack growth and copyingcontainer-aware GOMAXPROCS (Go 1.25)

These are the notes for Go episode 44. Raju's cloud kitchen has 2 stoves (CPU cores) and 10,000 orders (goroutines). Someone must decide which order cooks on which stove, for how long, and what to do with orders waiting for a delivery. That someone is the scheduler, and you invent it: first the obvious design, one OS thread per goroutine, which crashes with fatal error: thread exhaustion; then one shared queue with one lock, which doesn't scale; then the real design, G, M and P, with local run queues, a global run queue and work stealing. Then you handle a goroutine that never stops (cooperative and asynchronous preemption), goroutines that wait for the kernel (blocking system calls and handoff) or the network (the network poller), how goroutine stacks start small and grow by copying, and container-aware GOMAXPROCS (Go 1.25).

All outputs and errors here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Thread counts, queue lengths and timings change a little on every run and every machine; the shape stays the same.

Facts and sources

  • Dmitry Vyukov, "Scalable Go Scheduler Design Doc", 2 May 2012 (Google Doc). Problems of the scheduler of that time: a single global mutex (Sched.Lock) and centralised state for all goroutine operations; threads handing goroutines to each other; a memory cache per thread; frequent blocking and unblocking of threads around system calls. The proposal: "P represents a resource that is required to execute Go code"; there are exactly GOMAXPROCS Ps; each P has its own list of runnable goroutines; a P with an empty list picks a random other P and steals about half of its runnable goroutines; when a thread enters a system call, its P can be handed to another thread.
  • Go runtime source, src/runtime/proc.go (Go 1.24.7), top comment: "G - goroutine. M - worker thread, or machine. P - processor, a resource that is required to execute Go code. M must have an associated P to execute Go code, however it can be blocked or in a syscall w/o an associated P. Design doc at https://golang.org/s/go11sched." In the same file: the global run queue is checked "once in a while to ensure fairness" (if pp.schedtick%61 == 0); forcePreemptNS = 10 * 1000 * 1000 // 10ms; a P is retaken from a syscall "if it's there for more than 1 sysmon tick (at least 20us)"; maxstacksize = 1000000000 on 64-bit. src/runtime/runtime2.go: each P has runq [256]guintptr (a local run queue of 256). src/runtime/stack.go: a growing stack gets newsize := oldsize * 2.
  • Go 1.14 release notes: "Goroutines are now asynchronously preemptible. As a result, loops without function calls no longer potentially deadlock the scheduler or significantly delay garbage collection." Supported on all platforms except windows/arm, darwin/arm, js/wasm and plan9/*. A consequence: programs "will see more slow system calls fail with EINTR errors".
  • Go 1.4 release notes: stacks are contiguous: "When a stack limit is reached, a new, larger stack is allocated, all active frames for the goroutine are copied there, and any pointers into the stack are updated." And "the default starting size for a goroutine's stack in 1.4 has been reduced from 8192 bytes to 2048 bytes."
  • Go 1.1 release notes: "Due to tighter coupling of the run-time and network libraries, fewer context switches are required on network operations."
  • Go 1.25 release notes (Go 1.25 "arrives in August 2025"), Container-aware GOMAXPROCS: "In prior versions of Go, GOMAXPROCS defaults to the number of logical CPUs available at startup (runtime.NumCPU)." "On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any. If the CPU bandwidth limit is lower than the number of logical CPUs available, GOMAXPROCS will default to the lower limit." "In container runtime systems like Kubernetes, cgroup CPU bandwidth limits generally correspond to the "CPU limit" option. The Go runtime does not consider the "CPU requests" option." "On all OSes, the runtime periodically updates GOMAXPROCS if the number of logical CPUs available or the cgroup CPU bandwidth limit change." "Both of these behaviors are automatically disabled if GOMAXPROCS is set manually via the GOMAXPROCS environment variable or a call to runtime.GOMAXPROCS. They can also be disabled explicitly with the GODEBUG settings containermaxprocs=0 and updatemaxprocs=0, respectively."
  • go doc runtime/debug.SetMaxThreads: "If it attempts to use more than this many, the program crashes. … The initial setting is 10,000 threads."
  • go doc runtime/debug.SetMaxStack: "The initial setting is 1 GB on 64-bit systems, 250 MB on 32-bit systems."
  • Network poller back ends in the runtime source: netpoll_epoll.go (Linux), netpoll_kqueue.go (macOS and BSDs), netpoll_windows.go (I/O completion ports).

The problem: 10,000 orders, 2 stoves

Raju runs a cloud kitchen:

In the kitchen In the computer
2 stoves: only two dishes cook at any moment 2 CPU cores: each runs one piece of code at a time
10,000 orders at lunch 10,000 goroutines (small jobs started with go, episode 43)
an order waiting for the milkman or a phone call a goroutine waiting for I/O (input/output: files, disks, the network)
the person who decides which order goes on which stove the scheduler

A scheduler is the part of a system that decides which job runs on which worker, when, and for how long. Go has its own scheduler inside every Go program, as part of the runtime (the code Go adds to every program to manage goroutines, memory and garbage collection).

Design 1: one cook per order (one OS thread per goroutine)

An OS thread is a worker the operating system (Linux, Windows, macOS) can run on a CPU. The obvious design: give every order its own cook, every goroutine its own thread. runtime.LockOSThread() glues the calling goroutine to its own thread, so we can simulate that design. debug.SetMaxThreads(100) sets the most threads the program may use (normally 10,000).

package main

import (
    "fmt"
    "runtime"
    "runtime/debug"
    "time"
)

// one cook (OS thread) for every order
func cook(order int) {
    runtime.LockOSThread() // this goroutine owns its thread
    time.Sleep(time.Second) // the order takes a second
}

func main() {
    debug.SetMaxThreads(100) // our kitchen has room for 100 cooks
    for order := 1; order <= 1000; order++ {
        go cook(order)
    }
    time.Sleep(2 * time.Second)
    fmt.Println("all 1000 orders done")
}

Real output (the start of it):

runtime: program exceeds 100-thread limit
fatal error: thread exhaustion

runtime stack:
runtime.throw({0x4b6ca6?, 0x0?})
    /usr/local/go1.24.7/src/runtime/panic.go:1101 +0x48 fp=0xc0007e7e08 sp=0xc0007e7dd8 pc=0x467ae8
...

Why it fails. Threads are heavy:

  • The kernel (the core of the operating system, which controls the hardware) reserves a stack for each thread, usually megabytes (often 8 MB of address space on Linux).
  • Switching a CPU from one thread to another is a context switch: the kernel saves one thread's registers and loads another's. It costs far more than switching goroutines inside Go.
  • So a server with 100,000 connections and one thread each would hit the default 10,000-thread limit and crash the same way.

What Go really does

Same limit of 100 threads, ten times more orders, no LockOSThread. pprof.Lookup("threadcreate").Count() returns how many OS threads the program has created so far (runtime/pprof is the profiling package; profiling gets its own episode).

package main

import (
    "fmt"
    "runtime"
    "runtime/debug"
    "runtime/pprof"
    "time"
)

func cook(order int) {
    time.Sleep(time.Second) // the order takes a second
}

func main() {
    debug.SetMaxThreads(100) // the same small kitchen
    for order := 1; order <= 10000; order++ {
        go cook(order)
    }
    fmt.Println("goroutines:", runtime.NumGoroutine())
    time.Sleep(2 * time.Second)
    fmt.Println("all 10000 orders done")
    threads := pprof.Lookup("threadcreate").Count()
    fmt.Println("OS threads created:", threads)
}

Real output:

goroutines: 10001
all 10000 orders done
OS threads created: 4

10,001 goroutines (10,000 cooks plus main) ran on 4 threads. Running many goroutines on a few threads is called M:N scheduling (M goroutines on N threads).

Design 2: one queue, one lock

A few cooks and one order rail at the door. A list of jobs waiting to run is a run queue. Two cooks must not take the same ticket, so the rail gets a lock (a mutex: only one thread may use the protected thing at a time; sync.Mutex is next episode).

With 2 cooks this is fine. With 32 cores, every thread waits in line for the same lock all day: lock contention. Go's scheduler in 2012 worked like this, and Dmitry Vyukov's design document listed the single global mutex as the first problem. His fix: give each stove its own rail and touch the shared one rarely. That is the G-M-P model.

G, M and P (the kitchen)

Kitchen Runtime What it is
an order ticket G (goroutine) the code to run, its own stack, and how far it got
a cook M (machine) an OS thread; only an M actually runs code on a CPU
a stove station with its own rail P (processor) the right to run Go code, plus a local run queue of up to 256 Gs
the rail at the door global run queue shared queue for overflow, protected by a lock

Rules:

  • An M must hold a P to run Go code. An M can exist without a P (blocked in a system call, or idle).
  • There are exactly GOMAXPROCS Ps (by default, the number of CPUs; 2 on this machine). So at most GOMAXPROCS goroutines run Go code at the same moment.
  • A new goroutine (go f()) goes on the local run queue of the current P. No lock is needed, because only that P uses it.
  • If the local queue is full (256), half of it moves to the global run queue.
  • For fairness, a P looks at the global queue every 61st time it schedules, so goroutines there are never forgotten.

Run queues and work stealing (seen for real)

GODEBUG is an environment variable with runtime debugging settings. GODEBUG=schedtrace=100 makes the runtime print a line about the scheduler every 100 ms. It must be set before the program starts, so the program runs itself again as a child process (exec.Command(os.Args[0])) with that setting. os.Args[0] is the path of the running program.

package main

import (
    "fmt"
    "os"
    "os/exec"
    "time"
)

func work() { // some chopping: pure CPU work
    for i, x := 0, 0; i < 30_000_000; i++ {
        x += i
    }
}

func main() {
    if os.Getenv("CHILD") == "" { // re-run ourselves with tracing on
        cmd := exec.Command(os.Args[0])
        cmd.Env = append(os.Environ(), "CHILD=1", "GODEBUG=schedtrace=100")
        cmd.Stdout, cmd.Stderr = os.Stdout, os.Stdout
        cmd.Run()
        return
    }
    for i := 0; i < 1000; i++ {
        go work() // 1000 orders, all created by main
    }
    time.Sleep(450 * time.Millisecond)
    fmt.Println("done")
}

Real output:

SCHED 0ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [2 0]
SCHED 105ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=648 [118 218]
SCHED 210ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=652 [109 210]
SCHED 311ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=655 [100 202]
SCHED 412ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=656 [90 193]
done

How to read a line:

Field Meaning
gomaxprocs=2 2 Ps
idleprocs=0 no P is idle: both stoves are busy
threads=4 4 Ms exist
spinningthreads Ms actively looking for work
idlethreads Ms parked with nothing to do
runqueue=648 goroutines in the global run queue
[118 218] the local run queues of P0 and P1

main created all 1000 goroutines on one P. That P's local queue filled at 256, and half of it moved to the global queue again and again (hundreds there). Yet both local queues have work: the other P got some by work stealing.

Work stealing

When a P has nothing in its local queue, it looks for work in this order: its local queue, the global queue, the network poller (below), and then it picks another P at random and steals about half of that P's local queue. Like a free cook taking half the tickets from a busy station's rail, so no stove stands idle while there is work.

Preemption: the cook who never looks up

Preemption means taking the CPU away from a running goroutine so another can run.

  • Cooperative preemption: the goroutine steps aside by itself at safe moments: function calls (the function's start checks a flag), blocking on channels, locks, sleeps, I/O, or runtime.Gosched() (episode 43). It only works if every goroutine reaches such a moment.
  • A loop with no function calls inside never reaches one. With one P, it would keep the CPU forever.
  • sysmon (system monitor) is a runtime thread that runs without a P, wakes up regularly (starting every 20 microseconds, sleeping longer when idle) and watches the Ps. If a goroutine has run for more than 10 ms, sysmon asks it to step aside.
  • Asynchronous preemption (Go 1.14): the runtime sends the thread a signal (an interrupt the operating system delivers to a thread, like SIGURG on Unix), and the goroutine is stopped right where it is, even in the middle of a loop.

GODEBUG=asyncpreemptoff=1 switches asynchronous preemption off, which lets us see the old behaviour. context.WithTimeout (episode 51) kills the child process after 2 seconds.

package main

import (
    "context"
    "fmt"
    "os"
    "os/exec"
    "runtime"
    "time"
)

func hog() { // a cook who never looks up from the pan
    for i := 0; ; i++ {
    }
}

func main() {
    if os.Getenv("CHILD") == "" {
        for _, debug := range []string{"asyncpreemptoff=1", ""} {
            ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
            cmd := exec.CommandContext(ctx, os.Args[0])
            cmd.Env = append(os.Environ(), "CHILD=1", "GODEBUG="+debug)
            cmd.Stdout = os.Stdout
            fmt.Printf("GODEBUG=%q: ", debug)
            if cmd.Run() != nil {
                fmt.Println("killed after 2s, main never got the CPU back")
            }
            cancel()
        }
        return
    }
    runtime.GOMAXPROCS(1) // one stove
    start := time.Now()
    go hog()
    time.Sleep(10 * time.Millisecond) // main steps away briefly
    fmt.Println("main is back after", time.Since(start).Round(10*time.Millisecond))
}

Real output (five more runs gave 10ms, 20ms, 20ms, 20ms, 10ms):

GODEBUG="asyncpreemptoff=1": killed after 2s, main never got the CPU back
GODEBUG="": main is back after 20ms

Without asynchronous preemption, main never gets the only P back: the program is stuck, exactly how Go behaved before 1.14. With it, sysmon notices hog running too long and the signal pushes it aside in about 10-20 ms.

Blocking system calls and handoff

A system call (syscall) is a request from a program to the kernel: read a file, write to a pipe, open a socket. A blocking system call puts the thread to sleep inside the kernel until it can finish, like a cook waiting at the door for the milkman.

Handoff: when an M blocks in a syscall, its P would sit idle. So sysmon (after at least 20 microseconds) or the scheduler takes the P away and gives it to another M (a spare idle thread, or a new one). When the syscall returns, the goroutine must get a P again before it continues. That is why blocking syscalls can create extra threads, but never stop other goroutines.

A pipe is a one-way channel between two file descriptors (small numbers the kernel uses for open files). syscall.Pipe makes a plain pipe whose reads really block (pipes from os.Pipe are non-blocking and use the network poller instead).

package main

import (
    "fmt"
    "runtime"
    "syscall"
    "time"
)

func main() {
    runtime.GOMAXPROCS(1) // one stove
    var fd [2]int
    syscall.Pipe(fd[:]) // a plain, blocking pipe

    go func() { // waits for the milk delivery
        buf := make([]byte, 16)
        n, _ := syscall.Read(fd[0], buf) // blocks in the kernel
        fmt.Println("delivery arrived:", string(buf[:n]))
    }()
    go func() { // the milkman comes just before 1 second
        time.Sleep(900 * time.Millisecond)
        syscall.Write(fd[1], []byte("milk"))
    }()

    for i := 1; i <= 6; i++ { // meanwhile, keep cooking
        fmt.Println("tick", i, "- still cooking other orders")
        time.Sleep(250 * time.Millisecond)
    }
}

Real output:

tick 1 - still cooking other orders
tick 2 - still cooking other orders
tick 3 - still cooking other orders
tick 4 - still cooking other orders
delivery arrived: milk
tick 5 - still cooking other orders
tick 6 - still cooking other orders

With only one P, main kept ticking while the reader slept in the kernel: the P was handed off.

The network poller

Blocking one thread per network connection is design 1 again. Instead, Go asks the kernel to watch all sockets at once (epoll on Linux, kqueue on macOS/BSD, I/O completion ports on Windows) and to report which ones are ready. This watcher is the network poller (netpoller). A goroutine that reads from a connection with no data is parked (set aside, not running, holding no thread); when the poller reports the socket ready, the goroutine goes back on a run queue. Like a receptionist who watches every phone and calls the kitchen only when someone answers.

A TCP connection is a reliable two-way network connection between two programs; net.Listen waits for connections, net.Dial makes one.

package main

import (
    "fmt"
    "net"
    "runtime/pprof"
    "syscall"
    "time"
)

func threads() int { return pprof.Lookup("threadcreate").Count() }

func main() {
    // 1000 customers on the network, each waiting for a reply
    ln, _ := net.Listen("tcp", "127.0.0.1:0")
    for i := 0; i < 1000; i++ {
        go func() {
            conn, _ := net.Dial("tcp", ln.Addr().String())
            conn.Read(make([]byte, 1)) // waits; nobody replies
        }()
    }
    time.Sleep(time.Second)
    fmt.Println("1000 network waits -> OS threads:", threads())

    // 50 waits on plain blocking pipes (raw system calls)
    for i := 0; i < 50; i++ {
        go func() {
            var fd [2]int
            syscall.Pipe(fd[:])
            syscall.Read(fd[0], make([]byte, 1)) // blocks a thread
        }()
    }
    time.Sleep(time.Second)
    fmt.Println("+50 blocking syscalls -> OS threads:", threads())
}

Real output (three runs: 6/55, 6/55, 7/55):

1000 network waits -> OS threads: 6
+50 blocking syscalls -> OS threads: 55

1000 network waits used a handful of threads; 50 raw blocking syscalls needed about one thread each. Network code in Go scales to huge numbers of connections; raw blocking syscalls each cost a thread. (127.0.0.1:0 means "this machine, any free port".)

Goroutine stacks grow by copying

A stack is the memory where a function keeps its local variables; each call gets a block called a frame. Threads get a big, fixed stack. A goroutine starts with 2 KB (since Go 1.4; 8 KB before). At the start of each function call, a short check sees whether the stack has room; if not, the runtime allocates a stack twice the size, copies every frame there, and updates every pointer into the old stack. This is stack growth by copying (contiguous stacks). Like moving all the plates to a shelf twice as big, in the same order.

Recursion is a function calling itself (episode 10). unsafe.Pointer(&x) turned into a uintptr gives the address of x as a plain number; the unsafe package does low-level things that bypass Go's safety, and we only use it to look.

package main

import (
    "fmt"
    "unsafe"
)

// depth counts plates down a stack of n plates
func depth(n int) int {
    if n == 0 {
        return 0
    }
    return 1 + depth(n-1)
}

func main() {
    x := 42 // lives on main's stack
    before := uintptr(unsafe.Pointer(&x))
    fmt.Println("deep recursion:", depth(1_000_000))
    after := uintptr(unsafe.Pointer(&x))
    fmt.Println("x is still", x)
    fmt.Println("x moved in memory:", before != after)
}

Real output:

deep recursion: 1000000
x is still 42
x moved in memory: true

A million frames do not fit in 2 KB, so the stack grew many times; x moved with it and your code never noticed. This is one reason Go has no pointer arithmetic (episode 11): the runtime must be free to move stacks.

The limit: stack overflow

debug.SetMaxStack sets the largest stack a single goroutine may have (default 1 GB on 64-bit).

package main

import (
    "fmt"
    "runtime/debug"
)

func depth(n int) int {
    if n == 0 {
        return 0
    }
    return 1 + depth(n-1)
}

func main() {
    fmt.Println("100 plates:", depth(100))
    debug.SetMaxStack(1 << 20) // allow only 1 MB of stack
    fmt.Println("1,000,000 plates:", depth(1_000_000))
}

Real output (start):

100 plates: 100
runtime: goroutine stack exceeds 1048576-byte limit
runtime: sp=0xc0001e0388 stack=[0xc0001e0000, 0xc0002e0000]
fatal error: stack overflow

runtime stack:
...

Recursion with no base case hits the default limit (real output, after about 4 seconds of growing to 1 GB):

func forever(n int) int { return forever(n+1) + 1 }
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020160398 stack=[0xc020160000, 0xc040160000]
fatal error: stack overflow

A stack overflow is a fatal error, not a panic: recover (episode 26) cannot catch it. Fix the recursion (add a base case, or use a loop).

GOMAXPROCS and container-aware GOMAXPROCS (Go 1.25)

GOMAXPROCS is the number of Ps: how many goroutines can run Go code at the same moment. runtime.GOMAXPROCS(0) only reads it; runtime.GOMAXPROCS(n) sets it and returns the old value; the environment variable GOMAXPROCS=n sets it at start.

package main

import (
    "fmt"
    "os"
    "os/exec"
    "runtime"
)

func main() {
    fmt.Println("NumCPU:", runtime.NumCPU(), " GOMAXPROCS:", runtime.GOMAXPROCS(0))
    if os.Getenv("GOMAXPROCS") == "" { // run again, set by hand
        cmd := exec.Command(os.Args[0])
        cmd.Env = append(os.Environ(), "GOMAXPROCS=1")
        cmd.Stdout = os.Stdout
        fmt.Print("with GOMAXPROCS=1 -> ")
        cmd.Run()
    }
}

Real output:

NumCPU: 2  GOMAXPROCS: 2
with GOMAXPROCS=1 -> NumCPU: 2  GOMAXPROCS: 1

Containers. A container packages one app with what it needs and runs it on a shared machine with a share of the resources. On Linux the share is enforced by cgroups (control groups). Kubernetes (a widely used system for running containers on many machines) calls the CPU share a CPU limit. The cgroup CPU bandwidth limit says how much CPU time the container may use in each short period.

  • Before Go 1.25: GOMAXPROCS defaulted to runtime.NumCPU(), the CPUs the machine has. On a 64-CPU machine with a 2-CPU limit, Go used 64 Ps; together they used up the 2-CPU allowance early in each period and Linux paused the app until the next one. That pause is CPU throttling, and it shows up as slow, jumpy response times. The fix used to be setting GOMAXPROCS by hand (or a library that reads the cgroup).
  • Go 1.25 (August 2025): on Linux, if the cgroup CPU limit is lower than the CPU count, GOMAXPROCS defaults to the limit; and on all OSes the runtime periodically updates GOMAXPROCS if the CPU count or the limit changes. Kubernetes "CPU requests" are not considered.
  • Both behaviours are switched off if you set GOMAXPROCS yourself (env var or runtime.GOMAXPROCS), or with GODEBUG=containermaxprocs=0 and GODEBUG=updatemaxprocs=0.

(The runs above used Go 1.24.7 on a machine without a CPU limit, so the container rule is described from the release notes, not shown.)

Mini project: kitchen inspector

Write a program that runs itself twice, with GOMAXPROCS=1 and GOMAXPROCS=2. Each run starts 4 CPU-heavy goroutines and 20 goroutines blocked on a raw pipe read, waits for the CPU work, and prints the time taken and the number of OS threads created. Bonus: add GODEBUG=schedtrace=100 and read the queues.

Solution

sync.WaitGroup waits until all the CPU goroutines have called Done; it is explained properly in episode 45.

package main

import (
    "fmt"
    "os"
    "os/exec"
    "runtime"
    "runtime/pprof"
    "sync"
    "syscall"
    "time"
)

// chop is pure CPU work.
func chop() {
    x := 0
    for i := 0; i < 300_000_000; i++ {
        x += i
    }
    _ = x
}

func main() {
    if os.Getenv("CHILD") == "" {
        for _, procs := range []string{"1", "2"} {
            cmd := exec.Command(os.Args[0])
            cmd.Env = append(os.Environ(), "CHILD=1", "GOMAXPROCS="+procs)
            cmd.Stdout = os.Stdout
            cmd.Run()
        }
        return
    }
    start := time.Now()
    var wg sync.WaitGroup // waits for all goroutines (episode 45)
    for i := 0; i < 4; i++ { // 4 CPU-heavy orders
        wg.Add(1)
        go func() { defer wg.Done(); chop() }()
    }
    for i := 0; i < 20; i++ { // 20 orders stuck waiting on a pipe
        go func() {
            var fd [2]int
            syscall.Pipe(fd[:])
            syscall.Read(fd[0], make([]byte, 1))
        }()
    }
    wg.Wait()
    fmt.Printf("GOMAXPROCS=%d: 4 CPU jobs took %v, OS threads: %d\n",
        runtime.GOMAXPROCS(0), time.Since(start).Round(100*time.Millisecond),
        pprof.Lookup("threadcreate").Count())
}

Real output (two runs):

GOMAXPROCS=1: 4 CPU jobs took 600ms, OS threads: 22
GOMAXPROCS=2: 4 CPU jobs took 200ms, OS threads: 23
GOMAXPROCS=1: 4 CPU jobs took 500ms, OS threads: 22
GOMAXPROCS=2: 4 CPU jobs took 200ms, OS threads: 23

With 2 Ps the CPU work runs on both cores at once (about 2-3 times as fast in these runs). The 20 blocked readers need about 20 threads whatever GOMAXPROCS is: GOMAXPROCS limits threads running Go code, not threads blocked in syscalls.

Common mistakes (and the real messages)

Mistake Real message / result Why Fix
one locked thread per job (LockOSThread everywhere) runtime: program exceeds 100-thread limit / fatal error: thread exhaustion threads are heavy and limited plain goroutines
thousands of raw blocking syscalls +50 blocking syscalls -> OS threads: 55 each blocked syscall holds a thread use net/os (poller), limit concurrency
recursion with no base case runtime: goroutine stack exceeds 1000000000-byte limit / fatal error: stack overflow stack limit (1 GB) add a base case, or a loop
recover to survive a stack overflow fatal error: stack overflow (program ends anyway) fatal errors are not panics fix the recursion
GOMAXPROCS = all CPUs inside a container with a lower CPU limit (before Go 1.25) slow, jumpy responses (CPU throttling) more Ps than the CPU limit Go 1.25+, or set GOMAXPROCS
relying on a busy loop never being interrupted (Go 1.14+) the loop is preempted after ~10 ms asynchronous preemption use real synchronisation (episodes 45-50)
expecting goroutines to run in creation order order varies run to run local queues, stealing, preemption never depend on scheduling order

Practice

  1. Predict the output, then run it (Go 1.14 or newer):
package main

import (
    "fmt"
    "runtime"
    "time"
)

func main() {
    runtime.GOMAXPROCS(1)
    go func() {
        for {
        }
    }()
    time.Sleep(time.Millisecond)
    fmt.Println("done")
}
  1. Why does a goroutine blocked in syscall.Read not freeze a program with GOMAXPROCS=1?
  2. Print the old value returned by runtime.GOMAXPROCS(1) and the new value.
  3. (Homework) Run any program of yours with GODEBUG=schedtrace=1000 go run main.go and explain each field.
  4. (Homework) Read the top comment of $(go env GOROOT)/src/runtime/proc.go.
Solution 1

Real output: done. The empty loop has no function calls, but since Go 1.14 sysmon preempts it with a signal, so main gets the only P back.

Solution 2

The M (thread) blocks in the kernel, but its P is handed off to another M, so other goroutines keep running. When the syscall returns, the goroutine waits for a P again.

Solution 3
package main

import (
    "fmt"
    "runtime"
)

func main() {
    old := runtime.GOMAXPROCS(1) // set to 1, returns the previous value
    fmt.Println("previous:", old, "now:", runtime.GOMAXPROCS(0))
}

Real output on a 2-CPU machine: previous: 2 now: 1

Quick quiz

1. One P, a goroutine spinning in for {}, main sleeps 1 ms then prints done. Go 1.24: what happens? (From the video.)

A. Hangs forever · B. Prints done · C. fatal error: all goroutines are asleep · D. panic: deadlock

Answer

B. Asynchronous preemption (Go 1.14) takes the P from the loop. Before Go 1.14 it could hang (A).

2. What do the numbers in runqueue=648 [118 218] mean?

Answer

648 goroutines in the global run queue; 118 and 218 in the local run queues of P0 and P1.

3. How big is a new goroutine's stack, and what happens when it is full?

Answer

2 KB. The runtime allocates a stack twice as big, copies all frames and fixes pointers into it.

4. A P's local run queue is empty and the global queue is empty. What does it do?

Answer

It checks the network poller, then steals about half of the goroutines from a random other P.

Interview questions

  1. Explain the G-M-P model. G = goroutine, M = OS thread, P = processor (permission to run Go code plus a local run queue). An M needs a P to run Go code; there are GOMAXPROCS Ps.
  2. Why not one OS thread per goroutine? Threads have big stacks, kernel-level context switches and a limit (10,000 by default in Go); goroutines start at 2 KB and switch in user space.
  3. What is work stealing? An idle P takes about half of a random busy P's local run queue, balancing load without a global lock.
  4. Why does the global run queue exist, and how is starvation avoided? It holds overflow (local queues hold 256) and goroutines without a P; each P checks it every 61st schedule.
  5. Cooperative vs asynchronous preemption? Cooperative: goroutines yield at safe points (function calls, blocking). Asynchronous (Go 1.14): sysmon signals a goroutine that ran over ~10 ms and stops it anywhere.
  6. What happens when a goroutine makes a blocking system call? Its M blocks in the kernel; the P is handed off to another M; on return the goroutine needs a P again.
  7. What is the network poller? A runtime component using epoll/kqueue/IOCP to wait on many sockets with few threads; waiting goroutines are parked, not blocking threads.
  8. How do goroutine stacks grow? Start at 2 KB; when full, a stack twice the size is allocated and frames copied (contiguous stacks, Go 1.4). Max 1 GB on 64-bit.
  9. What is GOMAXPROCS and what changed in Go 1.25? The number of Ps. Go 1.25 defaults it to the cgroup CPU limit on Linux when that is lower than the CPU count, and updates it periodically.
  10. Is a stack overflow recoverable? No: it is a fatal error, not a panic.

Glossary

Term Meaning Example
CPU core a part of the processor that runs one piece of code at a time this machine has 2
Goroutine a lightweight function running concurrently, started with go go cook(order)
Runtime code Go adds to every program to manage goroutines, memory and GC runtime.NumGoroutine()
Scheduler the part of the runtime deciding which goroutine runs where and when Go's G-M-P scheduler
OS thread a worker the operating system can run on a CPU each M is one
Kernel the core of the operating system that controls the hardware Linux
Context switch the CPU switching from one thread to another, saving and loading state done by the kernel for threads
runtime.LockOSThread glue the calling goroutine to its current thread used in design 1
debug.SetMaxThreads set the most OS threads the program may use (default 10,000) debug.SetMaxThreads(100)
Thread exhaustion fatal error when the thread limit is exceeded fatal error: thread exhaustion
M:N scheduling running M goroutines on N OS threads 10,000 goroutines on 4 threads
pprof.Lookup("threadcreate") profile counting created OS threads .Count() → 4
Run queue a list of jobs waiting to run a ticket rail
Lock (mutex) lets only one thread use something at a time the rail's key
Lock contention many threads waiting for the same lock 32 cooks, one key
G-M-P model Go's scheduler design: goroutines (G) run on threads (M) holding processors (P) gomaxprocs=2
G a goroutine: code, stack and state the order ticket
M (machine) an OS thread that runs code the cook
P (processor) the right to run Go code plus a local run queue; GOMAXPROCS of them the stove station
Local run queue a P's own queue of runnable goroutines, up to 256 [118 218]
Global run queue a shared queue for overflow, behind a lock runqueue=648
Work stealing an idle P taking about half of another P's queue P1 got work from P0
Fairness every waiting goroutine eventually runs global queue checked every 61st schedule
GODEBUG environment variable with runtime debug settings GODEBUG=schedtrace=100
schedtrace GODEBUG setting printing scheduler state periodically SCHED 105ms: ...
Child process a program started by another program exec.Command(os.Args[0])
Preemption taking the CPU away from a running goroutine after ~10 ms
Cooperative preemption goroutines stepping aside at safe points function calls, runtime.Gosched()
Asynchronous preemption stopping a goroutine anywhere using a signal (Go 1.14) the hog loop
Signal an interrupt the OS delivers to a thread SIGURG
sysmon runtime monitor thread running without a P preempts hogs, retakes Ps
asyncpreemptoff=1 GODEBUG setting that disables async preemption the stuck run
Deadlock no goroutine can make progress main never gets the P back
I/O input and output: files, disks, network reading a pipe
System call (syscall) a request from a program to the kernel syscall.Read
Blocking system call a syscall that sleeps in the kernel until it can finish reading an empty pipe
Handoff moving a P from a blocked M to another M ticks continue
Pipe a one-way channel between two file descriptors syscall.Pipe(fd[:])
File descriptor a small number the kernel uses for an open file or socket fd[0]
TCP connection a reliable two-way network connection net.Dial("tcp", addr)
Socket the program's end of a network connection conn
Network poller (netpoller) runtime part that waits on many sockets with epoll/kqueue/IOCP 1000 waits, 6 threads
epoll / kqueue / IOCP kernel ways to wait on many sockets (Linux / macOS, BSD / Windows) netpoll_epoll.go
Parked goroutine a goroutine set aside, not running and holding no thread waiting on conn.Read
Stack memory holding a function's local variables where x lives
Frame one function call's block on the stack one per depth call
Recursion a function calling itself depth(n-1)
Stack growth by copying allocate a 2x stack, copy frames, fix pointers x moved in memory: true
Contiguous stack one block of stack memory, moved when it grows (Go 1.4) replaced segmented stacks
unsafe / uintptr low-level package / an integer that can hold an address uintptr(unsafe.Pointer(&x))
debug.SetMaxStack set the largest stack a goroutine may have (default 1 GB) debug.SetMaxStack(1 << 20)
Stack overflow a stack growing past its limit: fatal error fatal error: stack overflow
Fatal error an error that ends the program and cannot be recovered thread exhaustion, stack overflow
GOMAXPROCS the number of Ps runtime.GOMAXPROCS(0)
runtime.NumCPU logical CPUs the machine has 2
Container a package holding one app, run with a share of a machine a Docker container
cgroup Linux control group: enforces a container's resource share cpu.max
Kubernetes CPU limit the CPU share a container may use limits: cpu: 2
CPU throttling Linux pausing a container that used up its CPU share slow, jumpy responses
Container-aware GOMAXPROCS Go 1.25: default GOMAXPROCS follows the cgroup CPU limit and updates periodically 2 on a 64-CPU machine with limit 2

Summary

  • One OS thread per goroutine runs out of threads (fatal error: thread exhaustion); one global queue with one lock doesn't scale.
  • G-M-P: G = goroutine, M = OS thread, P = processor. An M needs a P to run Go code; there are GOMAXPROCS Ps.
  • Each P has a local run queue (256); overflow goes to the global run queue, checked every 61st schedule.
  • An idle P checks the global queue and the network, then steals about half of a random P's queue.
  • Cooperative preemption at safe points; asynchronous preemption (Go 1.14) by signal after ~10 ms, driven by sysmon.
  • A blocking system call keeps its thread but hands off the P; the network poller parks network waits without holding threads.
  • Goroutine stacks start at 2 KB and grow by copying (2x); the limit is 1 GB, and a stack overflow is fatal.
  • Go 1.25: GOMAXPROCS follows the container's CPU limit and is updated periodically, unless you set it yourself.

Next episode: WaitGroup and Mutex: sync.WaitGroup, WaitGroup.Go (Go 1.25), race conditions, critical sections, sync.Mutex, sync.RWMutex, lock granularity, copying a lock bug and go vet copylocks.