Go GMP and GOMAXPROCS — container CPU limits and goroutines

The runtime is a layer between your code and the kernel

The Go fundamentals series was about what your code means. This series is about how that code actually runs on a machine. Between your code and the CPU and memory sits a layer, the Go runtime, and it has two big jobs: the scheduler, which puts goroutines on threads, and the GC, which cleans up the heap.

The runtime makes decisions based on the whole machine, while the kernel limits each container. Those two views disagree in one place for CPU and one place for memory. This post takes the CPU side, the scheduler. We’ll first see why goroutines are cheap from the structure itself, then look at how that structure can drift inside a container. For a broader tour of the runtime, see How static typing and compilation affect runtime performance.

Do 100,000 goroutines mean 100,000 threads?

No. A goroutine is work to be done; a thread is a resource that does the work. With 100,000 goroutines parked and waiting, there were 4 OS threads (GOMAXPROCS=2, see the lab below). The runtime scheduler connects the two, and a resource called P sits in the middle.

A kitchen with a thousand order tickets doesn’t need a thousand cooks. Tickets are work; cooks are the resource doing it. Splitting Go the same way makes it easier to start.

A P is not a physical CPU core. The workstation analogy describes its role; a P isn’t pinned one-to-one to a real core.

To run Go code, an M needs a P, and the runtime pairs up G, M, and P. With GOMAXPROCS at 2 there are two Ps, so two sets of resources can run Go code at once, and the remaining runnable Gs wait their turn in a queue.

Why not just G and M? If all work piled into one global queue, every thread would contend for it. Go gives each P a local run queue and spreads work with work stealing, where the side running short takes work from another. There’s no need to memorize queue lengths or internal constants; it’s enough to know that P holds the state needed to divide the work.¹

package main

import (
  "flag"
  "fmt"
  "runtime"
  "sync"
  "time"
)

func spin(seed uint64) uint64 {
  x := seed
  for i := 0; i < 40_000_000; i++ {
    x ^= x << 13
    x ^= x >> 7
    x ^= x << 17
  }
  return x
}

func main() {
  wait := flag.Bool("wait", false, "wait 200ms instead of computing")
  flag.Parse()
  fmt.Println("Go:", runtime.Version(), "NumCPU:", runtime.NumCPU(),
    "GOMAXPROCS:", runtime.GOMAXPROCS(0))

  start := time.Now()
  results := make([]uint64, 4)
  var wg sync.WaitGroup
  for i := range results {
    wg.Add(1)
    go func(idx int) {
      defer wg.Done()
      if *wait {
        time.Sleep(200 * time.Millisecond)
        results[idx] = uint64(idx + 1)
        return
      }
      results[idx] = spin(uint64(idx + 1))
    }(i)
  }
  wg.Wait() // each goroutine writes only its own element; reads happen after Wait

  var sum uint64
  for _, v := range results {
    sum ^= v
  }
  fmt.Println("done:", len(results), "checksum:", sum,
    "elapsed:", time.Since(start).Round(time.Millisecond))
}

$ go build -o sched sched/main.go
$ GOMAXPROCS=1 ./sched
done: 4 checksum: 6185594040096594035 elapsed: 321ms
$ GOMAXPROCS=2 ./sched
done: 4 checksum: 6185594040096594035 elapsed: 171ms
$ GOMAXPROCS=1 ./sched -wait
done: 4 checksum: 4 elapsed: 200ms
$ GOMAXPROCS=2 ./sched -wait
done: 4 checksum: 4 elapsed: 201ms

package main

import (
  "fmt"
  "os"
  "regexp"
  "runtime"
  "time"
)

func main() {
  var before, after runtime.MemStats
  runtime.GC()
  runtime.ReadMemStats(&before)

  block := make(chan struct{})
  for i := 0; i < 100_000; i++ {
    go func() { <-block }() // every one of them blocks and waits
  }
  time.Sleep(500 * time.Millisecond)
  runtime.ReadMemStats(&after)

  b, _ := os.ReadFile("/proc/self/status") // OS thread count on Linux
  threads := regexp.MustCompile(`Threads:\s+(\d+)`).FindStringSubmatch(string(b))[1]

  fmt.Println("goroutines :", runtime.NumGoroutine())
  fmt.Printf("StackInuse : %.1f MB (%.2f KB/G)\n",
    float64(after.StackInuse)/1e6, float64(after.StackInuse)/100_000/1024)
  fmt.Println("OS threads :", threads, "/ GOMAXPROCS:", runtime.GOMAXPROCS(0))
  close(block)
}

$ go run gor/main.go
goroutines : 100001
StackInuse : 205.1 MB (2.00 KB/G)
OS threads : 4 / GOMAXPROCS: 2

When a thread blocks, the P moves: handoff and the netpoller

A G waiting on a channel has nothing to do right now, so the runtime parks it and runs another G. The harder case is when the OS thread itself gets stuck, as with a blocking system call or a cgo call.

So GOMAXPROCS=2 doesn’t mean exactly two OS threads in total. The blocked M waits in the kernel without a P, and another M has to take that P over.

Network I/O does one better. For I/O the runtime’s netpoller supports (epoll on Linux, kqueue on macOS), the runtime registers the socket with the netpoller, parks only the G, and wakes it when the socket is ready. Meanwhile the M picks up other Gs. That’s why your code is a single conn.Read() line and yet no thread blocks.

package main

import (
  "fmt"
  "runtime"
  "time"
)

func main() {
  runtime.GOMAXPROCS(1) // limit the runtime to a single P
  start := time.Now()
  var n int
  go func() {
    for { // a compute loop with no function calls
      n++
    }
  }()
  time.Sleep(10 * time.Millisecond)
  fmt.Println("main woke up after:", time.Since(start).Round(time.Millisecond))
}

$ go run preempt/main.go
main woke up after: 10ms
$ GODEBUG=asyncpreemptoff=1 go run preempt/main.go
(no output; stopped by a 5-second timeout)

One container CPU is not a dedicated core

This is where the practical trap starts. A Linux container’s CPU limit usually works as a CPU time budget per period, set by the cgroup. That’s different from reserving one CPU for this process. Running on several CPUs at once adds those times together and drains the budget.

Total CPU use is the same in both cases. The difference is whether the work is spread evenly across the period or bunched at the start, followed by a wait. That’s why average CPU utilization alone is hard to read: the average mixes time spent running with time spent waiting after the budget ran out. Running in bursts and then waiting can show up in tail latency.

That said, don’t blame the CPU quota every time p99 spikes. GC, lock contention, and external I/O need to be checked separately.

If you’re on Kubernetes, keep CPU request and limit apart. The request affects scheduling and how resources are shared under contention; the quota-based cap described here comes from the limit. Apply this to a deployment with no limit and the reading goes wrong.

Even after Go 1.25, the limit and P don’t always match

Up to Go 1.24, the default ignored the container’s CPU quota. P was based on the visible logical CPU count, so a pod with limit: 1 on a 64-core node could end up with execution parallelism far out of line with its allowed CPU time. Go 1.25 added a default on Linux that takes the quota into account.

It’s still wrong to shorten that to “limit 1, so P is 1.” The default now considers the logical CPU count, the CPUs allowed by affinity, and the quota. A fractional quota is rounded up, and as long as the logical CPU count or affinity is 2 or more, the quota won’t push the default P below 2. So a CPU limit of 1 can still give you 2 Ps, and that alone doesn’t mean the setting failed.

Upgrading the toolchain doesn’t change every default either. These are the values to check in a deployment:

On a cgroup v1 node there’s one more catch. When a container shares the host’s cgroup namespace, Go 1.25 builds the wrong path to the quota files, can’t read the limit, and falls back to the logical CPU count. On cgroup v1, Docker and podman start containers that way by default, and so were the pods on the RKE2 v1.28.2 node I checked. The bug was reported as Go issue #76390 and fixed in Go 1.26. Running with a CPU limit of 1 on a 4-vCPU node, Go 1.25.8 got GOMAXPROCS 4 and Go 1.26.10 got 2. The experiment is in the requests and limits post.

So don’t guess the value; read it at run time. If you log runtime.GOMAXPROCS(0) and runtime.NumCPU() at startup, as the lab code does, you can compare the actual values every time you upgrade or edit a manifest.

On older versions you can set it explicitly or use a library such as automaxprocs. Before adding one on a new runtime, though, it’s better to check what your existing deployment settings already pin.

Where do you read CPU throttling?

Throttling is recorded on the cgroup side. On Linux cgroup v2, cpu.max holds the quota, and nr_throttled and throttled_usec in cpu.stat give the number of times and the total time the group stopped after spending its budget. The Go scheduler trace sits alongside, showing how much is piled up in the run queues.

$ GODEBUG=schedtrace=200 GOMAXPROCS=2 ./sched
SCHED 0ms: gomaxprocs=2 idleprocs=1 threads=2 spinningthreads=0 needspinning=0 idlethreads=0 runqueue=0 [0 0]
Go: go1.23.11 NumCPU: 10 GOMAXPROCS: 2
SCHED 212ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=1 [0 1]
SCHED 412ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=1 [0 1]
SCHED 622ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=2 [0 0]
SCHED 829ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=2 [0 0]
SCHED 1036ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=0 [1 1]
done: 4 checksum: 11855647230730954544 elapsed: 1.082s

cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat

These paths work in the common setup where the container has its own cgroup root mounted. With a different mount layout, or from the host, find the real path from /proc/self/cgroup and the mount information.

A cpu.max of 100000 100000 means 100,000μs of CPU time every 100,000μs period, and max 100000 means this file sets no quota cap. nr_throttled and throttled_usec in cpu.stat are cumulative, so look at the difference across the same load window instead of comparing one running total directly with service latency.

Each observation answers a different question.

Before and after changing P, compare throughput, p95 and p99, CPU time, and the growth in throttling together. Raising P just because the run queue is long can burn through the budget faster, and lowering it can cut CPU throughput. My local measurements ran outside a container, so they don’t reproduce throttling.

Wrapping up

I knew goroutines were cheap, but only after drawing these figures did it click that the reasons all sit in one layer, P. Local queues cut contention, a fixed count keeps threads from growing without bound, and because a P can be detached and handed over, one blocked thread doesn’t stop the rest.

That fixed count is exactly what needs checking in a container, because who decides it, and based on what, has changed across Go versions. After upgrading Go or editing a manifest, the first two values I look at are GOMAXPROCS(0) and NumCPU() in the startup log.

References

• Go 1.25 Release Notes · Container-aware GOMAXPROCS: quota awareness, automatic updates, and how to turn them off
• runtime.GOMAXPROCS: how the default is chosen, the floor of 2, and rounding up
• Go issue #76390: on cgroup v1, the quota goes unread when the mount root isn’t /; fixed in Go 1.26
• GODEBUG: containermaxprocs, updatemaxprocs, and their defaults by go.mod version
• Go 1.14 Release Notes · Runtime: asynchronous preemption
• Linux cgroup v2 · CPU Interface Files: cpu.max and cpu.stat
• runtime/proc.go: the G, M, P scheduler implementation and its comments
• runtime/pprof · Profile: the threadcreate profile used to count threads on macOS