Go GOMEMLIMIT and GOGC — OOMKilled with an 800MB heap

The compiler decides between stack and heap

Go has no syntax that says “put this on the heap,” the way malloc does in C. Using new or & doesn’t change that. The compiler decides with escape analysis, and the question it asks is whether the value is still needed after the function returns.

Think of a function as a meeting room. Notes on the whiteboard get wiped when the meeting ends; that’s the stack, and cleaning up costs nothing extra. If a note has to leave the room, it goes on paper into a storeroom, and the GC looks after the storeroom. That’s the heap.

Two things are worth stating up front. Returning the address of a local variable in Go doesn’t produce a dangling pointer. The compiler arranges storage that lives as long as it needs to, so the bug you’d get in C by returning the address of a stack local doesn’t exist in Go.

And “does not escape” doesn’t necessarily mean “one object sits on the stack.” The value may live only in registers, or the computation may simplify until the object disappears. The language guarantees behavior; the exact placement depends on the compiler version and its optimizations. So rather than guessing placement from syntax, we ask the tools.

package main

func main() {}

type Point struct{ X, Y int }

func onStack() Point    { p := Point{1, 2}; return p }  // returns a copy
func byPointer() *Point { p := Point{1, 2}; return &p } // ① returns a pointer
func toIface() any      { p := Point{1, 2}; return p }  // ② returns an interface
func captured() func() int {
  n := 0
  return func() int { n++; return n } // ③ closure capture
}
func unknownSize(n int) int {
  buf := make([]byte, n) // ④ size unknown at compile time
  return copy(buf, "hello")
}

$ go build -gcflags='-m' . 2>&1 | grep -v inlin
# escape
./main.go:8:27: moved to heap: p                  ← ① p in byPointer
./main.go:9:52: p escapes to heap                 ← ② boxing in toIface
./main.go:11:2: moved to heap: n                  ← ③ n in captured
./main.go:12:9: func literal escapes to heap      ← ③ the closure itself
./main.go:15:13: make([]byte, n) escapes to heap  ← ④
$ GOTOOLCHAIN=go1.27.0 go build -gcflags='-m' . 2>&1 | grep make
./main.go:15:13: make([]byte, n) does not escape

package vmake

import "testing"

//go:noinline
func use(n int) int {
  buf := make([]byte, n)
  return copy(buf, "hello")
}

func BenchmarkMake32(b *testing.B) {
  for i := 0; i < b.N; i++ {
    use(32)
  }
}

func BenchmarkMake33(b *testing.B) {
  for i := 0; i < b.N; i++ {
    use(33)
  }
}

$ GOTOOLCHAIN=go1.24.7 go test -run '^$' -bench . -benchmem -benchtime=100ms
BenchmarkMake32-10    13048869     9.401 ns/op    32 B/op    1 allocs/op
BenchmarkMake33-10    12464780     9.748 ns/op    48 B/op    1 allocs/op
$ GOTOOLCHAIN=go1.27.0 go test -run '^$' -bench . -benchmem -benchtime=100ms
BenchmarkMake32-10    55769622     1.802 ns/op     0 B/op    0 allocs/op
BenchmarkMake33-10    15216597     7.681 ns/op    48 B/op    1 allocs/op

package allocation

type Pair struct {
  A, B int64
}

func NewPair(n int64) *Pair {
  return &Pair{A: n, B: n + 1}
}

func SumLocal(n int64) int64 {
  p := NewPair(n)
  return p.A + p.B
}

var Published *Pair

func Publish(n int64) {
  Published = NewPair(n)
}

package allocation

import "testing"

var sink int64

func BenchmarkLocal(b *testing.B) {
  for i := 0; i < b.N; i++ {
    sink += SumLocal(int64(i))
  }
}

func BenchmarkPublished(b *testing.B) {
  for i := 0; i < b.N; i++ {
    Publish(int64(i))
  }
}

$ go build -gcflags='-m=2' allocation.go 2>&1 | grep -E 'inline NewPair|inlining call|escapes to heap$|does not escape'
./allocation.go:7:6: can inline NewPair with cost 9 as: func(int64) *Pair { return &Pair{...} }
./allocation.go:12:14: inlining call to NewPair
./allocation.go:19:21: inlining call to NewPair
./allocation.go:8:9: &Pair{...} escapes to heap
./allocation.go:12:14: &Pair{...} does not escape
./allocation.go:19:21: &Pair{...} escapes to heap

$ go test -run '^$' -bench . -benchmem -benchtime=200ms -count=3 allocation.go allocation_test.go
BenchmarkLocal-2         396647101    0.6096 ns/op     0 B/op    0 allocs/op
BenchmarkLocal-2         396067387    0.6077 ns/op     0 B/op    0 allocs/op
BenchmarkLocal-2         401408246    0.6848 ns/op     0 B/op    0 allocs/op
BenchmarkPublished-2      11122455   18.36   ns/op    16 B/op    1 allocs/op
BenchmarkPublished-2      14999492   16.15   ns/op    16 B/op    1 allocs/op
BenchmarkPublished-2      13671566   16.87   ns/op    16 B/op    1 allocs/op

$ go test -run '^$' -bench . -benchmem -benchtime=200ms -gcflags='-l' allocation.go allocation_test.go
BenchmarkLocal-2          13994080   16.01 ns/op    16 B/op    1 allocs/op
BenchmarkPublished-2      13299541   15.88 ns/op    16 B/op    1 allocs/op

Is removing every escape faster?

Not necessarily. Removing one heap allocation can buy you extra copying or shared state instead. Once you can read the diagnostics, it’s tempting to eliminate every escape; first, look at these two costs.

// copies the whole Config on every call
func Load() Config

// shared across requests
var buf [64 << 10]byte

A fixed order keeps this straight. Correct lifetimes and API meaning come first, finding the slow path with a profiler comes next, and only then do you ask whether that path’s allocations can shrink. After a change, compare correctness, latency, and allocations together. The win has to survive real input distributions, not just a short benchmark.

The GC runs alongside the program and never moves objects

Go’s GC is a tricolor mark-and-sweep collector. It runs concurrently with the program and doesn’t move objects. Unlike the JVM’s collectors, it isn’t generational and doesn’t compact. The design trades fragmentation for simplicity and short pauses.

The three colors name how far the traversal has gotten. Objects don’t actually carry a color field. The GC reclaims objects that can’t be reached from the roots, which include goroutine stacks and global variables. Nothing is freed for being old or looking unused, so no setting can shrink the data that’s still live.

The price of marking alongside the program is the write barrier. To avoid losing a live object when a pointer changes mid-mark, each pointer write does a little extra work. There are also two short stop-the-world (STW) pauses, so a concurrent GC still has pauses and costs.

Go can mark precisely because it’s statically typed. The compiler knows which fields are pointers, so the GC never has to guess whether some 8 bytes are a pointer or an integer. The marking itself keeps evolving: the Green Tea GC arrived as an experiment in Go 1.25 and became the default in Go 1.26. It scans small objects in batches per memory block for better cache locality, and the release notes expect a 10–40% cut in GC overhead for programs that lean on the GC heavily.¹

Cost depends on how much data is live, how pointers connect it, and how fast new memory is allocated. A big byte array and a dense web of pointers of the same size can cost very different amounts to scan. No setting changes the algorithm. What you can change is when the GC starts, and that’s what GOGC and GOMEMLIMIT control.

GOGC sets the goal as a ratio of the live heap

GOGC decides when the GC runs. The default is 100, and the next heap goal comes from this formula.

Target heap = Live heap + (Live heap + GC roots) × GOGC / 100

When the roots are small, GOGC=100 means roughly twice the surviving heap. With 200MB live the goal is around 400MB; with 600MB live it’s around 1.2GB. A larger value allows a bigger heap in exchange for fewer collections.

A common misreading of the formula goes like this.

The GC starts the moment the heap reaches the goal.

The runtime starts marking earlier, so that the cycle finishes near the goal, and it adjusts the pace as it goes. Other inputs, such as the minimum heap goal and the memory limit, come into play too. Let’s see it in real output.

package main

import "fmt"

var live [][]byte // data that stays live
var sink []byte   // data dropped right away

func main() {
  for i := 0; i < 200; i++ { // about 200MiB live
    live = append(live, make([]byte, 1<<20))
  }
  for i := 0; i < 100_000; i++ { // keep making 64KiB of garbage
    sink = make([]byte, 64<<10)
  }
  fmt.Println("live MiB:", len(live))
}

$ go build -o saw main.go
$ GODEBUG=gctrace=1 GOMAXPROCS=2 ./saw
gc 1 @0.000s 2%: … 4->4->4 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 2 @0.001s 3%: … 9->9->9 MB, 9 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 3 @0.001s 4%: … 18->18->18 MB, 18 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 4 @0.001s 4%: … 36->36->36 MB, 36 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 5 @0.001s 5%: … 71->72->72 MB, 72 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 6 @0.001s 6%: … 141->151->151 MB, 144 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 7 @0.002s 5%: … 285->286->200 MB, 302 MB goal, 0 MB stacks, 0 MB globals, 2 P
…
gc 36 @0.099s 0%: … 398->402->203 MB, 402 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 37 @0.102s 0%: … 402->404->201 MB, 407 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 38 @0.104s 0%: … 398->399->201 MB, 403 MB goal, 0 MB stacks, 0 MB globals, 2 P
live MiB: 200

Why does an 800MB heap get OOMKilled?

Because the GC goal can be set above the container limit. With 600MB live and GOGC=100, the goal is near 1.2GB; with a 1GB limit, memory crosses the limit before the GC runs on its schedule. On top of that, what the kernel counts is RSS, which is larger than the heap. A container memory limit is an absolute rule (kill this process past 1GB), and the kernel has no idea when the GC plans to run.

Besides the heap, RSS includes goroutine stacks, runtime metadata, and pages not yet returned to the OS. So the heap number says there’s room left, and the process dies anyway. The proportions in the second card are illustrative; real ones vary by service.

When a GC cycle shrinks the heap, the memory the OS sees doesn’t drop by the same amount right away. The runtime may keep the space for reuse, and returning it to the OS happens separately. If you change a setting and expect RSS to fall immediately, you’ll misread what you see.

I didn’t reproduce a container OOMKill for this post. Every experiment below ran outside a container and reads numbers the runtime reports.

GOMEMLIMIT adds an absolute ceiling

GOMEMLIMIT is a soft memory limit added in Go 1.19. As memory approaches it, the GC runs early even if the GOGC goal is still far away. It puts an absolute rule next to the ratio, in the same units the kernel uses.

“Runtime-managed memory” means Sys minus HeapReleased in runtime.MemStats terms. It covers more than the heap, including other memory the runtime maps and manages. It leaves out the Go binary itself, memory allocated directly by cgo, and memory the OS holds on the process’s behalf. That’s why using the container limit as-is isn’t enough. Near the limit, the runtime also returns memory to the OS more aggressively, not just runs the GC sooner.

$ GOMEMLIMIT=300MiB GODEBUG=gctrace=1 GOMAXPROCS=2 ./saw 2>&1 | tail -4
gc 80 @0.078s 1%: … 275->276->201 MB, 287 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 81 @0.079s 1%: … 275->275->200 MB, 287 MB goal, 0 MB stacks, 0 MB globals, 2 P
gc 82 @0.080s 1%: … 275->285->210 MB, 287 MB goal, 0 MB stacks, 0 MB globals, 2 P
live MiB: 200

package main

import (
  "fmt"
  "os"
  "runtime"
  "time"
)

var recent [16][]byte

func main() {
  fmt.Println("Go:", runtime.Version(), "GOGC:", os.Getenv("GOGC"), "GOMEMLIMIT:", os.Getenv("GOMEMLIMIT"))
  base := make([]byte, 8<<20)
  for i := range base {
    base[i] = 1
  }
  runtime.GC() // align the baseline; not counted in the GC delta below
  var before, after runtime.MemStats
  runtime.ReadMemStats(&before)

  start := time.Now()
  var checksum uint64
  var peak uint64
  for i := 0; i < 4096; i++ {
    data := make([]byte, 64<<10)
    data[0] = byte(i)
    data[len(data)-1] = byte(i >> 8)
    recent[i%len(recent)] = data
    checksum += uint64(data[0]) + uint64(data[len(data)-1])
    if i%64 == 0 {
      var m runtime.MemStats
      runtime.ReadMemStats(&m)
      if m.HeapAlloc > peak {
        peak = m.HeapAlloc
      }
    }
  }
  elapsed := time.Since(start)
  runtime.ReadMemStats(&after)
  runtime.KeepAlive(base)

  const mib = 1 << 20
  fmt.Printf("TotalAlloc delta: %.1f MiB\n", float64(after.TotalAlloc-before.TotalAlloc)/mib)
  fmt.Printf("NumGC delta: %d\n", after.NumGC-before.NumGC)
  fmt.Printf("sampled peak heap: %.1f MiB\n", float64(peak)/mib)
  fmt.Printf("NextGC at exit: %.1f MiB\n", float64(after.NextGC)/mib)
  fmt.Printf("Sys-HeapReleased at exit: %.1f MiB\n", float64(after.Sys-after.HeapReleased)/mib)
  fmt.Println("checksum:", checksum, "elapsed:", elapsed)
}

go build -o gc-demo main.go
GOGC=50  GOMEMLIMIT=off    ./gc-demo
GOGC=200 GOMEMLIMIT=off    ./gc-demo
GOGC=200 GOMEMLIMIT=24MiB  ./gc-demo
GOGC=off GOMEMLIMIT=24MiB  ./gc-demo
GOGC=off GOMEMLIMIT=off    ./gc-demo

When reading these numbers, start with what each one counts.

HeapAlloc still includes dead objects that haven’t been collected, and NextGC is a goal, not a ceiling. With that in mind, here’s how I’d set things up.

// env: GOMEMLIMIT=900MiB  ← around 90% of the limit; RSS includes memory the runtime can't see
// env: GOGC=100           ← keep the ratio rule; with little live data it triggers first

// or set it in code
debug.SetMemoryLimit(900 << 20)

// GOGC=off + GOMEMLIMIT means "don't collect until the limit."
// Fine for batch jobs that can fill memory, wrong for latency-sensitive services.

The GC guide recommends leaving an extra 5–10% of headroom for memory the runtime doesn’t know about, in fixed-memory environments like containers. With heavy cgo use or large file mappings, leave more.

Remember that the limit is soft. If live data alone exceeds it, no amount of collecting brings memory back under. The runtime caps GC CPU use at roughly 50% so the GC can’t take over the whole CPU, which means the process keeps running above the limit until it gets OOMKilled anyway. GOMEMLIMIT protects you when garbage piles up, and it can’t help when the live data itself is too large. The guide says the same: for a program already close to its environment’s limit, raise the limit or lower GOGC instead of leaning on the memory limit.

Wrap-up

• The compiler picks stack or heap. The same &Pair{} changed with inlining, and the same make changed with the toolchain, so measure with -gcflags=-m and -benchmem instead of counting syntax.
• Go’s GC is a concurrent tricolor mark-and-sweep that never moves objects. Short pauses come at the price of write barriers.
• GOGC is a ratio, so the goal grows as live data grows. Marking starts before the heap reaches the goal.
• The container kills on absolute RSS. That’s how an 800MB heap can die under a 1GB limit.
• Setting GOMEMLIMIT below the limit pulls the goal under it, and GC cycles go up accordingly. Even with GOGC=off, a limit keeps the GC running.

When I studied the GC, I thought tricolor marking and write barriers were the heart of it. But where the service actually died, the problem was when the GC started, not the algorithm, and that rule and the kernel’s rule didn’t even use the same units.

While writing this up I also noticed it has the same shape as GOMAXPROCS in the previous post. The runtime decides from the host’s cores and the heap ratio; the container limits by CPU quota and RSS. One setting reconciles each side: GOMAXPROCS for CPU and GOMEMLIMIT for memory.

Through all five settings, the 256MiB of total allocation didn’t move. Shrinking that number is the code’s job, and next I’ll look at reusing buffers with sync.Pool.

References

• A Guide to the Go Garbage Collector · GOGC: the heap goal formula
• A Guide to the Go Garbage Collector · Memory limit: the soft limit, the ~50% GC CPU cap, the 5–10% headroom advice
• runtime/debug.SetMemoryLimit: what the limit includes and excludes, and that it holds even with GOGC=off
• runtime · Environment Variables: GOGC, GOMEMLIMIT, and the gctrace format
• runtime.MemStats: HeapAlloc, NextGC, Sys, HeapReleased
• Go FAQ · Stack or heap: the compiler decides where values live
• Go 1.19 Release Notes · Runtime: the soft memory limit
• Go 1.25 Release Notes · Faster slices: more slice backing stores on the stack
• Go 1.26 Release Notes · New garbage collector: Green Tea GC on by default
• Proposal: Eliminate STW stack re-scanning: the Go 1.8 hybrid write barrier
• Linux cgroup v2 · Memory Interface Files: memory.max and OOM