Go goroutine leaks and pprof — channels, mutexes, and exit paths

Channels and mutexes solve different problems

“It’s Go, so use channels” is only half right. The standard library uses mutexes all over the place. What separates the two is whether the data moves. A channel is a conveyor belt that hands a value to the next goroutine; a mutex is a door lock that lets several goroutines use the same spot one at a time.

Don’t read the table too literally. RWMutex isn’t always faster just because reads dominate, and sync.Map isn’t the default replacement for a plain map. Decide what state you’re protecting and how it’s accessed first, and if performance matters, measure under real load.

Write the fourth row, the shared map, with channels and you can see why “always channels” goes wrong.

type getReq struct {
  key   string
  reply chan int
}

go func() { // the goroutine that owns the cache
  for r := range reqs {
    r.reply <- cache[r.key]
  }
}()

reply := make(chan int)
reqs <- getReq{"k", reply}
v := <-reply

mu.RLock()
v := cache["k"] // many readers at once
mu.RUnlock()

The left side makes data that never moves pretend to move. It also raises a new question: what happens when the goroutine that owns the cache dies? The standard library itself uses mutexes throughout net/http and inside sync.Map¹.

package main

import "fmt"

func main() {
  packets := make(chan []byte, 1)

  data := []byte("go")
  packets <- data // send the slice over the channel
  data[0] = 'n'   // the sender can still write to the same array
  fmt.Println("shared:", string(<-packets))

  copied := append([]byte(nil), data...)
  packets <- copied // this time, send a copy
  data[0] = 'x'
  fmt.Println("copied:", string(<-packets))
  fmt.Println("original:", string(data))
}

$ go run ./chown
shared: no
copied: no
original: xo

So in practice we pick one of three and say so. Ownership transfer means the sender never touches the data again. Copying means sending only the part that’s needed. Sharing means touching it only under the same lock. If you choose ownership transfer, that’s a promise in your code and API; the compiler won’t enforce it. When the elements are structs holding pointers or maps, the copy has to go deeper too.

close means sending is over

A send on an unbuffered channel proceeds only when a receiver arrives. With a buffer, it drops the value in without waiting while there’s room and waits again once the buffer is full. Raising the capacity adds waiting room; it doesn’t make the receiver faster.

Making a sender wait while the receiver is slow is called backpressure. But if you start a new goroutine for each job and let it wait inside, the number of waiting goroutines can keep growing. First work out whether you’re limiting queue length, running jobs, or goroutines.

close(ch) doesn’t clear the buffer or stop the receiver. It signals that nothing more will be sent on this channel. Values already inside come out in order, and only after they’re drained do you get the element type’s zero value and false. For a chan int, receive with v, ok := <-ch to tell that apart from a real 0.

Put three states (nil, open, closed) against three operations (send, receive, close) and you get this.

package main

import (
  "fmt"
  "time"
)

func try(name string, f func()) {
  defer func() {
    if r := recover(); r != nil {
      fmt.Printf("%-17s → panic: %v\n", name, r)
    }
  }()
  f()
}

func main() {
  c := make(chan int, 1)
  c <- 7
  close(c)
  v, ok := <-c
  fmt.Println("closed recv       →", v, ok) // value left in the buffer
  v, ok = <-c
  fmt.Println("closed recv again →", v, ok) // zero value, no blocking

  try("closed send", func() { c <- 1 })
  try("double close", func() { close(c) })

  var n chan int // nil channel
  try("nil close", func() { close(n) })
  select {
  case <-n:
  case <-time.After(200 * time.Millisecond):
    fmt.Println("nil recv          → still blocked after 200ms (forever)")
  }
}

$ go run ./chrules
closed recv       → 7 true
closed recv again → 0 false
closed send       → panic: send on closed channel
double close      → panic: close of closed channel
nil close         → panic: close of nil channel
nil recv          → still blocked after 200ms (forever)

The three panics came from sending on a closed channel, closing it again, and closing a nil channel. All three are someone closing out of turn, which is why the sender owns the close. With several senders, none of them can close on its own, so one coordinator that can see every send finish does it. If a receiver closes the channel on its own, a remaining sender’s next send panics, and any sender already waiting to send panics the moment it closes.

You also don’t close a channel to free memory. Close it only when the receiver needs to know sending is over.

Blocking forever on a nil channel is sometimes deliberate. To switch off a select case for a while, set that channel variable to nil. The case won’t be chosen until you put a channel back.

A buffered channel counts concurrent jobs

Here’s another place channels fit well. If you want at most 10 calls to an external API at a time, a channel with a buffer of 10 is 10 slots. Put a value in to take a slot and take it out to give the slot back.

package main

import (
  "fmt"
  "sync"
  "sync/atomic"
  "time"
)

func main() {
  var running, peak atomic.Int64 // (for measuring) current and peak concurrency
  sem := make(chan struct{}, 10) // 10 slots
  var wg sync.WaitGroup
  start := time.Now()
  for i := 0; i < 100; i++ {
    wg.Add(1)
    sem <- struct{}{} // take a slot; waits here while all 10 are taken
    go func() {
      defer wg.Done()
      defer func() { <-sem }() // give the slot back
      n := running.Add(1)
      for p := peak.Load(); n > p && !peak.CompareAndSwap(p, n); p = peak.Load() {
      }
      time.Sleep(50 * time.Millisecond)
      running.Add(-1)
    }()
  }
  wg.Wait()
  fmt.Printf("100 jobs, peak concurrency: %d, total %v\n", peak.Load(), time.Since(start).Round(10*time.Millisecond))
}

$ go run ./sem
100 jobs, peak concurrency: 10, total 510ms

Know exactly what this code limits. Because sem <- struct{}{} runs outside the goroutine, it caps both running jobs and goroutines at 10. Move the slot grab inside the goroutine and running jobs stay at 10, but you get one waiting goroutine per input. A worker pool, which starts N goroutines up front and feeds them jobs over a channel, is a third option; its goroutine count doesn’t grow with the input. Which one to use depends on what you want to limit.

The GC doesn’t clean up goroutines

Let’s draw a line first. A server waiting for requests or a worker pool waiting for its next job is normal. If a goroutine waits within its expected lifetime and can stop when needed, being long-lived doesn’t make it a leak.

The trouble is work that should already be over but is left with no one to proceed with and no way to finish. If the caller has given up on the result while the worker keeps waiting at ch <- v, that goroutine outlives the request. Its stack, and everything the stack points to, stays with it.

A block with no exit path is a leak. Goroutine stacks start small², but one left behind per request keeps piling up.

A leak isn’t a deadlock either. Only part of the program is stuck while other goroutines keep serving requests, so you never see all goroutines are asleep - deadlock!. The CPU profile stays quiet too, because blocked goroutines aren’t computing anything. If the CPU is idle but memory keeps growing, suspect this first.

package main

import (
  "context"
  "fmt"
  "net/http"
  _ "net/http/pprof" // registers the /debug/pprof/* endpoints
  "runtime"
  "sync"
  "time"
)

func fetch() int { time.Sleep(300 * time.Millisecond); return 42 } // slow backend

// Leaks: once the caller gives up, the send blocks forever
func handleLeaky(ctx context.Context) (int, error) {
  ch := make(chan int)
  go func() { ch <- fetch() }() // stuck here forever once the receiver is gone
  select {
  case v := <-ch:
    return v, nil
  case <-ctx.Done(): // the caller leaves at 100ms
    return 0, ctx.Err()
  }
}

// Fixed: a buffer of 1 lets the send finish with no receiver
func handleFixed(ctx context.Context) (int, error) {
  ch := make(chan int, 1)
  go func() { ch <- fetch() }()
  select {
  case v := <-ch:
    return v, nil
  case <-ctx.Done():
    return 0, ctx.Err()
  }
}

// Sends 1000 requests with a 100ms timeout at once and counts goroutines a second later.
func load(name string, handle func(context.Context) (int, error)) {
  before := runtime.NumGoroutine()
  var wg sync.WaitGroup
  for i := 0; i < 1000; i++ {
    wg.Add(1)
    go func() {
      defer wg.Done()
      ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
      defer cancel()
      handle(ctx)
    }()
  }
  wg.Wait()
  time.Sleep(time.Second) // time for the slow backend to finish
  fmt.Printf("%-5s goroutines after 1000 requests: %d → %d\n", name, before, runtime.NumGoroutine())
}

func main() {
  go http.ListenAndServe("localhost:6060", nil) // internal network only in production
  time.Sleep(100 * time.Millisecond)
  load("leaky", handleLeaky)
  load("fixed", handleFixed)
  select {} // keep running so we can read the profile
}

$ go run ./timeout
leaky goroutines after 1000 requests: 2 → 1002
fixed goroutines after 1000 requests: 1002 → 1002

There are two fixes. Give the channel a buffer of 1, as handleFixed does, and the send finishes even with no receiver, so the block never happens. If a buffer won’t work or several values are sent, wrap the send inside the goroutine in a select with a ctx.Done() case.

The buffer fix comes with conditions. A capacity of 1 removes the wait only when each job sends exactly one value and the computation always finishes. A producer that keeps emitting values will fill any finite buffer eventually, so a buffer can’t stand in for an exit path.

Don’t close the result channel to wake a waiting sender, either. Sending on a closed channel panics. Keep the stop signal on a separate channel and close that one once.

Cancellation has no priority. When a select finds both the cancel case and a job ready, Go picks one at random. Long jobs sometimes check ctx.Err() again right after receiving, but cancellation can still land right after that check. Neither approach guarantees that no work happens after the moment of cancellation.

If you use a WaitGroup, call wg.Add(1) before starting the goroutine and put defer wg.Done() on its first line³. That way Done runs whether the goroutine finishes normally or is cancelled.

Leaks come in four shapes, all with the same cause.

In code review, when I see a line that starts a goroutine, the first thing I ask is how that goroutine ends. Whoever starts a goroutine is responsible for ending it, and the sender does the close.

Why doesn’t a select with ctx.Done() finish?

Because the expression that computes the value to send is evaluated before the select starts. With case out <- compute():, the select only starts looking at its cases once compute() has returned. Moving the computation to the line above doesn’t change that, so the code that has to know about cancellation is the computation itself.

select {
case out <- compute(): // there's a cancel case, so this looks safe
case <-ctx.Done():
}

The Go spec says that on entering a select, the channels and the values to send in every case are evaluated once, in source order. We can run three variants side by side.

package main

import (
  "context"
  "fmt"
  "time"
)

// A computation that knows nothing about cancellation.
func compute() int {
  time.Sleep(700 * time.Millisecond)
  return 42
}

// A computation that checks for cancellation as it goes.
func computeCtx(ctx context.Context) (int, error) {
  for i := 0; i < 70; i++ {
    select {
    case <-ctx.Done():
      return 0, ctx.Err()
    case <-time.After(10 * time.Millisecond):
    }
  }
  return 42, nil
}

// ① Has a cancel case, but the send value is evaluated on entering select.
func inSelect(ctx context.Context, out chan<- int) time.Duration {
  start := time.Now()
  select {
  case out <- compute():
  case <-ctx.Done():
  }
  return time.Since(start)
}

// ② Move the computation out of the select.
func beforeSelect(ctx context.Context, out chan<- int) time.Duration {
  start := time.Now()
  v := compute()
  select {
  case out <- v:
  case <-ctx.Done():
  }
  return time.Since(start)
}

// ③ The computation itself honors cancellation.
func ctxAware(ctx context.Context, out chan<- int) time.Duration {
  start := time.Now()
  v, err := computeCtx(ctx)
  if err != nil {
    return time.Since(start)
  }
  select {
  case out <- v:
  case <-ctx.Done():
  }
  return time.Since(start)
}

func run(name string, f func(context.Context, chan<- int) time.Duration) {
  out := make(chan int) // nobody receives
  ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
  defer cancel()
  d := f(ctx, out)
  fmt.Printf("%-24s returned after %v\n", name, d.Round(10*time.Millisecond))
}

func main() {
  fmt.Println("all three cancel at 100ms")
  run("① compute inside select", inSelect)
  run("② compute moved up", beforeSelect)
  run("③ compute checks ctx", ctxAware)
}

$ go run ./sel
all three cancel at 100ms
① compute inside select  returned after 700ms
② compute moved up       returned after 700ms
③ compute checks ctx     returned after 100ms

In practice, pass ctx straight into network APIs that accept it, and check ctx.Err() on each iteration of a long computation. Wrapping a call that can’t be cancelled in a new goroutine so the caller can return early doesn’t fix anything. The wrapped call stays behind, which is the same leak we just looked at.

So when reading code, the question isn’t whether there’s a select. It’s which operations before it or inside it can block.

Finding leaks with the goroutine profile

When a service’s memory creeps up and you suspect a goroutine leak, take two goroutine profiles some time apart and compare them. The stack that only ever grows is the culprit, and the profile tells you which line those goroutines stopped on.

The timeout program above already imports net/http/pprof, so we can read it right away. In production, expose that port only on an internal network. I built it with -trimpath so the paths print short.

$ curl -s 'localhost:6060/debug/pprof/goroutine?debug=1' | head -3
goroutine profile: total 1004
1000 @ 0x100089da8 0x100023628 0x100023258 0x10023a6e0 0x100091b64
#  0x10023a6df  main.handleLeaky.func1+0x3f  examples/timeout/main.go:18

1000 @ means 1000 goroutines share this stack, and main.go:18 below it is the go func() { ch <- fetch() }() line. In production, take two profiles 5 to 10 minutes apart and look only at the stacks that grew. Taking two separates leaks from goroutines that are waiting normally, such as HTTP connection handlers. Normal waiting holds steady; a leak keeps climbing.

curl -s 'localhost:6060/debug/pprof/goroutine?debug=1' > a.txt
sleep 600
curl -s 'localhost:6060/debug/pprof/goroutine?debug=1' > b.txt
diff <(grep -E '^[0-9]+ @' a.txt) <(grep -E '^[0-9]+ @' b.txt)

You can practice locally without a server, too. This program makes three sends with no receiver, twice, and saves a snapshot with runtime/pprof each time.

package main

import (
  "context"
  "flag"
  "fmt"
  "os"
  "path/filepath"
  "runtime"
  "runtime/pprof"
  "time"
)

// A send whose receiver is gone. There is no way out.
func leakySend(out chan<- int, ready chan<- struct{}) {
  ready <- struct{}{}
  out <- 42
}

// The same work with a cancel path and a done signal.
func fixedSend(ctx context.Context, out chan<- int, ready chan<- struct{}, done chan<- struct{}) {
  defer close(done)
  ready <- struct{}{}
  select {
  case out <- 42:
  case <-ctx.Done():
  }
}

func snapshot(dir, name string) {
  os.MkdirAll(dir, 0o755)
  p := pprof.Lookup("goroutine")
  f, _ := os.Create(filepath.Join(dir, name+".pprof"))
  p.WriteTo(f, 0) // for analysis tools
  f.Close()
  t, _ := os.Create(filepath.Join(dir, name+".txt"))
  p.WriteTo(t, 2) // stack dump for humans
  t.Close()
}

func main() {
  mode := flag.String("mode", "leak", "leak or fixed")
  flag.Parse()
  dir := filepath.Join("profiles", *mode)

  snapshot(dir, "before")
  fmt.Printf("before          : %d\n", runtime.NumGoroutine())

  for round := 1; round <= 2; round++ {
    if *mode == "leak" {
      for i := 0; i < 3; i++ {
        out := make(chan int)
        ready := make(chan struct{})
        go leakySend(out, ready)
        <-ready
      }
    } else {
      for i := 0; i < 3; i++ {
        ctx, cancel := context.WithCancel(context.Background())
        out := make(chan int)
        ready := make(chan struct{})
        done := make(chan struct{})
        go fixedSend(ctx, out, ready, done)
        <-ready
        cancel()
        <-done // wait for the exit, not just the cancel request
      }
    }
    time.Sleep(30 * time.Millisecond)
    snapshot(dir, fmt.Sprintf("after-%d", round))
    fmt.Printf("after round %d   : %d\n", round, runtime.NumGoroutine())
  }
}

ready is a start signal for the experiment. main always receives it, so the signal send never gets stuck. Receiving it doesn’t guarantee the worker has already stopped on the next line, though, so the program waits 30ms before each snapshot. That’s slack to delay the observation, not a guarantee that anything has finished.

$ go build -trimpath -o leakdemo . && ./leakdemo -mode=leak
before          : 1
after round 1   : 4
after round 2   : 7
$ ./leakdemo -mode=fixed
before          : 1
after round 1   : 1
after round 2   : 1

Now look only at the difference between the before and after profiles. -base subtracts the earlier profile so only what grew afterward remains.

$ go tool pprof -top -nodecount=8 -base profiles/leak/before.pprof profiles/leak/after-2.pprof | grep -v -e '^File' -e '^Time'
Type: goroutine
Showing nodes accounting for 6, 75.00% of 8 total
      flat  flat%   sum%        cum   cum%
         6 75.00% 75.00%          6 75.00%  runtime.gopark
         0     0% 75.00%          6 75.00%  main.leakySend
         0     0% 75.00%          6 75.00%  runtime.chansend
         0     0% 75.00%          6 75.00%  runtime.chansend1

Two lines are enough. flat counts samples at that location itself, and cum includes everything called beneath it. A flat of 6 on runtime.gopark means the six new goroutines sit at a runtime wait point, and a cum of 6 on main.leakySend means all six went through our send function. Seeing runtime.gopark at the top doesn’t mean the runtime needs fixing.

What we compared here is the goroutine profile, not the CPU profile, so a sample is a goroutine, not a unit of time. To see how long goroutines spent waiting on synchronization, use the block profile.

Don’t feed the text dump into analysis tools. The program saves both a debug=0 .pprof file and a debug=2 .txt file. The comparison takes the .pprof; the text is for reading wait states and creation sites by eye.

$ grep -c 'main.leakySend' profiles/leak/after-2.txt
6
$ grep -m1 -B1 -A4 'main.leakySend' profiles/leak/after-2.txt
goroutine 34 [chan send]:
main.leakySend(...)
  leakdemo/main.go:17
created by main.main in goroutine 1
  leakdemo/main.go:54 +0x464

[chan send] is the wait state, main.go:17 is the out <- 42 line, and main.go:54 is where the goroutine was started. Goroutine IDs and line numbers change from run to run and edit to edit, so there’s nothing to memorize. In fixed mode, the dump didn’t contain a single main.fixedSend.

The goroutineleak profile in Go 1.27

The goroutineleak profile, an experiment in Go 1.26, became a regular feature in Go 1.27. The GC finds leaks through reachability: if no running goroutine can reach the channel or lock a blocked goroutine is waiting on, that goroutine can never wake up. I built the same timeout program with Go 1.27.0 and opened this endpoint as well.

$ curl -s 'localhost:6060/debug/pprof/goroutineleak?debug=1' | head -3
goroutineleak profile: total 1001
1000 @ 0x1030586d8 0x102fe98a4 0x102fe94b8 0x103204010 0x10305f6f4
#  0x10320400f  main.handleLeaky.func1+0x3f  examples/timeout/main.go:18

The goroutine profile listed all 1004, normal waiters included; this one picked out the 1000 leaks directly. The remaining entry in the total showed up with an empty stack, and I don’t know why yet. The GC doesn’t reclaim the leaked goroutines; it only finds and reports them. The release notes also say it can miss leaks on channels reachable through global variables or a running goroutine’s locals, so comparing two profiles is still worth doing.

Where to start depends on the symptom.

After the fix, test the exit signal

Once a leak is fixed, leave a test so it doesn’t come back. I didn’t make a short Sleep or a matching total goroutine count the success condition, because both fluctuate for unrelated reasons. The test checks for the event of the done channel closing instead.

package main

import (
  "context"
  "testing"
  "time"
)

// With no receiver: does cancel close the done signal?
func TestCancelEndsSender(t *testing.T) {
  ctx, cancel := context.WithCancel(context.Background())
  out := make(chan int)
  ready := make(chan struct{})
  done := make(chan struct{})

  go fixedSend(ctx, out, ready, done)
  <-ready
  cancel()

  select {
  case <-done:
  case <-time.After(2 * time.Second): // a failure limit, not a wait
    t.Fatal("sender goroutine still running after cancel")
  }
}

// With a receiver: is done closed after delivery?
func TestDeliveryEndsSender(t *testing.T) {
  ctx, cancel := context.WithCancel(context.Background())
  defer cancel()
  out := make(chan int)
  ready := make(chan struct{})
  done := make(chan struct{})

  go fixedSend(ctx, out, ready, done)
  <-ready
  if v := <-out; v != 42 {
    t.Fatalf("got %d, want 42", v)
  }
  select {
  case <-done:
  case <-time.After(2 * time.Second):
    t.Fatal("sender goroutine still running after delivery")
  }
}

$ go test -race -timeout=10s -run EndsSender -v .
=== RUN   TestCancelEndsSender
--- PASS: TestCancelEndsSender (0.00s)
=== RUN   TestDeliveryEndsSender
--- PASS: TestDeliveryEndsSender (0.00s)
PASS
ok    leakdemo  1.692s

$ go test -race -timeout=10s -run EndsSender .
--- FAIL: TestCancelEndsSender (2.00s)
    main_test.go:23: sender goroutine still running after cancel
FAIL
FAIL  brk  2.285s
FAIL

Note that passing the race check wasn’t taken as proof there’s no leak. The exit signal and the profiles were checked separately. Each check covers a different range.

The last row is where real systems most often go wrong. If the computation is short and has no side effects, “drop it on cancel” is a fine contract. But if a save or a payment already ran, cancellation won’t undo it. Retries, idempotency, and transactions need their own design.

Wrap-up

• Use a channel when you hand data off and won’t look at it again, and a mutex when several goroutines share the same spot. A slice sent over a channel still shares its array, so decide in code whether it’s an ownership transfer, a copy, or shared under a lock.
• close marks the end of sending. Buffered values still come out after it, and all three panics came from closing out of turn. The sender closes, once.
• A buffered channel counts slots. Running 100 jobs 10 at a time peaked at 10 and took 510ms.
• The GC doesn’t clean up blocked goroutines. A block with no exit path is a leak, and leaks come in four shapes.
• Even with a cancel case in a select, the value to send is computed first. The computation has to check for cancellation itself.
• To find a leak, diff two goroutine profiles, and on Go 1.27 or later check the goroutineleak profile too. Profiles tell you where things stopped; why the other side never arrives takes reading the code.

Closing the series

After four posts, three things were left that the runtime won’t do for me: matching GOMAXPROCS to the container’s CPU limit, matching GOMEMLIMIT to its memory limit, and giving every goroutine a way out. Since Go 1.25 the runtime takes the CPU quota into account for GOMAXPROCS, but the limit and P still didn’t always match; GOMEMLIMIT is still something you set yourself in Go 1.27. As for ways out, Go 1.27’s goroutineleak profile now finds some of the missing ones, but whether a goroutine waiting forever is intended or a bug is still up to the code.

The sync.Pool post in the middle belongs here too. A pool cuts allocations but doesn’t tell you what you must not touch after putting something back. Ownership and exit paths were both promises the code had to keep.

References

• Effective Go · Share by communicating: handing ownership over channels
• The Go Memory Model · Channel communication: the synchronization between a send and a receive
• The Go Programming Language Specification · Select statements: channels and send values are evaluated on entering select
• The Go Programming Language Specification · Close: sending on or re-closing a closed channel panics; receives get leftovers, then the zero value
• runtime/pprof: the goroutine profile and its debug levels
• net/http/pprof: the /debug/pprof/* endpoints
• Go 1.26 Release Notes · Experimental goroutine leak profile: how the GC finds leaks through reachability, with an example
• Go 1.27 Release Notes · Goroutine leak profile: now a regular feature; may miss leaks reachable through globals
• Go 1.25 Release Notes · sync: WaitGroup.Go
• Data Race Detector · How To Use: it only finds races that happen at runtime