Kubernetes requests and limits — who killed it, OOMKilled or Evicted?

Who killed the pod: OOMKilled or Evicted?

Different killers. OOMKilled means the Linux kernel killed one container; Evicted means kubelet removed the pod from the node. OOMKilled happens because that container went over its own memory limit; Evicted happens because the whole node’s memory or disk dropped below a threshold. So after OOMKilled only the container comes back, on the same node, while an evicted pod ends right there.

Where you actually tell them apart is kubectl describe pod. Both are “it died”, but the record lands in different places. The two outputs below are examples copied to show the format.

$ kubectl describe pod api-7d9c6b5f4-x2k8p
Status:         Running
Containers:
  api:
    State:          Running
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  3

$ kubectl describe pod api-7d9c6b5f4-q8w2n
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: memory. Threshold quantity: 100Mi,
                available: 87232Ki. Container api was using 412Mi, request is 256Mi,
                has larger consumption of memory.

With OOMKilled, the pod stays Running and the record lives only in the container’s Last State. I had memorized exit code 137 as the OOM marker, but 137 is just 128+9: the process got SIGKILL. A container killed after its termination grace period also shows 137. So read Reason along with the number. With Evicted, the pod itself becomes Failed, and the message says which resource ran short and how far that container was over its requests.

requests and limits are used at different times

I had memorized requests as a “guaranteed minimum”. But a guarantee needs someone to keep enforcing it, and I couldn’t think of who looks at requests again after the scheduler places the pod. Tracing it, requests turned out to be mostly a number for choosing where to place a pod, and limits a number for while the container runs.

The scheduler treats a node’s free space as its allocatable minus the sum of requests of pods already on it. It doesn’t look at actual usage. If a node is quiet in practice but its requests are full, like node-1 above, a new pod can’t go there and stays Pending. The more generously you set requests, the more of these quiet-but-full nodes you get.

resources:
  requests:
    cpu: 500m        # used by the scheduler for placement (unrelated to real usage)
    memory: 256Mi
  limits:
    cpu: "1"         # written as the cgroup CPU quota → over it, you wait
    memory: 512Mi    # written as the cgroup memory limit → over it, OOM kill

limits end up as numbers in cgroup files

When kubelet creates a container, it passes the resources values through containerd and runc into Linux cgroup files. I wanted to see that directly, but I had no Kubernetes cluster at hand. Docker writes the same cgroup files through runc, so I passed the same numbers as the YAML above as Docker options and read the files from inside the container.

$ docker run --rm --memory=512m --cpus=1 --cpu-shares=512 lab/catcg
/sys/fs/cgroup/memory/memory.limit_in_bytes   536870912
/sys/fs/cgroup/cpu/cpu.cfs_period_us          100000
/sys/fs/cgroup/cpu/cpu.cfs_quota_us           100000
/sys/fs/cgroup/cpu/cpu.shares                 512

536870912 is 512Mi in bytes. A CPU limit of 1 became “run 100ms (quota) worth every 100ms (period)”. requests.cpu 500m became cpu.shares 512, because 500 × 1024 / 1000. That value sets no ceiling; it’s only the ratio for sharing CPU when things are busy.

The last row surprised me most. I had vaguely assumed requests.memory was reserved somewhere, but no file held such a value. requests.memory isn’t written as a fence in any cgroup file. A pod with requests and no limits can use as much node memory as it likes, and it pays for that in the eviction order when the node fills up. The experiment above ran on a cgroup v1 host; on today’s Kubernetes nodes the names you’ll see are the v2 ones.²

While moving this post I got access to a Kubernetes v1.28.2 node (RKE2, Rocky Linux 8.10), and it was cgroup v1 as well. Among the pods already running there, I read the cgroup files of a container with requests of 25m and 16Mi and limits of 100m and 64Mi: memory.limit_in_bytes was 67108864 (64Mi), cpu.cfs_quota_us was 10000 (10ms every 100ms), and cpu.shares was 25. kubelet was writing them with the same arithmetic as above, and the 16Mi request appeared in no file of the container’s or the pod’s memory cgroup. Running lab/catcg with podman 4.9.4 on the same node gave the same four values as the card above.

// memhog: grabs 10MiB at a time and writes to it (touch) so RSS grows.
package main

import (
  "fmt"
  "os"
  "time"
)

func main() {
  var hold [][]byte
  for i := 1; ; i++ {
    b := make([]byte, 10<<20)
    for j := 0; j < len(b); j += 4096 {
      b[j] = 1 // a page only counts toward RSS once touched
    }
    hold = append(hold, b)
    fmt.Fprintf(os.Stdout, "allocated: %d MiB\n", i*10)
    time.Sleep(100 * time.Millisecond)
  }
}

$ docker run --name hog --memory=64m --memory-swap=64m lab/memhog
allocated: 10 MiB
allocated: 20 MiB
allocated: 30 MiB
allocated: 40 MiB
allocated: 50 MiB
allocated: 60 MiB
$ docker inspect hog --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
OOMKilled=true ExitCode=137

Over the CPU limit, you wait instead of dying

Memory you’ve used can’t be handed back later. CPU time is different: if you can’t use it now, you can use it a moment later. So a CPU limit isn’t a line that kills you when you cross it; it only sets how much you can use in the current window. The 100ms period and 100ms quota above are that amount.

At first I thought each thread got its own share. In fact the whole container splits one share. So even with a limit of 1 CPU, two threads run at once and use up 100ms worth in 50ms. For the next 50ms both stop. A request that hits this pause gets up to almost 50ms slower. In Go, that thread count is GOMAXPROCS, which ties back to the GOMAXPROCS and container CPU limits post.

n := runtime.GOMAXPROCS(0)
before := cpuStat() // reads /sys/fs/cgroup/cpu/cpu.stat
end := time.Now().Add(2 * time.Second)
for i := 0; i < n; i++ {
  wg.Add(1)
  go func() {
    defer wg.Done()
    for time.Now().Before(end) {
      // a CPU-only loop
    }
  }()
}
wg.Wait()
after := cpuStat() // prints the nr_periods and nr_throttled deltas

$ docker run --rm --cpus=1 lab/cpuspin /spin124
go1.24.7 · GOMAXPROCS=2
cgroup: nr_periods 1→21, nr_throttled 0→20
loop ran 2809 million times · process still alive

$ docker run --rm --cpus=1 -e GOMAXPROCS=1 lab/cpuspin /spin124
go1.24.7 · GOMAXPROCS=1
cgroup: nr_periods 1→21, nr_throttled 0→1
loop ran 3729 million times · process still alive

$ docker run --rm lab/cpuspin /spin124
go1.24.7 · GOMAXPROCS=2
cgroup: nr_periods 0→0, nr_throttled 0→0
loop ran 4996 million times · process still alive

Since Go 1.25 the runtime is said to set the default GOMAXPROCS from the cgroup CPU limit, so I bumped go.mod to go 1.25 and built with 1.25.1 too. It was still 2. The runtime docs say that even when it lowers the value for cgroups, it never goes below 2 (unless the machine has fewer than 2 cores). This machine had only 2 cores, so there was nothing to lower.

The 4-vCPU node did leave room to lower it, yet the same program built with Go 1.25.8 still printed GOMAXPROCS 4 under a limit of 1. Giving the container its own cgroup namespace (--cgroupns=private) brought it down to 2, and a Go 1.26.10 build got 2 without any option.

$ podman run --rm --cpus=1 lab/cpuspin /spin125 | grep GOMAXPROCS
go1.25.8 · GOMAXPROCS=4
$ podman run --rm --cpus=1 --cgroupns=private lab/cpuspin /spin125 | grep GOMAXPROCS
go1.25.8 · GOMAXPROCS=2
$ podman run --rm --cpus=1 lab/cpuspin /spin126 | grep GOMAXPROCS
go1.26.10 · GOMAXPROCS=2

The Go 1.25 runtime finds the cgroup directory by appending the path from /proc/self/cgroup to the cgroup mount point. On cgroup v1, Docker and podman share the host’s cgroup namespace by default. /proc/self/cgroup then shows a host-side path like /machine.slice/libpod-….scope, while /sys/fs/cgroup/cpu inside the container already points at that directory itself. The joined path doesn’t exist, so the runtime assumes there’s no limit and uses the core count.

The bug was reported as Go issue #76390 and fixed in Go 1.26; the fix didn’t go into 1.25. My original setup, Docker on cgroup v1, met the same condition. The pods on the RKE2 node above also showed full /kubepods.slice/… paths in /proc/self/cgroup. So on a cgroup v1 node, it’s safest to assume a service built with Go 1.25 gets GOMAXPROCS equal to the node’s core count, whatever its limit.

OOMKilled with a heap smaller than the limit

My study notes asked, “What if a Go service gets OOMKilled with a heap below its limit?” I covered the reason in the GOMEMLIMIT and GOGC post, and this time I recreated it directly with a cgroup limit.

With the default GOGC=100, the Go GC runs the next cycle when the heap reaches twice what survived the previous one. With 35MiB of live data, the next GC target is 70MiB. If the container limit is 64MiB, you cross it before the GC even runs. That’s how a heap below the limit still dies. Setting GOMEMLIMIT makes the runtime pull that target under the limit.

// gcpress: 35MiB of data kept alive + temporary allocations thrown away constantly.
// Prints whether the heap stays under the limit and how many GCs ran.
package main

import (
  "fmt"
  "runtime"
  "runtime/debug"
  "time"
)

var sink []byte

func main() {
  live := make([][]byte, 35)
  for i := range live {
    live[i] = make([]byte, 1<<20)
    for j := 0; j < len(live[i]); j += 4096 {
      live[i][j] = 1
    }
  }
  limit := "none"
  if l := debug.SetMemoryLimit(-1); l < 1<<40 {
    limit = fmt.Sprintf("%d MiB", l>>20)
  }
  fmt.Printf("GOMEMLIMIT=%s\n", limit)

  deadline := time.Now().Add(3 * time.Second)
  var peak uint64
  var ms runtime.MemStats
  next := time.Now()
  for time.Now().Before(deadline) {
    b := make([]byte, 512<<10) // temporary allocation, discarded
    for j := 0; j < len(b); j += 4096 {
      b[j] = 1
    }
    sink = b
    runtime.ReadMemStats(&ms)
    if ms.HeapAlloc > peak {
      peak = ms.HeapAlloc
    }
    if time.Now().After(next) {
      fmt.Printf("  heap %d MiB · runtime got %d MiB from the OS\n", ms.HeapAlloc>>20, ms.Sys>>20)
      next = time.Now().Add(500 * time.Millisecond)
    }
  }
  runtime.KeepAlive(live)
  fmt.Printf("peak heap %d MiB · %d GCs · survived to the end\n", peak>>20, ms.NumGC)
}

$ docker run --memory=64m --memory-swap=64m lab/gcpress
GOMEMLIMIT=none
  heap 35 MiB · runtime got 42 MiB from the OS
(exit 137 · OOMKilled=true)

$ docker run --memory=64m --memory-swap=64m -e GOMEMLIMIT=48MiB lab/gcpress
GOMEMLIMIT=48 MiB
  heap 35 MiB · runtime got 42 MiB from the OS
  heap 37 MiB · runtime got 50 MiB from the OS
  heap 39 MiB · runtime got 50 MiB from the OS
  heap 43 MiB · runtime got 50 MiB from the OS
  heap 37 MiB · runtime got 50 MiB from the OS
  heap 40 MiB · runtime got 50 MiB from the OS
peak heap 45 MiB · 1632 GCs · survived to the end
(exit 0 · OOMKilled=false)

When a node fills up, who leaves first?

The QoS class comes from how requests and limits are set. If every container sets requests = limits for both CPU and memory, it’s Guaranteed; if anything is set at all, Burstable; if nothing is, BestEffort. At first I memorized “when the node fills up, BestEffort leaves first, then Burstable, then Guaranteed.”

Reading the kubelet docs, the order isn’t decided by the class name. kubelet first checks whether a pod is using more than its requests, then PriorityClass, then how far usage is over requests. BestEffort leaves first because its requests are 0, so it’s always over; Guaranteed leaves last because its limit equals its requests, so it can never be over.

So a Burstable pod that stays within its requests lands in the last group. Conversely, a pod with small requests that routinely uses more stands near the front, whatever its class. Matching requests to real usage also sets your place in the eviction order, which I only learned this time.

The last column is the score used when node memory runs out before kubelet can act and the kernel OOM killer moves first. Higher scores die first. kubelet and the kernel are set up to produce an order in the same direction.

An evicted pod isn’t moved

My notes described Evicted as “rescheduled to another node”, and that phrasing made me picture the pod being moved whole to the next node. That’s not what happens. An evicted pod stays where it was in Failed state, and for a Deployment the ReplicaSet creates one more new pod to keep the count. The scheduler then places that new pod. A node short on memory gets a MemoryPressure condition plus a taint that keeps new pods away, so the new pod usually lands on another node.

$ kubectl get pods
NAME                  READY   STATUS      RESTARTS   AGE
api-7d9c6b5f4-q8w2n   0/1     Evicted     0          12m
api-7d9c6b5f4-r5tzm   0/1     Evicted     0          9m
api-7d9c6b5f4-z7hbc   1/1     Running     0          9m

That’s why clusters with frequent evictions keep piling up Evicted pods in the list. A pod started directly without a controller has nothing to recreate it, so it’s simply gone. From now on, when I see “rescheduled”, I read it as “one more new pod was created”.

How to pick requests and limits

By this point it was clear that picking values by feel, like “512Mi of requests should do”, brings every problem above along with it. Too large and you get quiet-but-full nodes; too small and you stand in the first eviction group. The order I settled on while studying: first measure the real usage distribution for a week or two, then set requests at p95–p99 plus 10–20% headroom.

# p99 of container memory use over the last 7 days
quantile_over_time(0.99, container_memory_working_set_bytes{container="api"}[7d])

# p99 of the 5-minute CPU average
quantile_over_time(0.99, rate(container_cpu_usage_seconds_total{container="api"}[5m])[7d:5m])

# throttling ratio: if this stays high, the CPU limit is too tight
rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])

Set the memory limit equal to requests. Memory is the resource that kills you when you go over, so leaving room above requests means you can get evicted first because of someone else when the node fills up. CPU limits are where opinions split. Set them tight, as in the experiment above, and throttling makes latency spike; so unless you share nodes with other teams, some people set them loose or not at all. The catch is that the pod becomes Burstable, since Guaranteed needs CPU requests and limits to match too. As long as usage stays within requests, the pod is in the last eviction group anyway, so that seemed like an acceptable trade.

The three probes do different things on failure

Talk about resources long enough and probes come up. The three names sound alike, but each failure does something different. A failed readiness probe only removes the pod from the Service endpoints; the pod stays alive. A failed liveness probe restarts the container. The startup probe holds off the other two until it succeeds, protecting apps that boot slowly.

The case my notes flagged as most dangerous was putting a DB connection check in liveness. If the DB wobbles briefly, every pod’s liveness fails at once and they all restart together. A brief blip becomes a full outage. External dependencies belong in readiness, where a failure only takes the pod out of traffic.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 24     # 5s × 24 = up to 120s to boot
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3      # out of traffic within 15s
  timeoutSeconds: 3
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3      # restart only after 30s of failures
  timeoutSeconds: 5        # the 1s default can kill healthy pods under load

Choosing values got easier once I rephrased them as “so what happens after how many seconds?” Detection takes about periodSeconds × failureThreshold. Taking a pod out of traffic is cheap, so do it fast; restarting is expensive, so do it slowly. timeoutSeconds defaults to 1 second, so if you leave it, any response slower than a second under load counts as a failure. With a startup probe you also don’t need to guess boot time with initialDelaySeconds.

// health: an HTTP server where liveness and readiness check different things.
package main

import (
  "fmt"
  "log"
  "net/http"
  "sync/atomic"
  "time"
)

var (
  lastBeat atomic.Int64 // last time the work loop ran (UnixNano)
  depOK    atomic.Bool  // state of an external dependency like a DB (a switch here)
)

func worker() {
  for {
    lastBeat.Store(time.Now().UnixNano())
    time.Sleep(time.Second) // a real service would process its queue here
  }
}

// healthz is for liveness. It only checks that this process itself is running.
func healthz(w http.ResponseWriter, _ *http.Request) {
  // Ignore external dependencies; fail only if my loop stalled for over 5s.
  if time.Since(time.Unix(0, lastBeat.Load())) > 5*time.Second {
    http.Error(w, "worker stuck", http.StatusInternalServerError)
    return
  }
  fmt.Fprintln(w, "ok")
}

// ready is for readiness. It checks whether we can take requests now, dependencies included.
func ready(w http.ResponseWriter, _ *http.Request) {
  if !depOK.Load() {
    http.Error(w, "dependency down", http.StatusServiceUnavailable)
    return
  }
  fmt.Fprintln(w, "ready")
}

func main() {
  depOK.Store(true)
  go worker()
  http.HandleFunc("/healthz", healthz)
  http.HandleFunc("/ready", ready)
  // a switch that simulates a dependency outage (example only)
  http.HandleFunc("/debug/dep", func(w http.ResponseWriter, r *http.Request) {
    depOK.Store(r.URL.Query().Get("up") == "1")
    fmt.Fprintf(w, "dependency up=%v\n", depOK.Load())
  })
  log.Fatal(http.ListenAndServe(":8080", nil))
}

$ go run . &
$ curl -s -w ' %{http_code}\n' localhost:8080/healthz   →  ok 200
$ curl -s -w ' %{http_code}\n' localhost:8080/ready     →  ready 200
$ curl -s localhost:8080/debug/dep?up=0                 →  dependency up=false
$ curl -s -w ' %{http_code}\n' localhost:8080/healthz   →  ok 200
$ curl -s -w ' %{http_code}\n' localhost:8080/ready     →  dependency down 503
$ curl -s localhost:8080/debug/dep?up=1                 →  dependency up=true
$ curl -s -w ' %{http_code}\n' localhost:8080/ready     →  ready 200

Summary

• OOMKilled is the kernel killing a container over its limit; Evicted is kubelet removing a pod because of node pressure. Read Reason before exit code 137.
• requests are used when choosing a node, limits while running. requests.memory isn’t written as a fence in any cgroup file.
• Going over a CPU limit means waiting for the next 100ms, not dying. More threads used up the share faster and stopped more often (19–20 of 20 periods versus 0–1 in 2 seconds).
• A Go app can die with a heap below the limit if its GC target goes over the limit. GOMEMLIMIT prevents that at the cost of more GC cycles.
• Eviction order goes by whether a pod is over its requests, then priority, then how far over. Evicted pods aren’t moved; new ones are created.
• Probes do different things on failure. Put external dependencies in readiness only.

This time I only reproduced the cgroup side without a cluster, so I never triggered a real kubelet eviction. Next, on a test cluster, I want to put the node under memory pressure and watch the moment Evicted appears, and see how the eviction order shifts when I change requests.

References

• Kubernetes docs · Resource Management for Pods and Containers
• Kubernetes docs · Node-pressure Eviction
• Kubernetes docs · Pod Quality of Service Classes
• Kubernetes docs · Configure Liveness, Readiness and Startup Probes
• Linux kernel docs · Control Group v2
• Go 1.25 release notes · container-aware GOMAXPROCS
• Go issue #76390 · CPU limit detection not working for non-root cgroup mount