Kubernetes on‑premises ErrImagePull — pulling an image that's already there

Why does ErrImagePull happen at an on-premises customer site?

Because kubelet decided the image had to be pulled. It decides that under two conditions: the policy is Always, or the policy is IfNotPresent and the image doesn’t appear to be on the node. At a customer site with no route to a registry, a pull fails the moment it starts, so the real question becomes “why did kubelet decide to pull?”

Lining up the cases we handled against those conditions gives three groups. The tag was latest, so the policy became Always. The name or tag the manifest asked for differed from the image we brought in. Or the image was loaded, but not anywhere Kubernetes looks.

The customer site has no registry

Most of our on-premises customer sites restrict outbound traffic, and you can only bring in images and files approved ahead of time. So we export the images built in-house as tar files, get them approved along with checksums, put them on the customer VM’s local disk, and import them straight into containerd. Kubernetes is a single RKE2 node installed from offline install files. Because there’s no registry in between, image names keep their in-house registry paths, and the manifests’ image: values never need editing.

In this setup images skip the registry and go straight into containerd, so everything depends on whether kubelet recognizes the image as present. If it does, the pod starts. If it doesn’t, or never checks, kubelet tries to reach a registry and gets stuck.

The two conditions under which kubelet pulls

How the policy gets set is spelled out in the Kubernetes source. If a manifest leaves imagePullPolicy empty, the API server fills in a default: Always if the tag is latest, IfNotPresent otherwise. An image with no tag at all is read as latest, so it also gets Always.

func SetDefaults_Container(obj *v1.Container) {
  if obj.ImagePullPolicy == "" {
    // Ignore error and assume it has been validated elsewhere
    _, tag, _, _ := parsers.ParseImageName(obj.Image)

    // Check image tag
    if tag == "latest" {
      obj.ImagePullPolicy = v1.PullAlways
    } else {
      obj.ImagePullPolicy = v1.PullIfNotPresent
    }
  }

The function that combines the policy with whether the image is on the node lives in kubelet. Unless the policy is Never, Always pulls every time and IfNotPresent pulls only when the image is missing.¹

// shouldPullImage returns whether we should pull an image according to
// the presence and pull policy of the image.
func shouldPullImage(container *v1.Container, imagePresent bool) bool {
  if container.ImagePullPolicy == v1.PullNever {
    return false
  }

  if container.ImagePullPolicy == v1.PullAlways ||
    (container.ImagePullPolicy == v1.PullIfNotPresent && (!imagePresent)) {
    return true
  }

  return false
}

The easy part to miss is what “present” means. kubelet doesn’t search the disk; it asks CRI (the Container Runtime Interface) by name and tag. So an image sitting somewhere on disk counts as missing if CRI doesn’t know about it. The third case came from exactly this gap.

ErrImagePull and ImagePullBackOff are the same failure

The two states look like different problems, but they’re one pull failure named by timing. Right after a failed attempt it’s ErrImagePull; while waiting for the next attempt it’s ImagePullBackOff. As of kubelet v1.28, the wait starts at 10 seconds, doubles each time, and stops at 300 seconds.

package main

import (
  "fmt"
  "time"
)

// A simplified model of the kubelet image manager.
// After a failed pull, backoff starts at 10s, doubles, and stops at 300s.
// A sync during backoff returns ImagePullBackOff without pulling.
const (
  base = 10 * time.Second
  max  = 300 * time.Second
  sync = 10 * time.Second // assume a 10s pod sync period
)

func main() {
  var nextTry time.Duration
  backoff := time.Duration(0)
  last := ""
  for now := time.Duration(0); now <= 12*time.Minute; now += sync {
    state := "ImagePullBackOff"
    if now >= nextTry {
      state = "ErrImagePull" // no registry, so every pull fails
      if backoff == 0 {
        backoff = base
      } else if backoff *= 2; backoff > max {
        backoff = max
      }
      nextTry = now + backoff
    }
    if state == "ErrImagePull" || state != last {
      fmt.Printf("%5s  %-17s next pull in %v\n", fmtDur(now), state, nextTry-now)
    }
    last = state
  }
}

func fmtDur(d time.Duration) string {
  return fmt.Sprintf("%d:%02d", int(d.Minutes()), int(d.Seconds())%60)
}

$ go run ./pullsim
 0:00  ErrImagePull      next pull in 10s
 0:10  ErrImagePull      next pull in 20s
 0:20  ImagePullBackOff  next pull in 10s
 0:30  ErrImagePull      next pull in 40s
 0:40  ImagePullBackOff  next pull in 30s
 1:10  ErrImagePull      next pull in 1m20s
 1:20  ImagePullBackOff  next pull in 1m10s
 2:30  ErrImagePull      next pull in 2m40s
 2:40  ImagePullBackOff  next pull in 2m30s
 5:10  ErrImagePull      next pull in 5m0s
 5:20  ImagePullBackOff  next pull in 4m50s
10:10  ErrImagePull      next pull in 5m0s
10:20  ImagePullBackOff  next pull in 4m50s

A latest tag, or a tag that doesn’t match

Case 1: a latest tag pulls even when the image is there

The first case is the simplest. The package included a latest-tagged image like busybox:latest, and the manifest set no policy. By the default rule the policy becomes Always, and kubelet goes to the registry to check even though the image is sitting on the node. In-house there’s a registry, so nothing goes wrong; it only breaks at the customer site.

Here’s the same thing reproduced at the CRI level. Ask for an already-imported image again with crictl pull, and the runtime goes to the registry for the manifest regardless of the local copy, failing at name resolution. That’s what a kubelet with an Always policy does every time.

$ crictl pull registry.example.internal/demo-api:v1.4.2   # already imported
E1021 02:06:22.793327  remote_image.go:180] "PullImage from image service failed"
  err="rpc error: code = Unknown desc = failed to pull and unpack image
  \"registry.example.internal/demo-api:v1.4.2\": failed to resolve reference ...:
  failed to do request: Head \"https://registry.example.internal/v2/demo-api/manifests/v1.4.2\":
  dial tcp: lookup registry.example.internal on 8.8.8.8:53: no such host"
(exit 1, 0.02s)

It failed in 0.02 seconds because name resolution failed immediately in the repro environment. At a customer site where DNS or a firewall answers slowly, each attempt waits for a timeout instead. Either way, the result is ErrImagePull.

While moving this post I got access to an RKE2 v1.28.2 node (containerd 1.7.3-k3s1), so I tried it again there. With the image imported, three runs of crictl pull gave the same error as above apart from the DNS server address, and each failed within 0.11–0.12 seconds, crictl startup included.

Case 2: if the name or tag differs at all, the image is missing

In the second case kubelet couldn’t find the image. CRI treats two images as the same only if name and tag match exactly. If the manifest asks for demo-worker:v1.4.3 and the tar we brought holds v1.4.2, the node doesn’t have the image, however similar the contents, and IfNotPresent moves on to a pull.

Names and tags drifted apart in a few ways. The most common was updating the manifest to a new version while loading the old image tar. Images also went missing on the packaging side. Our image export script only collects images from running pods, so if an optional component was scaled down at export time, its image wasn’t in the tar list at all. And crictl rmi --prune, which we ran to clean up unused images before shipping, also deleted images belonging to stopped pods. Neither caused any symptom in-house; only that one pod failed at the customer site.

On premises, a missing image is one whose name and tag aren’t in CRI’s list. Whether the file is on disk doesn’t matter. The only way to prevent this ahead of time is to compare the list the manifests need with the list inside the tars before shipping.

Case 3: imported for sure, but CRI doesn’t know

The third case was the hardest to track down. During an on-site install, we loaded the image tar with ctr images import, the command finished without errors, and the pod still tried to pull. crictl images didn’t list the image either. The cause was the path ctr used to load the image. Without the --local option, import goes through containerd’s transfer service, and images loaded that way weren’t registered as CRI-managed images.² The fix was to import again with --local --platform linux/amd64.

$ ctr version   # daemon
Version:  v1.7.22

$ ctr images import demo-api_v1.4.2.tar            # ctr 1.7.22
unpacking registry.example.internal/demo-api:v1.4.2 (sha256:1563d1f2424d3e44d2187fcd6966353431ea85eca3a1247cd04b900f38dd4fd3)...done

$ ctr images import demo-worker_v1.4.2.tar         # ctr 2.0.6
registry.example.internal/demo worker:v1  saved
Importing  elapsed: 0.0 s  total:   0.0 B  (0.0 B/s)

$ crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
registry.example.internal/demo-api   v1.4.2              719e08e7d70f4       723B

$ ctr images ls   # labels only
REF LABELS
registry.example.internal/demo-api:v1.4.2 io.cri-containerd.image=managed
registry.example.internal/demo-worker:v1.4.2 -

$ ctr images check            # stored state
REF                                          UNPACKED
registry.example.internal/demo-worker:v1.4.2 true

$ crictl images               # what CRI sees
IMAGE                                TAG       IMAGE ID        SIZE
registry.example.internal/demo-api   v1.4.2    719e08e7d70f4   723B

# after restarting containerd
$ crictl images
IMAGE                                   TAG       IMAGE ID        SIZE
registry.example.internal/demo-api      v1.4.2    719e08e7d70f4   723B
registry.example.internal/demo-worker   v1.4.2    719e08e7d70f4   723B

The repro taught me one more thing. Here the stored state was UNPACKED true, and after a containerd restart CRI reloaded the image and it appeared in the list. Our install logs showed UNPACKED as false in the same situation, so back then we deleted and re-imported. That means the outcome can depend on the runtime and snapshotter setup. So we fixed an order of checks: first confirm the import printed unpacking ...done, then compare the number of tars with the number of images in crictl images. Only if the counts differ do we decide between a restart and a re-import.

I repeated this experiment on the same RKE2 node too. I pointed the node’s own ctr 1.7.3 and a separately downloaded ctr 2.0.6 at the node’s containerd 1.7.3-k3s1, and built fresh tars holding a minimal one-layer image. The card below leaves out the socket address and the -n k8s.io option.

$ ctr images import demo-api_v1.4.2.tar            # ctr 1.7.3
unpacking registry.example.internal/demo-api:v1.4.2 (sha256:3370bff1e1639e3c69d511ecb1f0a0e47a24c2f265486aed7ad5287c3a1e44b5)...done

$ ctr images import demo-worker_v1.4.2.tar         # ctr 2.0.6
registry.example.internal/demo worker:v1  saved
Importing  elapsed: 0.2 s  total:   0.0 B  (0.0 B/s)

$ crictl images | grep example
registry.example.internal/demo-api      v1.4.2    7acac72ffcb1e   10.8kB

$ ctr images import --local --platform linux/amd64 demo-worker_v1.4.2.tar   # ctr 2.0.6
unpacking registry.example.internal/demo-worker:v1.4.2 (sha256:3370bff1e1639e3c69d511ecb1f0a0e47a24c2f265486aed7ad5287c3a1e44b5)...done

$ crictl images | grep example
registry.example.internal/demo-api      v1.4.2    7acac72ffcb1e   10.8kB
registry.example.internal/demo-worker   v1.4.2    7acac72ffcb1e   10.8kB

The result matched the earlier repro. The image imported with ctr 2.0.6 showed UNPACKED true in ctr images check, yet it had no label and was missing from the crictl list. Since this containerd serves the node’s Kubernetes, I didn’t try a restart. Instead I re-imported with --local --platform linux/amd64 without deleting anything, and the label appeared and crictl listed the image right away.

What I check when a call comes in

The order of checks is almost always the same. First the events in kubectl describe pod, then kubectl logs to see whether the container started, and if that’s not enough, down to the Kubernetes logs on the node.

The last few lines of the Events in describe hold most of the answer: which image kubelet tried to pull and which error it failed with. Below, the runtime error from the repro above is shown in kubelet’s event format.

$ kubectl describe pod demo-api-7c9f6b5d8-x2k4p
Events:
  Type     Reason   Age   From     Message
  ----     ------   ----  ----     -------
  Normal   Pulling  14s   kubelet  Pulling image "registry.example.internal/busybox:latest"
  Warning  Failed   14s   kubelet  Failed to pull image "registry.example.internal/busybox:latest":
           rpc error: code = Unknown desc = failed to pull and unpack image ...:
           dial tcp: lookup registry.example.internal ...: no such host
  Warning  Failed   14s   kubelet  Error: ErrImagePull
  Normal   BackOff  2s    kubelet  Back-off pulling image "registry.example.internal/busybox:latest"
  Warning  Failed   2s    kubelet  Error: ImagePullBackOff

The thing to look at is the image name. A latest tag means case 1; a tag that differs from the shipped list means case 2. If the name and tag match the shipped list exactly and kubelet still pulls, suspect case 3 and go down to the node. With an image problem, logs comes back empty or says the container is still waiting to start. An empty log is itself a sign that the application isn’t the problem.

journalctl -u rke2-server --since "30 min ago" | grep -iE "pull|image"
tail -n 200 /var/lib/rancher/rke2/agent/logs/kubelet.log
tail -n 200 /var/lib/rancher/rke2/agent/containerd/containerd.log

# the image list CRI actually sees
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
crictl images | grep <image name>
ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images ls | grep <image name>

The last two lines go together because of case 3. If an image is in ctr images ls but not in crictl images, it was loaded but never registered with CRI.

Two things we changed before releases

After these three cases, the development side changed two things. We check the image list before every release, and we set imagePullPolicy explicitly in the manifests. The list check stops missing images; the fixed policy stops pulls of images that are already there.

We chose to pin IfNotPresent. Dev and customer sites can share one manifest, and it’s the simplest thing to read on site. The cost, as the table says, is a tagging rule. If anyone pushes a rebuilt image under the same tag, IfNotPresent keeps running the old one, so we set a separate tagging rule to prevent that.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-api
spec:
  template:
    spec:
      initContainers:
        - name: wait-db
          image: registry.example.internal/busybox:latest
      containers:
        - name: api
          image: registry.example.internal/demo-api:v1.4.2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: registry.example.internal/demo-worker:v1.4.3
          imagePullPolicy: IfNotPresent

package main

import (
  "archive/tar"
  "bufio"
  "encoding/json"
  "fmt"
  "os"
  "path/filepath"
  "regexp"
  "sort"
  "strings"
)

// Before a release, compare "images the manifests use" with "images in the tars we carry in".
var (
  reImage  = regexp.MustCompile(`^\s*-?\s*image:\s*"?([^"\s]+)"?`)
  rePolicy = regexp.MustCompile(`^\s*imagePullPolicy:\s*(\w+)`)
)

type use struct{ file, image, policy string }

func main() {
  uses := scanManifests(os.Args[1])
  packed := scanTars(os.Args[2])
  problems := 0
  for _, u := range uses {
    var notes []string
    if !packed[u.image] {
      notes = append(notes, "not in the tars")
    }
    policy := u.policy
    if policy == "" {
      policy = defaultPolicy(u.image) + "(default)"
    }
    if strings.HasPrefix(policy, "Always") {
      notes = append(notes, "Always → always pulls on-prem")
    }
    if u.policy == "" {
      notes = append(notes, "no policy set")
    }
    mark := "OK "
    if len(notes) > 0 {
      mark, problems = "NG ", problems+1
    }
    fmt.Printf("%s %-22s %-46s %-22s %s\n", mark, u.file, u.image, policy, strings.Join(notes, ", "))
  }
  used := map[string]bool{}
  for _, u := range uses {
    used[u.image] = true
  }
  for img := range packed {
    if !used[img] {
      fmt.Printf("-- tar no manifest uses: %s\n", img)
    }
  }
  fmt.Printf("\n%d problems\n", problems)
  if problems > 0 {
    os.Exit(1)
  }
}

func defaultPolicy(image string) string {
  ref := image[strings.LastIndex(image, "/")+1:]
  if !strings.Contains(ref, ":") || strings.HasSuffix(ref, ":latest") {
    return "Always" // Kubernetes default rule: no tag or latest → Always
  }
  return "IfNotPresent"
}

func scanManifests(dir string) []use {
  files, _ := filepath.Glob(filepath.Join(dir, "*.yaml"))
  sort.Strings(files)
  var out []use
  for _, f := range files {
    fh, _ := os.Open(f)
    sc := bufio.NewScanner(fh)
    for sc.Scan() {
      if m := reImage.FindStringSubmatch(sc.Text()); m != nil {
        out = append(out, use{file: filepath.Base(f), image: m[1]})
      } else if m := rePolicy.FindStringSubmatch(sc.Text()); m != nil && len(out) > 0 {
        out[len(out)-1].policy = m[1] // treat it as the previous container's policy (simplified)
      }
    }
    fh.Close()
  }
  return out
}

// Read image names from index.json in an OCI tar.
func scanTars(dir string) map[string]bool {
  files, _ := filepath.Glob(filepath.Join(dir, "*.tar"))
  names := map[string]bool{}
  for _, f := range files {
    fh, _ := os.Open(f)
    tr := tar.NewReader(fh)
    for {
      h, err := tr.Next()
      if err != nil {
        break
      }
      if h.Name != "index.json" {
        continue
      }
      var idx struct {
        Manifests []struct {
          Annotations map[string]string `json:"annotations"`
        } `json:"manifests"`
      }
      json.NewDecoder(tr).Decode(&idx)
      for _, m := range idx.Manifests {
        if n := m.Annotations["io.containerd.image.name"]; n != "" {
          names[n] = true
        }
      }
    }
    fh.Close()
  }
  return names
}

$ go run ./imagecheck testdata/manifests testdata/images
NG  deployment_api.yaml    registry.example.internal/busybox:latest       Always(default)        Always → always pulls on-prem, no policy set
NG  deployment_api.yaml    registry.example.internal/demo-api:v1.4.2      IfNotPresent(default)  no policy set
NG  deployment_worker.yaml registry.example.internal/demo-worker:v1.4.3   IfNotPresent           not in the tars
-- tar no manifest uses: registry.example.internal/demo-worker:v1.4.2

3 problems
exit status 1

Summary

• ErrImagePull on premises means kubelet decided to pull. That happens when the policy is Always, or when it’s IfNotPresent and CRI doesn’t have the image.
• ErrImagePull and ImagePullBackOff are one failure, named for the moment of the attempt and the wait that follows.
• A latest tag with no policy becomes Always. A name or tag that differs at all counts as a missing image. Depending on the import path, CRI may not know about an image that was loaded.
• Diagnosis starts from the image name in describe and goes down through logs, node logs, and a comparison of the ctr and crictl lists.

Here is how far the repros reach.

The describe output is a format example made without a cluster; everything else is real output. The repro ran on containerd 1.7.22, ctr 2.0.6, and crictl 1.28.0. While moving this post, I ran the crictl pull and ctr repros once more on an RKE2 v1.28.2 node (containerd 1.7.3-k3s1, ctr 1.7.3 and 2.0.6, crictl 1.26.1). While moving this post, I ran pullsim and imagecheck again with Go 1.23.11 on darwin/arm64 and got the same output.

When I first saw ErrImagePull, I assumed the import had gone wrong and re-checked the tars. Now I first ask why kubelet decided to pull, before asking whether the image is there. Sorting it into policy, a name mismatch, or CRI not knowing settles most cases with a single describe. In-house, with a registry available, none of the three cases showed any symptom.

References

• Kubernetes docs · Images (Image pull policy)
• kubernetes v1.28.2 · pkg/apis/core/v1/defaults.go
• kubernetes v1.28.2 · pkg/kubelet/images/image_manager.go
• containerd v2.0.6 · cmd/ctr/commands/images/import.go
• RKE2 docs · Air-Gap Install
• KEP-2535 · Ensure Secret Pulled Images