How DNS works — why one name lookup sent 8 queries in Kubernetes

Why do we need DNS?

Computers find each other by IP address, a number like 203.0.113.10. People find names like api.example.com far easier to remember than numbers. DNS (the Domain Name System) is the system that turns a name into an IP address, and browsers and server code alike go through it before they connect to anything.

Asking once, as in step 2, “what’s the IP address for this name?” is called a query. This whole post is about counting how many of those queries go out.

It works a lot like directory assistance on the phone. You give a name, they give you a number, and you call that number. The analogy breaks in one place, though. No single operator knows every number; you get passed along with “ask over there” a few times until you reach the server that knows. That handing-off is what lets DNS work at the scale of the whole internet.

Ask the nearest place first

To see who knows what, we need to take the name apart. api.example.com looks like three labels separated by dots, but there’s one more dot at the very end that we usually leave out, so the full name is api.example.com.. Labels further to the right are bigger units.

Each label has its own servers. A root server knows which server handles com, and the TLD (top-level domain) server for com knows which server handles example.com. Only the authoritative server, run by whoever registered example.com, knows the actual IP of api.example.com. The “ask over there” from earlier is this walk from the rightmost label to the left.

A lookup doesn’t start at the root every time, though. It starts at the nearest place.

• First it checks my own machine: the cache, which holds answers received recently, and the /etc/hosts file, where names and IPs are written down by hand. On Linux, the hosts: files dns line in /etc/nsswitch.conf sets this order.
• If neither has it, it asks a resolver. A resolver is a DNS server that looks up names it doesn’t know on your behalf. At home that’s usually your ISP’s DNS; in Kubernetes it’s CoreDNS. The nameserver line in /etc/resolv.conf says which resolver to ask.

A name you’ve never looked up has to go through all those trips, so it’s slow. Ask for the same name again and some hop along the way has the answer remembered, so it’s fast.

How many queries go out?

To see how many queries really go out, I wrote a tiny DNS server that prints each query it receives, one line per query. It sits in the resolver’s seat at 127.0.0.1:53, answers the two names I gave it, and says “no such name” for everything else. It never goes on to a root or TLD server. The server code is at the end of section 8.

package main

import (
  "context"
  "fmt"
  "net"
  "os"
  "time"
)

// Look up each name given as an argument and print the result and how long it took.
func main() {
  r := &net.Resolver{PreferGo: true} // the Go resolver, not cgo
  for _, name := range os.Args[1:] {
    start := time.Now()
    addrs, err := r.LookupHost(context.Background(), name)
    took := time.Since(start).Round(100 * time.Microsecond)
    if err != nil {
      fmt.Printf("%-20s failed  %v (%v)\n", name, err, took)
      continue
    }
    fmt.Printf("%-20s %v (%v)\n", name, addrs, took)
  }
}

$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ dnslookup api.example.com
api.example.com      [203.0.113.10] (1.5ms)
--- queries the DNS server received
 1  A    api.example.com.                         → NOERROR (A)
 2  AAAA api.example.com.                         → NOERROR (empty)

Each piece of information a DNS server stores is called a record, and A and AAAA are also the names of record types. When you’re starting out, these four are enough.

When the root and TLD servers in figure 3 say “ask that server,” what they hand back is an NS record.

$ dnslookup db.local
db.local             [203.0.113.99] (800µs)
--- queries the DNS server received

Answers are remembered for a TTL

Along with the IP, the authoritative server sends a TTL (time to live). It means “you can trust this answer for 30 seconds,” and the lab server attaches a 30-second TTL to every answer too. Most places an answer passes through, such as resolvers, CoreDNS, the OS, and browsers, hold on to it for the TTL and hand it back without asking again when the same question comes in. Remembering an answer like this for a while is called caching. It’s also why changing a server’s IP doesn’t reach everyone at once; the change spreads as the TTLs run out.

My Go program isn’t one of those places that remember, though. Let’s check with numbers.

$ dnslookup api.example.com api.example.com
api.example.com      [203.0.113.10] (4.4ms)
api.example.com      [203.0.113.10] (12.2ms)
--- queries the DNS server received
 1  AAAA api.example.com.                         → NOERROR (empty)
 2  A    api.example.com.                         → NOERROR (A)
 3  AAAA api.example.com.                         → NOERROR (empty)
 4  A    api.example.com.                         → NOERROR (A)

Short names work inside a Kubernetes pod

From here on, this is about Kubernetes. In Kubernetes, you can call a service in the same namespace by a short name like api. What DNS knows, though, is a longer name like api.demo.svc.cluster.local: the service name api, the namespace demo, and svc.cluster.local, which means “a service inside the cluster,” joined together.

Short names work because of the pod’s /etc/resolv.conf. When a pod starts, Kubernetes fills this file in like the one below. I added the comments.

# Resolver to ask: CoreDNS's service IP (differs per cluster)
nameserver 10.96.0.10
# Suffixes to try after a short name, in order
search demo.svc.cluster.local svc.cluster.local cluster.local
# Names with fewer than 5 dots try the suffixes first
options ndots:5

The search line is the list of suffixes. When you look up api inside the pod, the resolver tries each suffix in turn and uses the answer for the first name that exists.

$ cat /etc/resolv.conf
search demo.svc.cluster.local svc.cluster.local cluster.local
nameserver 127.0.0.1
options ndots:5
$ dnslookup api
api                  [10.96.0.20] (1.2ms)
--- queries the DNS server received
 1  AAAA api.demo.svc.cluster.local.              → NOERROR (empty)
 2  A    api.demo.svc.cluster.local.              → NOERROR (A)

Why one outside name sent 8 queries

So what happens if the same pod looks up an outside name, api.example.com? The resolver can’t tell whether it’s a short name or an outside one, so it decides by counting dots, and ndots sets the threshold. With ndots:5, a name with fewer than five dots might be a short name, so the resolver asks for the suffixed names before the name itself. api.example.com has two dots, so the suffixes go first.

func (conf *dnsConfig) nameList(name string) []string {
  rooted := len(name) > 0 && name[len(name)-1] == '.'
  if rooted { // trailing dot: ask for this name only
    return []string{name}
  }
  hasNdots := bytealg.CountString(name, '.') >= conf.ndots
  name += "."
  var names []string
  if hasNdots { // enough dots: the name itself goes first
    names = append(names, name)
  }
  for _, suffix := range conf.search { // a candidate per search suffix
    names = append(names, name+suffix)
  }
  if !hasNdots { // too few dots: the name itself goes last
    names = append(names, name)
  }
  return names
}

$ dnslookup api.example.com
api.example.com      [203.0.113.10] (2.2ms)
--- queries the DNS server received
 1  AAAA api.example.com.demo.svc.cluster.local.  → NXDOMAIN
 2  A    api.example.com.demo.svc.cluster.local.  → NXDOMAIN
 3  AAAA api.example.com.svc.cluster.local.       → NXDOMAIN
 4  A    api.example.com.svc.cluster.local.       → NXDOMAIN
 5  AAAA api.example.com.cluster.local.           → NXDOMAIN
 6  A    api.example.com.cluster.local.           → NXDOMAIN
 7  AAAA api.example.com.                         → NOERROR (empty)
 8  A    api.example.com.                         → NOERROR (A)

In the lab even the failed queries came back in under a millisecond, but in a real cluster each failed query is a round trip between the pod and CoreDNS. CoreDNS answers names ending in cluster.local with NXDOMAIN itself and keeps them in its 30-second cache, but the round trip still happens for every query. For a service that calls outside APIs often, that adds load on CoreDNS and latency to every lookup.

Cutting queries for outside names

The simplest fix is a trailing dot on the name. That final dot from figure 2 marks a fully qualified domain name (FQDN), written all the way to the root, so the resolver doesn’t add any suffix. It’s the rooted branch in the code above. The other fix is lowering ndots. With ndots:1, a name with at least one dot is asked as is first.

$ dnslookup api.example.com.
api.example.com.     [203.0.113.10] (2.8ms)
--- queries the DNS server received
 1  AAAA api.example.com.                         → NOERROR (empty)
 2  A    api.example.com.                         → NOERROR (A)

$ dnslookup api.example.com
api.example.com      [203.0.113.10] (1ms)
--- queries the DNS server received
 1  AAAA api.example.com.                         → NOERROR (empty)
 2  A    api.example.com.                         → NOERROR (A)

Neither one is free.

upstream: "api.example.com."

spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"

What you gain is fewer queries for outside names; what you give up is a consistent way of writing names, or efficient lookups for internal ones. If, like my service, a pod mostly calls services in its own namespace by short name and makes only a few outside calls, adding a dot to those few names has the smaller impact. If a pod mainly calls outside APIs, lowering ndots works better.

Lab setup and limits

I ran the labs with Go 1.22’s resolver (PreferGo) and a DNS server I wrote myself. There was no Kubernetes cluster; I only matched the pod’s resolv.conf format.

This is the whole server that prints the queries. It reads only the name and type (A or AAAA) from each query, answers names it knows, and returns NXDOMAIN for the rest. Since it listens on port 53, on Linux it usually runs as root.

package main

import (
  "encoding/binary"
  "fmt"
  "net"
  "strings"
)

// A tiny DNS server that only knows a few names. It prints every query it receives, in order.
var records = map[string]net.IP{
  "api.example.com.":            net.ParseIP("203.0.113.10"),
  "api.demo.svc.cluster.local.": net.ParseIP("10.96.0.20"),
}

func main() {
  conn, err := net.ListenPacket("udp", "127.0.0.1:53")
  if err != nil {
    panic(err)
  }
  buf := make([]byte, 512)
  for n := 1; ; n++ {
    size, addr, _ := conn.ReadFrom(buf)
    name, qtype, qend := parseQuestion(buf[:size])
    ip, ok := records[name]
    rcode := "NXDOMAIN"
    var resp []byte
    switch {
    case ok && qtype == 1: // A
      resp, rcode = answer(buf[:qend], ip.To4()), "NOERROR (A)"
    case ok: // AAAA etc.: the name exists but has no record of that type
      resp, rcode = answer(buf[:qend], nil), "NOERROR (empty)"
    default:
      resp = header(buf[:qend], 3, 0) // 3 = NXDOMAIN
    }
    fmt.Printf("%2d  %-4s %-40s → %s\n", n, typeName(qtype), name, rcode)
    conn.WriteTo(resp, addr)
  }
}

func parseQuestion(m []byte) (string, uint16, int) {
  var labels []string
  i := 12
  for m[i] != 0 {
    l := int(m[i])
    labels = append(labels, string(m[i+1:i+1+l]))
    i += 1 + l
  }
  i++
  return strings.Join(labels, ".") + ".", binary.BigEndian.Uint16(m[i:]), i + 4
}

// Copy the query as is and turn only the header into a response.
func header(q []byte, rcode byte, ancount uint16) []byte {
  r := append([]byte(nil), q...)
  r[2] = 0x81         // QR=1, RD=1
  r[3] = 0x80 | rcode // RA=1
  binary.BigEndian.PutUint16(r[6:], ancount)
  binary.BigEndian.PutUint16(r[8:], 0)
  binary.BigEndian.PutUint16(r[10:], 0)
  return r
}

func answer(q []byte, ip4 net.IP) []byte {
  if ip4 == nil {
    return header(q, 0, 0)
  }
  r := header(q, 0, 1)
  r = append(r, 0xc0, 0x0c, 0, 1, 0, 1)    // name pointer, TYPE A, CLASS IN
  r = binary.BigEndian.AppendUint32(r, 30) // TTL 30 seconds
  r = append(r, 0, 4)
  return append(r, ip4...)
}

func typeName(t uint16) string {
  switch t {
  case 1:
    return "A"
  case 28:
    return "AAAA"
  }
  return fmt.Sprint(t)
}

Instead of memorizing CS fundamentals one at a time, I decided to follow the path a single request takes in and out of a server. It finds the address, connects, sends the request, and then the server receives it, uses CPU and memory, finds the data, and responds. Each stop on that path has a CS topic, and next to it sits how Go and Kubernetes behave in everyday use.

With the name turned into an IP, the next stop is opening an actual connection to that IP. Next, I plan to write up how TCP opens and closes a connection, and what TLS checks on top of it.

References

• RFC 1034 · Domain Names - Concepts and Facilities: the name space, delegation, and resolvers
• DNS for Services and Pods: a pod’s resolv.conf, dnsPolicy, and dnsConfig
• resolv.conf(5): search, ndots, single-request
• Go net package · Name Resolution: the Go resolver and the cgo resolver
• Go 1.22 dnsclient_unix.go: nameList (candidate order) and the code that sends A and AAAA together
• CoreDNS cache plugin · kubernetes plugin