From 300 calls to 1 — how I learned that optimization changes failure modes

First, a map of the route this post follows.

The situation, and the first fork

There’s a feature where users bundle several resources into one plan and save it. Right before saving, a validation step checks that those resources actually exist, and with enough items in the plan, that step timed out. Small plans were fine; big ones died.

There were two broad ways to fix it, and this was the project’s first trade-off.

I chose the second. The reason came from the product side. This validation is part of a screen flow that tells you what’s wrong before you save. Going async would mean users hit save and only later hear that it didn’t work, which removes the whole reason for validating up front.

Looking back, this decision became the constraint on everything that followed. Pinning down “stay synchronous” meant latency really had to drop, and left no room to fudge it. Settling the constraint first narrowed the later choices quickly.

The first answer: counting calls

Opening the validation code, I found it calling an external service once per resource. A classic shape.

for _, id := range instanceIDs {
  if _, err := resourceService.GetInstance(ctx, id, WithSync(true)); err != nil {
    return err
  }
}

The count came out like this.

N        number of instances
V        volumes attached to each instance
M        networks used by each instance

calls = N  +  N×V  +  2×N×M  +  (distinct lookups for other kinds)
                        └ two IP/MAC duplicate checks per network

N=50, V=2, M=1  →  50 + 100 + 100 + α  ≈  300 calls in series

On top of that, the duplicate checks had no cache at all and went out every time. So far, textbook N+1. “Batch it” was the natural conclusion, and that’s exactly what I proposed.

Turn ①: the problem wasn’t the number of calls

Just as I was about to design the batching, something caught. I had counted the calls but never checked how expensive one call was. I followed the code into the external service.

GetInstance(id, WithSync(true))
  └─ syncOne(id, kindInstance)
       ├─ newSession(clusterID)
       │    ├─ look up cluster info + decrypt credentials
       │    └─ authenticate to the cloud platform, get a token   // external round trip
       ├─ fetch one resource                                    // external round trip
       └─ session.Close()                                       // close the session

WithSync(true) wasn’t the cache-refresh flag I had vaguely assumed. Every call opened and closed a new auth session, and because the flag forced it, there was no deduplication either. In other words, validating one resource paid for an auth round trip, an API round trip, and closing the session. Stack that 300 times in series and of course it times out.

Had I gone straight to batching on the call count alone, the cost per call would have stayed, and I’d have been asking why batching didn’t make it faster.

The answer was already in the repo

As I started designing the batching, I found the pattern already in the same package. The read path, which handles the same data, had been doing it this way for a while.

collectRequired() // ① walk the plan and collect only the needed IDs. Zero external calls
fetchByKind()     // ② one list call per kind, in parallel with errgroup
lookup(byID)      // ③ decide using an in-memory map. Zero external calls

Only the validation path had been left on the old approach. So the plan became: reuse this instead of building something new. Along the way I found one more thing, and it turned out to matter a lot.

There were two APIs that could serve the duplicate check, with wildly different costs, and only validation was taking the expensive one. The read path was already on the cheap one. Code review doesn’t catch this. It only shows up when someone who knows both APIs answer the same question compares them.

Turn ②: batching was a trap too

Reading the rest of the batch code I’d planned to reuse, I realized my estimate of the improvement was wrong.

if req.Sync {
  syncAll(clusterID, kindInstanceList) // resyncs the whole cluster's list
}
return readByIDs(req.IDs) // only then reads the requested IDs

The bulk lookup’s sync flag didn’t sync just the requested IDs; it resynced the entire cluster’s list. And the caller was splitting IDs into fixed-size chunks.

Batching was necessary but not sufficient. This is where the design’s axis shifted: the question of how to batch the calls turned into when to sync. The problem was that sync time and lookup time were glued together, and they had to be separated.

Splitting “fetch only what we need” three ways

During the discussion, someone suggested just loading the resources the plan has. Fair enough, but as I listened, it read three different ways. If people carry on designing with different readings of the same sentence, things drift far apart later. So I split it first.

② turned out to be impossible. The unit of syncing was either a whole kind or a single item, and splitting by item takes you right back to N+1. Given how the external list API works, “just these 20” wasn’t a valid request. The units the lower system offered constrained the design.

③ was bigger than I expected. The bulk search assembled every linked resource for each instance and made extra lookups per network: roughly nine DB queries per instance. And validation used none of that data.

So I decided to do both ① and ③. Fixing only one makes the other the new ceiling; cut the kinds and the field assembly still blocks you. The single sync in the figure above is, in the end, a sync over only the kinds the plan actually uses.

The catch was that ③ required deciding which fields validation actually uses. So I reread the validation code from the top, and that’s where I found the biggest thing in the whole project.

The biggest find: validation was throwing the response away

I went through every validation function, scattered by resource kind, and without exception they looked like this.

func validateVolume(ctx context.Context, id string) error {
  _, err := resourceService.GetVolume(ctx, id)
  return err // discard the response, look only at err
}

func validateNetwork(ctx context.Context, id string) error {
  _, err := resourceService.GetNetwork(ctx, id)
  return err // same here
}

// … all ten-odd kinds were identical

Every response went to _, and only err was checked. There were two exceptions, and they boiled down to whether a count was zero and whether something existed.

All validation needed, 100% of the time, was whether the thing exists. The detailed lookups had never been needed. Once I saw that, everything before it fell into place: we had been authenticating 300 times to build data nobody used.

That also settled ③. No fields were needed at all, just one bit: exists or doesn’t. But the existing lookup APIs only returned fully assembled objects, so there was no way to ask that question. It meant adding a lightweight existence-only lookup to the external service, and that comes back later as a deploy-order problem.

I wish this had come earlier. “What does this code actually need?” should have been the first question when the optimization started. I started with “how do I fetch this faster?” and only got to “does it need to fetch this at all?” at step three.

Arguing against “just merge them”: two consumers, two policies

Since the read path and validation path did similar work, someone proposed lifting the loading into one place both could share. No more duplication, one place to maintain; it was tempting. But I had to argue against it.

They needed different data and had opposite failure policies. Merging the fetch wholesale would make validation pull in detailed data it doesn’t need, undoing the slimming we had just achieved.

That doesn’t mean nothing was worth sharing. What really deserved sharing was the collection logic that extracts which resources a plan references, the part where a change to the plan’s structure should be fixed in exactly one place. The seam to split on was between collecting and fetching.

And this is where the biggest risk in the design was. The read path’s logic is fail-soft. If that policy leaked through the shared layer into validation, resources that don’t exist would pass validation. Breaking correctness while fixing performance is the worst possible outcome.

So the shared layer only holds errors and owns no policy. What to fetch was merged; what to do when it’s missing stayed with each consumer.

Optimization changes failure modes

Consolidating syncing into once per request left an uncomfortable question: what if that one sync fails?

Checking the existing behavior, a failed sync just logged and carried on with stale data. So even today, validation was only best effort, with no real-time guarantee. What I had to weigh was the blast radius of a failure.

After some thought, I kept allowing stale data. There were three reasons, in decreasing weight.

1. No regression. It matches current behavior; this change doesn’t introduce a new risk.
2. Availability comes first. A temporary outage in an external system blocking all saves would be worse.
3. A real-time guarantee was never achievable. A resource can be deleted right after passing validation. Validation as a real-time guarantee was impossible from the start.

The third was decisive. There’s always a gap between validating and actually using a resource, and state can change in between. Once I accepted this TOCTOU (time-of-check to time-of-use) gap, the picture changed: the real safety net is rechecking at execution time, and validating at save time is a filter in front of it. I had been giving validation more meaning than it could carry.

Instead, I decided to surface failures rather than swallow them quietly, and I recorded the decision in an ADR, because it’s hard to reverse, looks odd without context, and is a genuine trade-off. When someone later asks why we proceed even when this fails, the answer can’t be missing from the codebase.

Why I chose not to add a fallback

Adding the lightweight lookup API to the external service created a deploy order. What if the other side is still on the old version? The natural proposal was a safe fallback: if the new API isn’t there, use the old path.

I chose not to. The deciding factor was that the interface definitions are pinned as a module, so the build already enforces the order. What’s left is runtime compatibility, and I judged that to be a job for the deploy procedure.

In this case, the path the fallback would have kept alive was the very path that was timing out.

Being honest about the numbers

The final design’s expected effect came out like this.

It looks convincing, but this is arithmetic built on an assumed time per call. Nothing has been measured yet. So instrumentation went in as the very first task. You can’t capture a baseline after the fix; at that point you’ll never be able to say how much better it got.

On this project I got the math wrong twice: once by not looking at cost per call, and once by not looking inside the bulk API. So instead of a third calculation, I decided to capture a baseline before changing anything.

Summary

• Call count and cost per call multiply. I should have measured which one dominated first.
• The batch code already in the repo had its own trap once I read it to the end.
• Two APIs answered the same question at very different costs.
• Validation and lookup share what they fetch, but each keeps its own policy for missing resources. A fail-soft policy leaking into validation breaks correctness.
• Collapsing the sync into one call traded scattered partial failures for one all-or-nothing failure, and I recorded that decision in an ADR.
• The expected gains are still arithmetic. Capture the baseline before fixing anything.

If the batching answer from the first 30 minutes had been a little less plausible, I would have looked longer. Both turns came from doubting that answer and reading the code one level deeper.