Skip to main content

Daily Log — 2026-07-20: Validate a fix against the exact repro that caught the bug

· 5 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a loop-closing day: proving a memory fix actually holds, chasing a private image-pull that worked everywhere except in the cluster, and scoping a scalability repro around Kubernetes' request-fairness knobs. Names, tickets, and hostnames are private; the patterns transfer.

1. Validate the fix with the same experiment that exposed the bug​

A build landed that was supposed to bound memory on a read path when the upstream control plane is down. "Supposed to" isn't "does." I validated it by re-running the exact experiment that originally surfaced the leak, as a controlled A/B:

  • Baseline, then variable. Profile A: upstream healthy, normal load. Profile B: scale the upstream to zero, apply the same load. The only thing that changed between runs is the condition under test — so any difference is attributable to it.
  • Capture profiles, not vibes. pprof heap/alloc snapshots from both runs, saved and archived, turn "seems better" into a diff you can attach to the ticket and revisit later. A metrics dashboard corroborates the shape; the profile localizes where the allocations live.
  • Confirm the payload contains what you think. The build bundled several related changes (a cache-size cap, "don't allocate a cache entry on a miss," a read/write-lock race fix). I listed exactly which commits were in the artifact before crediting the result to "the fix."

Lesson: closing a performance bug is an experiment, not an assertion. Reproduce the original failure condition, A/B it against a healthy baseline with real profiles, and keep the artifacts — "trust me, it's better" is not a validation.

2. Same credentials, works locally, fails in-cluster: audit the pull path, methodically​

A pod was stuck unable to pull its image from a private registry — while docker login + docker pull --platform linux/amd64 <image> with the same credentials worked fine on my laptop. That contradiction narrows the search enormously; the problem is between the cluster and the registry, not the credentials themselves. My checklist:

  • Decode and diff the pull secret across every environment that works vs the one that doesn't. kubectl get secret <n> -o yaml | yq '.data.".dockerconfigjson"' | base64 -d | jq, dumped per cluster/namespace to files, then diff. A working cluster is the best oracle for what a broken one is missing.
  • Namespace scoping is the usual culprit. The secret must exist in the workload's namespace and be referenced (via imagePullSecrets or the pod's ServiceAccount). "It's there in another namespace" is the same as not there.
  • Registry host + arch must match the reference. The auth entry's host has to match how the image ref resolves, and the tag+platform must actually exist (that's what the local --platform linux/amd64 pull proves).
  • Then capture the recipe. Once it worked, I wrote the whole thing down as a small create-the-secret script so the next person (or the next cluster) doesn't rediscover it.

There's a deeper design fork here too: whether a service downloads its components at boot (from object storage) or ships them baked into the image. That single toggle changes your failure modes — a bundled image can't fail a runtime download but is bigger and needs registry access; a downloading image is smaller but adds a boot-time network dependency. Knowing which mode you're in tells you where a "won't start" failure will come from.

Lesson: "works locally, fails in-cluster" with identical creds is almost never the creds — it's namespace scope, the decoded secret contents, or the registry host/arch. Diff a working environment against the broken one, then script the fix.

3. A pod-burst scalability test is really a Kubernetes fairness test​

New work: reproduce a customer-shaped scenario where a burst of short-lived pods/jobs each fetch a secret at startup, all hitting the control plane and the Kubernetes API roughly at once. Designing that repro sent me straight into the platform's overload-protection machinery:

  • API Priority & Fairness is the API server's bouncer. Under a burst, the apiserver classifies and queues requests by flow; --max-mutating-requests-inflight and its read counterpart cap concurrent work. Your client doesn't see "server busy" — it sees 429s and added latency, which look like your slowness.
  • Client-side throttling stacks on top. client-go has its own QPS/burst limits; a stampede of clients each self-throttling produces retry waves that can be worse than the original burst.
  • Node limits shape the burst. Max-pods-per-node and scheduler throughput bound how fast the burst even materializes — the large-cluster best-practices guidance exists precisely because these interact non-linearly.
  • Build a self-contained env to test it. Rather than borrow a shared cluster, I stood up an isolated VM running the dependencies so I can crank the burst without collateral damage.

Lesson: "test it under a burst of pods" is not a load test of your component — it's a test of how your component behaves while the platform's fairness/throttling kicks in. Design the repro around API Priority & Fairness, client QPS limits, and node caps, or you'll measure the wrong bottleneck.

Two quick loop-closers worth a line. I merged a tiny fix for a broken documentation link surfaced during an auth flow — the kind of paper-cut that erodes trust in docs and is cheap to fix once someone actually files it. And I reviewed the pull request for last week's admission-webhook fix with yesterday's reproduction still on hand, so "does this preserve the field?" was something I could check, not just read.

Lesson: reviewing a fix is easiest when you still have the repro that proved the bug — keep the reproduction alive until the fix is merged, not just until you understand the cause.