Skip to main content

Daily Log — 2026-07-10: I couldn't reproduce the bug because my lab was too healthy

· 4 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a day closing out an offline-resilience investigation on a secrets gateway. The punchline: my inability to reproduce a customer's failure was itself the clue. Specifics are private; the patterns transfer.

1. If you can't reproduce a bug, check whether your lab is too healthy​

A customer reported that when their gateway lost its upstream connection, one client (an operator that syncs secrets into the cluster) failed on every refresh while another (an injector) kept working. I had the whole topology stood up — gateway, cache, both clients — hard-blocked the upstream, and… everything worked. No failure. For a while that's maddening: "works on my machine" is not a diagnosis.

The reason my lab wouldn't fail turned out to be the answer: my environment had proactive caching turned on, which had quietly warmed every secret into the cache. So every lookup was a cache hit, and nothing ever needed the (now-blocked) upstream. The customer's failing secret simply had never been cached, so its very first lookup went upstream and died.

The determinant wasn't "online vs offline" — it was cache-hit vs cache-miss on the specific item requested. Once I evicted / used a never-fetched key, my lab reproduced the failure instantly.

Lessons:

  • A repro that won't fail is data, not a dead end. Ask why it's healthy. An over-warmed cache, a pre-seeded row, a permission you happen to have — the thing keeping your lab green is often the exact variable the bug depends on.
  • Match the customer's state, not just their topology. Same components in the same shape can behave completely differently depending on what's already cached/populated. Reproduce their state (here: a cold, never-fetched key), not just their architecture.

2. Verify probe behavior empirically — don't trust the code you read​

Going in, I had a tidy hypothesis from reading the source: the liveness probe hits a /health endpoint that depends on the upstream, so an upstream outage should fail the probe and restart the pod. Clean story. It was also wrong.

When I actually blocked the upstream and watched, the probe endpoint kept returning 200 for the entire ~17-minute outage — the pod never restarted. It turned out the probe target is served by a local/cache health handler, not the upstream-dependent one (which lives on a different port that isn't wired to any probe). My code-reading had picked the wrong handler.

Lesson: liveness/readiness behavior is an empirical question. Reading the handler is a hypothesis; the truth is what the orchestrator observes under the actual failure. Block the dependency, poll the probe across the full failureThreshold × periodSeconds window, and watch the restart count. Two handlers named /health on two ports is exactly the kind of thing you'll never catch by reading alone.

3. Provenance you can't see is provenance you can't debug​

The single hardest part of the whole investigation was answering "where did this value actually come from — upstream, in-memory cache, or the shared cluster cache?" Nothing in the logs said. I inferred it from timing, from which replica answered, and from poking the cache directly — slow and circumstantial.

So the most durable fix wasn't a code change to the data path at all; it was filing a request to log the retrieval source on each lookup. When a system has multiple layers that can satisfy a request, "which layer served this?" should be a first-class, greppable log line — not something an engineer reverse-engineers during an incident.

Lesson: make data provenance observable before you need it. If a request can be answered by N sources, emit which one answered. It's a one-line log that saves hours of "is this cached or live?" during the next outage.

4. Close the inquiry, then split the engineering out of it​

Separately, I wrapped a "how do I do X via the API?" investigation that had uncovered real bugs. The temptation is to leave everything in one sprawling ticket. Instead I closed the question (the user got a working answer for the supported cases) and split the engineering into two clean, separately- prioritizable items:

  • a feature ticket ("support these additional key types / formats"), with the open product/contract questions written down (e.g. should the API accept a keystore passphrase and decrypt server-side?), and
  • a bug ticket for the crash (a validator that panicked instead of returning a clean "unsupported"), with the root-cause code references attached.

Lesson: an answered question and a code fix are different work items. Unbundling them lets support close the customer loop now, while engineering triages the feature and the bug on their own merits instead of staring at one ticket that's simultaneously "done" and "not started."