Skip to main content

Daily Log — 2026-07-09: how you block the network decides how it fails

· 5 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a day spent reproducing an "it fails when we're offline" report against a secrets gateway — and rediscovering that a repro is only useful if it fails the same way the customer's does. Specifics are private; the patterns transfer.

1. "It fails when offline" is not a bug report — find the real variable​

A customer reported that when their gateway couldn't reach the upstream SaaS, their downstream consumers (an operator, an injector, and the CLI) all failed to fetch secrets. The obvious framing — "offline breaks everything" — is a trap, because it's not actually true.

Once I stood up the same topology (gateway + a cluster cache, with a proxy I could flip on and off between the gateway and the SaaS) the real variable fell out immediately: whether the requested secret is already in the gateway's cache.

  • Cached secret + SaaS down → everything works. The operator reports synced, the injector serves, direct describe/get calls return 200.
  • Uncached secret + SaaS down → it fails, because the first lookup (a metadata/describe call) still tries to reach the SaaS.

So the customer's failing case wasn't "offline mode is broken" — it was "the specific secret they tested with had never been cached." That's a completely different conversation, and a much easier one: warm the cache, or accept that first-touch of a cold key needs connectivity.

Lesson: when someone says "X breaks when the network is down," don't try to make X work offline — find the state that separates the cases that work from the ones that don't. Here it was cache residency, not connectivity.

2. DROP vs REJECT: the same outage, two totally different failures​

The most interesting part: my repro and the customer's repro disagreed on the symptom, even though we were both "blocking the network." That disagreement was the actual clue.

  • The customer blocked traffic with an iptables ... -j DROP rule. A dropped packet gets no response at all, so the client sits there until a timeout fires — the failure surfaces late, as a context deadline exceeded hang.
  • My proxy rejected the connection fast. So instead of hanging, the client got an immediate, clean error — in this case a fast Not found in cache.

Same "the SaaS is unreachable" scenario, two completely different observable behaviors: a silent multi-second hang versus an instant, legible error. If you're trying to reproduce a customer issue and your symptom doesn't match theirs, the way you're simulating the fault is probably the difference.

Quick reference for simulating an outage on purpose:

# Hang: packets vanish, client blocks until its own timeout
iptables -A OUTPUT -p tcp --dport 443 -d <upstream> -j DROP

# Fast failure: connection actively refused/closed
iptables -A OUTPUT -p tcp --dport 443 -d <upstream> -j REJECT --reject-with tcp-reset

When you write up a network-failure repro, state which one you used — "we blocked egress" is ambiguous and will send the next person chasing the wrong symptom.

3. Don't let a health probe kill a service that's degraded-but-serving​

A related thread from the same investigation: the service's liveness probe pointed at a /health endpoint that returned unhealthy whenever the upstream dependency was unreachable. So the moment the network blipped, the orchestrator's probe failed and it restarted the pod — turning a recoverable degraded state into an outage, and throwing away any warm in-memory cache on the way down.

This is a classic liveness/readiness mistake:

  • Liveness should answer "is this process wedged and in need of a restart?" It must not depend on external dependencies, or a dependency outage becomes a restart storm.
  • Readiness can reflect "should I receive new traffic right now?" — that's the right place to shed load when a dependency is down, without killing the process.

If a component is designed to keep serving from cache during an upstream outage, a dependency-coupled liveness probe actively sabotages that design. The fix is to make liveness check only local health and move any dependency-awareness to readiness (or make the health endpoint tolerant of a known, survivable offline mode).

4. Same feature, two backends: the UI and the API don't always agree​

Unrelated, from a separate write-up: a "why does uploading a certificate work in the web console but fail over the REST API?" puzzle. The answer was that the two paths hit different backends. The UI took a lenient store-what-you-give-me path; the public API ran a stricter validator that (a) crashed on elliptic-curve keys via a nil-pointer path, (b) didn't recognize Ed25519 at all, and (c) had no way to receive a keystore passphrase.

Takeaways worth generalizing:

  • "Works in the UI" doesn't mean "works via the API." When a customer hits a wall on one surface, reproduce it on the exact surface they're using — the code paths, and their bugs, can be entirely different.
  • Validation that panics is worse than validation that rejects. A type-assertion that assumes RSA and dereferences a nil on anything else turns a clear "unsupported key type" into a 500. Handle key types generically (compare public keys through the standard crypto.PublicKey interface) instead of assuming one shape.