Skip to main content

Daily Log — 2026-07-15: A mutating webhook must not eat fields it doesn't understand

· 5 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a day spent mostly inside Kubernetes admission-webhook behavior, plus a Helm correctness fix and a couple of smaller escalations. Specifics are private; the patterns transfer.

1. A mutating admission webhook must round-trip fields it doesn't know about​

The headline bug: a mutating admission webhook that injects sidecars/config into pods was silently dropping a newer pod-spec field — Kubernetes Dynamic Resource Allocation (DRA) resourceClaims — on clusters running a recent Kubernetes version. Pods came out of admission missing a field the user put in.

The root cause is a very common webhook anti-pattern: the webhook deserializes the incoming pod into a struct built from an older client library, mutates it, and re-serializes. Any field the old struct doesn't know about isn't "passed through" — it evaporates, because it was never parsed in the first place.

How I approached it before touching the fix:

  • Build the smallest reproduction that isolates the field. A single pod manifest that sets the new field, applied to a namespace where the webhook is active, then kubectl get pod <name> -o yaml and diff the admitted spec against the manifest. If the field is gone, the webhook ate it — no need to read a line of webhook code yet.
  • Confirm it's the webhook, not the API server. Apply the same manifest with the webhook disabled/excluded. If the field survives there and dies with the webhook on, you've localized it.
  • The class of fix is "stop losing data," not "special-case this field." Patch strategies that preserve unknown fields (a JSON patch computed against the raw object, or keeping the decode target current with the cluster's API) fix the whole family of "webhook drops new field X" bugs — not just today's field.

Lesson: a webhook that decodes-mutates-reencodes is only as new as its client library. Any object field newer than that library gets quietly deleted. Treat "preserve fields I don't recognize" as a hard requirement of any mutating webhook, and prove it with a spec diff.

2. Only render a resource when the feature that needs it is enabled​

Separately, a Helm chart was always creating an internal Service, even when the feature that Service exists for (a secure-remote-access capability) was turned off. The fix was a one-liner in spirit — wrap the template in {{- if .Values.featureX.enabled }} — but the reasoning is worth keeping:

  • An always-on resource for an off-by-default feature is pure downside: extra attack surface, a confusing dangling Service in kubectl get svc, and a support question waiting to happen.
  • Chart correctness = "what I render matches what I enabled." If a value gates a feature, it should gate every resource that feature needs — Deployment, Service, RBAC, the lot — not just the obvious one.
  • Guard, then verify with helm template. Render with the feature on and off and eyeball the diff; the "off" case should simply not contain the resource.

Lesson: conditional features deserve conditional manifests. If a capability is disabled, none of its supporting objects should hit the cluster.

3. Reproduce "control plane is unavailable" on purpose​

A recurring escalation theme this week is client behavior when the SaaS control plane is unreachable. The only honest way to reason about it is to induce the condition in a lab: scale the upstream dependency to zero (or otherwise cut the path) and observe the client.

The subtlety that kept coming up: how the path is broken changes the symptom.

  • A silently dropped connection (packets go into a black hole) makes the client hang until a context deadline — long, ugly timeouts.
  • A fast rejection (connection refused / an immediate "not found") makes the client fail quickly and fall through to its cache/fallback logic.

Two "the backend is down" scenarios that look identical on a diagram produce completely different client experiences. If your repro uses one failure mode and production hits the other, you'll "fix" a problem the customer doesn't have.

Lesson: when testing degradation, reproduce the exact failure mode (drop vs reject vs slow), not just "the dependency is gone." The failure's shape is part of the bug.

4. A silent install failure is worse than a loud one​

A CLI's shell-completion install step was failing without an error — it just didn't work, and nothing told the user why. The debugging (reinstall clean, from package manager and from the raw release artifact, inspect the profile scripts and install dir) was routine; the lesson is about the failure mode:

Lesson: a step that can fail must exit non-zero and say what it needed (missing dir, wrong shell, no write permission). "Silently does nothing" is the worst outcome — the user believes it worked and only finds out later. Loud, specific failure beats quiet success-that-wasn't.