Daily Log — 2026-07-16: Your webhook is only as new as its client library

A sanitized log from a long build-and-deploy day: shipping the fix for yesterday's admission-webhook bug, then a multi-hour battle to stand up an edge gateway in a lab so I could reproduce a memory issue. Names, tickets, and hostnames are private; the patterns transfer.

1. Fixing "the webhook drops new fields": bump the client to match the cluster

Yesterday I proved a mutating admission webhook was silently eating a newer pod-spec field. Today's fix was, at its core, make the webhook's Kubernetes libraries as new as the clusters it runs on: bump client-go (and the controller-runtime that has to match it) to the version that actually knows about the field, then let the object round-trip intact.

What made it non-trivial and worth noting:

• The library bump is a lockstep, not a single dependency. controller-runtime, client-go, and the k8s.io/api* packages move together; bumping one and not the others gives you a compile error at best and subtle behavioral drift at worst. Bump the set, then go mod tidy + build.
• Pin the toolchain while you do it. A dependency bump can drag in a newer language/runtime requirement. I pinned the Go version explicitly so "works on my machine" and "works in the release workflow" stayed the same machine.
• Ship it through the real release pipeline and re-run the reproduction against the artifact. A green unit test is necessary but not sufficient; the thing that convinced me was applying yesterday's repro pod against a cluster running the released build and seeing the field survive.

Lesson: for a component that parses first-party platform objects, treat "keep the client library current with the platform" as ongoing maintenance, not a one-time bump. A lagging client doesn't error — it quietly discards whatever the platform added since.

2. "Works with docker pull, fails in the cluster" is almost always the pull secret

Standing up the gateway meant its image came from a private registry. Locally, docker login + docker pull worked fine. In the cluster, the pod sat in ImagePullBackOff. That gap is a well-worn checklist:

• The image pull secret is namespace-scoped. A docker-registry secret only helps pods in the same namespace (or a ServiceAccount in that namespace that references it). Creating it once in default does nothing for your app in another namespace.
• Inspect the decoded secret, don't trust that you created it right. kubectl get secret <n> -o yaml | yq '.data.".dockerconfigjson"' | base64 -d | jq shows the registry host, username, and auth actually stored. A typo here fails silently as "unauthorized."
• Match the registry host exactly. docker.io vs index.docker.io/v1/ vs a private host — the key in .dockerconfigjson must match what the image reference resolves to.
• Confirm the platform/arch exists. docker pull --platform linux/amd64 <image> locally verifies the tag+arch are real before you blame the cluster.

Lesson: when a pull works at your shell but not in-cluster, stop poking the deployment and audit the pull secret: right namespace, right decoded contents, right registry host, right arch.

3. Upstream image relocations will break your pulls without warning

A dependency (a cache/redis sidecar from a well-known image publisher) refused to become ready because the publisher relocated their public catalog — the previously-public images moved to a *legacy namespace on the registry, and the old tag no longer resolved.

This is an industry-wide pattern right now, not a one-off:

• You don't control your base images' availability. A vendor can re-org, re-license, or retire tags on their schedule, and your "pinned" tag can stop resolving.
• Mirror and pin what you depend on. Copy critical base/sidecar images into a registry you control and reference that. Renovate-style bots should propose bumps; they shouldn't be your only copy.
• Read the publisher's deprecation notes before chasing tag errors. Ten minutes on their registry page ("we moved public images to X") saves an hour of "why won't this tag pull."

Lesson: treat third-party base images as untrusted supply — mirror them into a registry you own so a vendor's catalog reshuffle is an annoyance, not an outage.

4. Reproducing a "dependency down" memory bug: induce it, then load it

The goal underneath all the yak-shaving was to reproduce a memory spike that only happens when an edge service's upstream control plane is unavailable. Two access problems and one method:

• The control plane's API was behind authorized networks. A raw nc -vz <host> 443 that hangs (rather than refuses) is the classic signature of "your source IP isn't allow-listed" — the packets are dropped, not rejected. Add your IP to the control-plane authorized-networks/firewall and the hang turns into a handshake.
• Induce the outage deliberately. Scale the upstream dependency to zero — but respect its autoscaler, or the HPA will fight you and scale it right back. Turn the autoscaler down too, or it isn't really "down."
• Then apply load and watch two things at once: the process's own logs (filtered down to signal) and a metrics dashboard for memory/latency. A leak shows up as memory that climbs and doesn't return after the load stops.

Lesson: a "when X is down" bug needs X to actually be down and traffic flowing. Reaching the environment (authorized networks), forcing the failure (scale-to-zero past the autoscaler), and loading it are three separate steps — skip one and you'll conclude "can't reproduce."