Skip to main content

Daily Log — 2026-07-02: two ways to front a service, and the stale-artifact trap

· 5 min read
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a day spent putting ingress in front of the same service two different ways, disproving a "CLI bug" that turned out to be a stale artifact, and wiring open-source analytics into a static site. The specifics are private; the patterns are the point.

1. Two ways to front the same service: Gateway API vs. cloud-native ingress

In one day I put two different front doors on the same backing service: the Kubernetes Gateway API (via Envoy Gateway) on a self-managed k3s cluster in the morning, and a cloud-provider-native ingress on a managed Kubernetes cluster in the afternoon. Doing both back-to-back made the tradeoff concrete.

Gateway API (portable, controller-agnostic). The resource model splits responsibilities:

  • A GatewayClass picks the controller implementation (here, Envoy Gateway).
  • A Gateway owns listeners, ports, and TLS.
  • An HTTPRoute owns host/path routing and attaches to the Gateway.

That separation is the whole selling point: the platform team owns the Gateway, app teams own their HTTPRoutes, and the routing config is portable across any conformant controller. The migration itself is a "rip and replace" of the old ingress controller — uninstall the previous one cleanly, install the new one into its own namespace, then apply the Gateway + route. Watch everything at once while it settles:

kubectl get pods,svc,gateway,httproute -A

Cloud-native ingress (less portable, more integrated). On managed Kubernetes, the provider's own ingress wires the Ingress object straight to a provider load balancer, and you tune it through provider-specific CRDs (backend config for health checks/timeouts, frontend config for TLS/redirect policy) rather than controller annotations alone. You give up portability and get tighter integration with the platform's load balancer, health checking, and logging.

Reusable takeaways:

  • Pick the abstraction that matches ownership. Gateway API shines when multiple teams share infra and you want portability. Provider-native ingress shines when you want the managed load balancer's features and don't plan to move.
  • Provisioning is asynchronous. A cloud LB can take minutes to get an address. Add the hostname to /etc/hosts locally so you can test the routing path before DNS/LB are ready.
  • Terminate TLS at the edge with a full chain. Whichever front door you choose, the listener needs the leaf plus intermediates, and SANs that cover both the public hostname and the in-cluster service name so the same cert works through kubectl port-forward.
  • Clean up before you rebuild. Half the afternoon was deleting stale namespaces, leftover services, and orphaned secrets from prior experiments. A cluster carrying old ingress controllers and dead operators will fight your new setup.

2. "Is it a bug?" — verify against the artifact, not the label

I spent time chasing a suspected tooling bug: a certificate stored in a secrets system reported the wrong issuer (a self-signed CN instead of my lab CA). The obvious hypothesis was that the CLI mangled the chain on import.

It didn't. Reproducing with a fresh CA-signed cert showed the tool parsed the chain leaf-first and stored the issuer verbatim — correct every time. The real problem: the stored item was a stale, self-signed cert from a year earlier. My regenerated CA-signed material never actually landed in the item, so the system was faithfully reporting an old artifact.

The trap inside the trap: the update path updates in place and does not bump the version number. So "version is still 1" looked like "nothing changed," when in fact a successful update also leaves the version at 1. The version field was useless as a change signal.

The reusable checklist for "is this a bug or stale state?":

  • Reproduce from scratch with a known-good input before blaming the tool.
  • Compare immutable identity, not metadata. For certs that's the serial number (and notBefore), not the version counter or a display field. Two certs with different serials are different certs, full stop.
  • Suspect the write path. If a value looks stale, ask whether your last write actually executed. A helper that "creates if missing, updates if present" can silently skip when a required binary isn't on PATH or the wrong profile/target is selected.
  • Don't trust counters as change signals unless you've confirmed they increment on the operation you care about.

"The field is wrong" was the symptom. "The write never happened and the field is correctly reporting old data" was the diagnosis.

3. Open-source analytics on a static site, and the Pages gotchas

I added Umami (open-source, privacy-friendly) to this Docusaurus site. Injecting the script is trivial; the friction was all in the GitHub Pages deploy.

  • Confirm the injection in production, not locally. A quick check beats guessing:
curl -s https://<your-site> | grep data-website
  • .nojekyll matters on Pages. Without it, GitHub's Jekyll processing can drop files/folders that start with an underscore — which static site generators emit. Make sure the published artifact contains .nojekyll.
  • Be deliberate about deploy triggers. After earlier scoping the deploy workflow to only run on content changes, dependency/config changes (like adding an analytics script) wouldn't publish. I added a manual trigger and broadened the paths so a package.json/lockfile change can also deploy. Path filters are a product decision — revisit them when the definition of "a change worth publishing" changes.

Commits on this public repo:

  • da5e0575a add Umami analytics.
  • 62ffa6fa8 add a manual deploy trigger (and deploy on dependency changes).
  • c88a4087e ensure .nojekyll is in the deployed artifact.

4. Small fix, right layer

A UI list "sorted alphabetically" but the items were clearly out of order. The cause was a classic one: the sort keyed off an internal identifier rather than the human-facing display name. The fix was a one-liner, but the lesson is where the bug lives — when a user says "this sort is wrong," check which field the comparator uses before assuming the sort algorithm is broken. Sort by what the user sees.