Skip to main content

Daily Log — 2026-07-14: A merged PR is not a deployed PR

· 4 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a debugging-heavy day at the boundary between a web frontend, an edge gateway, and the SaaS control plane it depends on. Names and tickets are private; the patterns transfer.

1. "It merged" and "it's live" are different claims — trace the pipeline​

The report: a UI change had merged, but the environment still showed the old behavior. The tempting conclusion is "the change is broken." The correct first question is "is the change even running here?"

I stopped guessing and followed the deployment path instead of the app:

  • Find the actual deploy workflow, not the CI that runs on the PR. Merging to the main branch triggers build; a separate deploy action promotes artifacts to each environment, often per-tenant and on its own schedule.
  • Compare SHAs, not PR state. The merge commit and the commit the deploy job actually shipped are different objects. Open the deploy run and read which SHA it published — that's ground truth, the PR's "Merged" badge is not.
  • Per-environment rollout means partial states are normal. One environment can be several deploys behind another. "Old behavior in tenant X" can be completely consistent with "merged and live in tenant Y."

Lesson: before debugging why a change misbehaves, prove the change is present. Ten minutes in the deploy pipeline's run history beats an hour bisecting code that was never shipped to the box you're looking at.

2. Promote a throwaway log grep into a parametrized script​

An HTTP 405 investigation meant summarizing a proxy's access logs — which method/route pairs appeared, how often, and where the rejected ones came from. I started with the usual pile of one-off grep | awk | sort and immediately hit the classic trap: the summarizer had a hard-coded path baked in from the last time someone used it.

The fix was small and worth it:

  • Take the log path as an argument, resolved with realpath, so the same script works whether you run it against a file you kubectl cp'd out of a pod or a directory of per-pod logs.
  • Keep it in the shared tools repo, not your scratch dir. The next person triaging a 405/404 should get the analysis, not reinvent the pipeline.
  • A 405 is a routing/method story. Summarizing method × path × count from the access log points straight at the mismatch (client using a verb or path the upstream doesn't accept) far faster than reading raw log lines.

Lesson: if you write the same ad-hoc log pipeline twice, spend five minutes making it a path-parametrized script in a shared repo. Triage tooling compounds.

3. Let the database tell you which query hurts​

Part of a broader "memory/latency under load" thread pointed at the primary relational database behind the control plane. Rather than theorize about which query was expensive, I opened the managed database's query insights: top statements by total execution time, over a 1-hour window and then a 7-day window.

  • Total time, not per-call time, is usually the target. A cheap query run millions of times outranks an expensive one run twice. Sort by aggregate load.
  • Widen the window to separate spikes from steady state. The 1h view shows what's hot now; the 7d view shows what's chronically heavy. A regression shows up as a shape change between them.
  • Managed insights beat guessing. Cloud SQL / RDS-style query insights already sample and aggregate; you don't need to bolt on tracing to find the top offender.

Lesson: "the DB is slow/hot" is a hypothesis. The provider's query-insights panel turns it into a ranked list in minutes — start there before instrumenting anything.

4. Know what layer your proxy actually operates at​

A side question — "can this proxy filter/inspect these requests?" — came down to whether it's a TCP (L4) proxy or an application-level (L7) HTTP proxy. It matters more than it sounds:

  • An L4 proxy forwards bytes. It can allow/deny by host:port and tunnel TLS, but it cannot see or log HTTP methods, paths, or headers.
  • An L7 proxy terminates/understands HTTP, so it can log and filter on method/path/header — which is exactly what you need to explain (or block) something like a stray 405.

Lesson: before promising a proxy "can enforce/observe X," confirm which OSI layer it works at. The answer decides whether X is a config change or an architectural non-starter.