Daily Log — 2026-07-13: You can't add a log line to a log that no longer exists

A sanitized log from a day split between a gateway observability task that turned into a regression hunt, and building a small API to actually learn Rust. Specifics are private; the patterns transfer.

1. Before you add a log field, confirm the log still exists

The task sounded tiny: "add a field to the request log that says where a value came from — upstream, in-memory cache, or the shared cluster cache." I'd already traced the exact decision point in the code and picked the log line to hang it on. Easy.

Then I went to verify which log file that line lands in on the current release, and the floor gave way: the latest major version no longer writes the per-request access log at all, and there's no knob to raise stdout verbosity. The app logs to stdout as JSON and that's it. The env var that used to enable local file logging produced nothing. Older releases persisted a proper access log; this one silently dropped it.

So the "add one field" story was resting on a substrate that had been removed. You can't append to a log line that no longer gets emitted.

The right move was not to hack the field in anyway — it was to stop, write down the regression with the evidence (which files exist, which env vars I tried, the diffing commit), and split it into its own high-priority bug that blocks the original feature. The feature ticket went to Blocked with a clean dependency edge, instead of me quietly building on sand.

Lessons:

• Verify the plumbing before you design the fixture. "Add a log/metric/field" implicitly assumes the emission path exists and is configurable. On a system you don't touch daily, confirm that assumption first — it's a five-minute check that can invalidate the whole task.
• A discovered regression is a separate work item, not a footnote. When your small task uncovers that something bigger broke, file it as its own ticket and make it an explicit blocker. Don't absorb someone else's regression into your feature's scope.
• "It used to work in the previous version" is a bug report, not a shrug. Pin the behavior change to a version/commit so the owning team has a starting point.

2. Observability on a hot path is a design constraint, not an afterthought

The reason this feature needs care: the read path in question runs thousands of times a second. The naive implementation — emit a new log line per lookup saying "served from cache/upstream" — would roughly double log volume under load. That's not observability, that's a self-inflicted incident.

The design that survives contact with production:

• Append one field to a line you already emit, don't add new lines. If there's already a structured per-request log, an extra key is nearly free; a whole new line per request is not.
• Capture the fact at the one place that knows it. The cache tier is decided in a single function — but today it returns only (value, found) and throws away which tier answered. Surface that (getWithSource) instead of re-deriving it later.
• Thread it without rewriting every signature. The retrieval code has a request context but not the request object. Stash a tiny mutable holder in the context at the middleware, write the tier into it during the lookup, read it back when building the log line. No dozen-signature refactor.
• Keep the hot path allocation-free. A small integer enum for the source, no string formatting until the log line is actually assembled, and only stamp it for the handful of cacheable read operations.

Lesson: "where did this come from?" should be a first-class, greppable field — added the way a performance engineer would add it. Provenance you can't see is provenance you can't debug during an outage, but provenance that doubles your log bill is its own outage. The interesting engineering is emitting it cheaply.

3. Learn a new language against a real, bounded task

The back half of the day I spent learning Rust — a language I hadn't written before — by building something real instead of skimming a tutorial: a small axum HTTP service that layers over a device-management API (Jamf Pro). Two endpoints — a POST /credentials that deserializes and echoes a typed body, and a GET /devices that authenticates upstream, paginates an inventory API, and returns a normalized device list with an os_is_latest flag. Coming from Go/Python, here's what actually taught me the language.

The compiler is the tutorial. You can't hand-wave Result/?, ownership, or error design when nothing compiles until you get them right:

• Immutable by default is the inverse of what I'm used to. let x is frozen; you opt into change with let mut x. References split the same way — &x is a shared reader, &mut x an exclusive writer, and the rule "many readers XOR one writer" is exactly what makes data races a compile error. In practice: a paginating loop's page/results accumulators are mut, the fixed page_size is not, and the compiler enforces the distinction.
• Modules are explicit — nothing is auto-discovered. Unlike Go (every file in a dir shares the package) a Rust file doesn't exist until a parent declares mod name;. A mod.rs turns a directory into a module and pub mod client; pulls in client.rs. Verbose at first, but the module tree becomes a precise map of the crate.
• Macros are typed codegen, not text substitution. #[tokio::main] rewrites async fn main into a runtime bootstrap; #[derive(Serialize, Deserialize)] expands ~5 lines into ~100 lines of field-by-field JSON logic; #[derive(PartialEq)] is what lets a test assert_eq! two structs. It's go generate/Python decorators, but built-in, hygienic, and type-aware.

Design decisions that make the code testable and safe:

• Separate pure logic from I/O. The mapping (inventory → Device, "is this the latest OS?") lives in pure functions with no network calls, so it's unit-testable without HTTP. The client (token auth + pagination) is a thin shell around it.
• Model only the fields you use. The upstream inventory response is enormous and full of null sections. Rather than model all of it, narrow structs deserialize just the needed fields — serde ignores unknown fields by default — and the service emits its own compact model with #[serde(skip_serializing_if = "Option::is_none")] to drop empties. No null-wrangling.
• One error type, generic to the client. An AppError enum via thiserror lets ? convert upstream/reqwest errors automatically (#[from]), and its IntoResponse maps each variant to a status code plus a generic JSON message — real details logged server-side, never leaked to the caller.
• Semantic comparison, not lexical. "Latest OS" uses the version-compare crate so 15.7.7 correctly ranks above 15.3.2 (string comparison gets this wrong). And I kept the "latest" reference an injectable parameter, so the source can change without touching the tested mapping.

Workflow that kept a learning project from collapsing into mush:

• Capture a real upstream response as a fixture. Curl the live API once, scrub the PII deterministically (index-based, so the file stays internally consistent), save the JSON, and include_str! it into the tests at compile time. Now parsing/logic is pinned without hitting the network — and a trivial "does it still parse?" test instantly catches a corrupted fixture (mine failed with expected ':' at line 9 after a bad hand-edit).
• In-file unit tests are idiomatic, not a hack. A #[cfg(test)] mod tests block at the bottom of the source compiles only under cargo test and can reach the module's private items via use super::* — closer to Go's in-package _test.go than to pytest's separate files.
• Small branches, one concept each — credentials endpoint, then devices + error handling, then tests, then persistence — each merged when green. The git history doubles as a study log.
• Use the assistant to explain, not just generate. The value wasn't the code; it was asking "why is this the idiomatic error type?", "what does this mut buy me?", "what is mod.rs vs a Go package?" on code I was about to commit — and writing the answers down in a NOTES.md.

Lesson: the fastest way to learn a language is a task with a definition of done. Constraints (a spec, a passing test suite, a compiler that won't let you cheat) turn vague "learning" into concrete reps — and keeping a running notes file of every "why?" turns the reps into something you can actually reread.

4. Confirm the fix landed with the person who reported it

A separate incident — a spike in authentication failures for a heavy user — got its confirmation today: the reporter verified it was resolved on their side, and I closed it. The failure had traced to expired short-lived credentials being rejected, not the scary-looking-but-flat certificate errors that were also in the logs.

Lesson: "probably fixed" isn't closed. The distractor signal (a constant stream of unrelated errors) was there the whole time and was not the cause — comparing its rate before and after the spike ruled it out. Resolution is a state the reporter confirms, not one you infer from your own dashboards.