Skip to main content

Daily Log — 2026-07-17: The fix for an out-of-memory path is often to do less, not more

· 3 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized, shorter-Friday log centered on one fix — making a read path degrade gracefully when its upstream is down — plus a bit of lab plumbing and a cold-storage restore. Specifics are private; the patterns transfer.

1. Graceful degradation means serving less, on purpose​

The bug: when an edge service's upstream control plane was unreachable, a "describe this one item" request went down a path that ballooned memory — the worst possible behavior exactly when the system is already unhealthy. The fix wasn't a clever new allocator; it was serve the single item from the local proactive cache and stop there when the upstream is known to be down.

The design principle behind the one-line-sounding change:

  • Fail toward the cheap, bounded answer. When the expensive path (go to the control plane, possibly fan out, possibly retry) can't work, don't attempt a degraded version of it that still allocates unboundedly. Return the one cached item you already have, or a clean "unavailable."
  • Scope the fallback to exactly what was asked. "Look up a single item from cache" is bounded by definition. The failure mode was code that, when the lookup missed upstream, tried to do far more work than the request required.
  • Read the cache layer's contract before you lean on it. I spent time in the cache package's API/docs first — what its Get returns, whether it distinguishes in-memory from shared tiers — because "just read from cache" is only safe if you know what the cache promises under a miss.

Lesson: the fix for an OOM-prone path under failure is usually do less: a bounded, cached, single-item answer beats a heroic best-effort that keeps allocating while the dependency is down. Degradation is a feature you design, not an accident you tolerate.

2. A targeted fix belongs on a targeted branch​

This shipped as a small, single-purpose change on a branch cut from the relevant base (the branch where the surrounding cache rework lives), not from the default branch. That sounds like process trivia; it isn't:

  • Branch from where your change has to live. If your fix depends on in-flight cache changes, basing it on main gives you a PR that won't behave like the target and a nightmare merge later. Base it on the branch that actually carries the context.
  • One behavior per PR. "When upstream is down, serve one item from cache" is a reviewable, revertable unit. Bundling unrelated cleanup into it would bury the one change reviewers must scrutinize.
  • Watch your own CI before declaring done. I sat on the PR's pipeline runs rather than assuming green — a memory-behavior change is exactly the kind that a test env catches and a laptop doesn't.

Lesson: small blast radius, correct base branch, one behavior per PR. It makes review honest and rollback trivial.

3. Cold-storage restores are asynchronous — kick them early​

A minor task: an archived object in object storage had aged into a cold/Glacier tier and needed to be readable again. The thing to internalize is that a restore from archival storage is not instant — you request it, and the object becomes available minutes to hours later depending on the retrieval tier.

Lesson: when a workflow needs an archived object, issue the restore first thing and go do something else; don't discover the multi-hour wait at the moment you need the data. And if objects you read routinely keep landing in cold storage, that's a lifecycle-policy smell worth fixing at the bucket, not one restore at a time.