Skip to main content

Daily Log — 2026-07-07: reproducing a 401, and CI logs you can read without logging in

· 5 min read
Kobbi Gal
Kobbi Gal
I like to pick things apart and see how they work inside

A sanitized log from a day spent chasing a spike of "credentials expired" 401s that nobody could attribute to a client, plus a responsible-disclosure detour when I noticed a CI provider's raw job logs were readable without authentication. Specifics are private; the patterns transfer.

1. When the error is real but you can't tell who hit it​

A service started returning a surge of 401 credentials have expired — roughly a 10x jump, from a low steady-state to a few hundred per minute, starting at a specific hour. Classic incident shape: the error is genuine, but the log line that emits it says what failed and not who.

The investigation had two useful moves worth stealing:

  • Rule out the noisy neighbor. The same logs were full of unrelated auth failures (an expired cert on a downstream dependency). Tempting to blame, but the fix is to compare volume before and after the spike: those errors were flat across the boundary, so they were pre-existing noise, not the cause. Don't anchor on the loudest error — anchor on the one whose rate changed.
  • Reproduce the exact string. I stood up my own instance of the service, tailed its trace log, and fired a request with a deliberately bogus expired-style credential:
curl -s -w "\nHTTP %{http_code}\n" -X POST "https://<my-instance>/get-secret-value" \
  -H "User-Agent: repro-test/1.0" \
  --data-urlencode "token=t-1111...1111" \
  --data-urlencode "name=/anything"

Getting the identical error message locally is the whole game: it proves your mental model of the code path and gives you a lab to test fixes in, instead of guessing against production.

The real gap turned out to be observability, not auth. The endpoint that produced the 401 had been excluded from both audit and access logs, so there was no record of which caller was replaying the stale credential — the team was, in their words, totally blind. The fix I put up enriches the client error itself with the identifying signal the logs were missing: source address, User-Agent, the originating command, and a SHA of the token (never the raw token). That's enough to fingerprint the misbehaving client without ever writing a secret to a log.

Reusable checklist for "the error is right, the attribution is missing":

  • Diff error rates across the incident boundary before blaming any single message.
  • Reproduce the literal error string in a lab before proposing a fix.
  • If you can't answer "which client caused this?", that's the bug to fix first — add the identifier (hash the sensitive parts), don't log the secret.
  • Audit your log exclusions. An endpoint quietly excluded from audit/access logs is a blind spot you'll only discover mid-incident.

2. CI job logs you can read without logging in​

While pulling the raw logs for one of my own pipeline runs, I noticed the log isn't served by the CI platform directly — it's a redirect to an object-storage URL carrying a pre-signed (SAS) token in the query string (?...&sig=...&se=...&sp=r). Out of curiosity I curl'd that URL with no cookies, no token, no session:

curl -s "https://<blob-store>/actions-results/<ids>/logs/job/job-logs.txt?<sas-query>"

It returned the full job log — no platform authentication required. The signed URL is the credential, and it's reachable from a redirect on the run page. For a public repo that's arguably expected; the sharp edge is that the same pattern means anyone who obtains the link (logs, referrers, screenshots, browser history sync) reads the raw logs, which routinely contain more than people assume.

I wrote it up and reported it through the platform's coordinated-disclosure channel (a dedicated security mailbox) rather than posting the details publicly. Keeping specifics out of this post is deliberate — that's the responsible-disclosure part.

Transferable lessons:

  • Pre-signed URLs are bearer credentials. Anyone with the link has the access the signature grants, for as long as it's valid. Treat them like short-lived secrets: short TTLs, never in shared logs, referrers, or chat.
  • CI logs leak. Build logs are a favorite hiding spot for tokens, internal hostnames, and env dumps. Assume any log that can be linked can be read.
  • Report before you publish. If you trip over something exploitable, use the vendor's private security channel first and sit on the details.

3. Restoring cold data mid-incident​

The investigation needed audit history that had already aged into a cold/archive storage tier — the kind you can't read directly and have to restore first. Two reminders that always bite under incident pressure:

  • Archive tiers are not instant. A restore request can take minutes to hours before the object is readable; kick it off early rather than discovering the latency when you're blocked on it.
  • Retention policy is an incident-response decision. If the logs you need to debug a spike have already been tiered into deep-freeze, your effective debugging window is shorter than your retention number suggests. Worth revisiting what stays "warm" and for how long.

4. A morning of reading the manual​

Not every block is heads-down debugging. I spent the morning re-reading product and platform docs — auth methods, RBAC and sub-claims, gateway/caching/telemetry behavior, secret injection — and the internal onboarding material. Low-glamour, high-leverage: the 401 investigation later leaned directly on knowing which auth path emits which error, and that only felt fast because the concepts were fresh. Scheduling a "reload the fundamentals" block near work you'll do anyway pays for itself the same day.