Daily Log — 2026-07-08: the flat line was a counter, and other Prometheus lies

A sanitized log from a day spent building an observability stack to chase an error-rate spike — and learning (again) that a metric that looks calm is often just a counter you're reading wrong. Specifics are private; the patterns transfer.

1. A "flat line" is usually a counter you forgot to rate()

I was staring at an HTTP-status metric in Prometheus, trying to see a spike of 401s. The graph showed a dead-flat line sitting at ~38 no matter the time range. My first instinct was "the instrumentation is broken." It wasn't — I was.

The metric was a cumulative counter: total responses since the process started. A counter only ever goes up (or resets on restart), so plotting its raw value gives you a slowly-rising or apparently-flat line, never the per-second shape you actually care about. To see "401s per minute" you have to differentiate it:

# wrong: raw counter — looks flat, tells you nothing about the current rate
akeyless_gw_system_http_response_status_code{status="401"}

# right: per-second rate over a 1m window
rate(akeyless_gw_system_http_response_status_code_total{status="401"}[1m])

# or, "how many in the last 5 minutes"
increase(akeyless_gw_system_http_response_status_code_total{status="401"}[5m])

Two things that bit me and are worth internalizing:

• Counters carry a _total suffix by convention. If a bare metric name gives you a flat line, check whether the real series is <name>_total — Prometheus client libraries append it, and the UI will happily autocomplete the un-suffixed name to something less useful.
• rate() vs increase(): rate is per-second (good for graphs and alerts), increase is the count over the whole window (good for "how many happened"). Both need a range vector ([1m]), and both handle counter resets for you — which is exactly why you should never subtract counter values by hand.

Rule of thumb: if a Prometheus panel is suspiciously flat or monotonic, assume it's a counter and wrap it in rate() before you go blaming the exporter.

2. The same bug, one layer up: dashboards that plot counters raw

Fresh off that, I noticed a published Grafana dashboard whose network I/O panels had the identical problem — they graphed cumulative byte counters, so every line just climbed forever and told you nothing about current throughput. Fixing it was the same move as above: wrap the series in rate(...) so the panel shows bytes/sec instead of bytes-since-boot.

Lessons:

• Audit inherited dashboards. A dashboard someone else published (or that you imported from a gallery) is not automatically correct. Cumulative-vs-rate is one of the most common mistakes, and it hides in "nice looking" panels that are quietly meaningless.
• Publishing a revision needs ownership. Pushing a fix back to a shared dashboard gallery requires access to the account/org that first published it. Fixing the JSON is the easy 20%; getting it back upstream so everyone benefits is the other 80%.

3. Exposing internal cluster tooling behind ingress + TLS

To investigate any of this I first needed the metrics stack reachable. Grafana and Prometheus were running in-cluster (a monitoring namespace), only reachable via kubectl port-forward — fine for a minute, miserable for a day of query iteration. I put them behind an ingress with real hostnames and reused an existing wildcard TLS cert already stored as a Kubernetes secret.

A few practical notes:

• Reuse the wildcard cert you already have. If you've got a *.example.dev cert in a secret, ingress can reference it directly — no need to provision a new cert per subdomain.
• Trim what you expose. A monitoring helm chart ships a pile of extra services (alertmanager, pushgateway, headless variants). If you're only exposing the UI, delete the services you don't need rather than leaving them dangling.
• /etc/hosts bridges the DNS gap. While real DNS propagates, a temporary hosts entry lets you hit the new hostname immediately and confirm the ingress + TLS actually work end-to-end.

4. Reproduce the signal before you trust the panel

Once the panels were "fixed," how do I know they're right? I wrote a tiny script to generate the exact errors on demand — fire a controlled stream of failing requests at a lab instance, with a knob for interval (steady drip vs. burst). Then I watched the rate(...) panel react in real time.

This closes a loop that's easy to skip: a green dashboard is only trustworthy if you've seen it move when you made the thing it measures happen. Synthetic load against a lab environment is cheap insurance against shipping a panel that's silently stuck.

5. Understand why a resource exists before you delete it

Unrelated thread: a Helm chart was rendering a Kubernetes Service that seemed redundant — it got created no matter how the chart was configured. The lazy fix is "delete it." The right first step was to find out who consumes it: it turned out an optional feature (a bastion/remote-access component) points an internal API env var at that service. So it's not redundant — it's conditional.

The fix wasn't removal, it was making the resource track the feature flag: only render the service when the feature that needs it is enabled.

{{- if .Values.someFeature.enabled }}
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}-internal
# ... only exists when the consuming feature is on
{{- end }}

Transferable takeaways:

• "Unused" is a hypothesis, not a fact. Grep for the resource's name across the chart/templates before you decide it's safe to drop — env vars and downstream components reference services by name.
• Prefer conditional rendering over deletion. Gating a resource behind the flag that needs it keeps the default lean without breaking the users who depend on it.