No errors is not a health signal: monitoring the work that stopped happening

heyimusaReading

A service can be quiet for two very different reasons. Maybe it is healthy and nobody is using it. Maybe the checkout pipeline stopped or the nightly report never ran. Error rate can still be zero.

That distinction matters because most alert rules start with an error condition. 5xx exceeds a threshold. Queue depth grows. Latency crosses a limit. Those are useful alerts. They answer a narrow question: did the system produce evidence of a bad thing?

They do not answer a second question that is often closer to the outage: did the work we expected actually happen?

An error-rate threshold remains green while an expected heartbeat stops; a freshness window later triggers an alert.

A quiet graph is ambiguous

I have seen dashboards that looked reassuring precisely because the failing path had stopped producing the data that drove the dashboard. There was no surge of errors. There was no obvious red line. There was just a missing stream of successful work.

That can happen in ordinary ways:

  • a scheduled job exits before its success metric is emitted;
  • a queue consumer is alive but no longer receives messages because a subscription, credential, or routing rule changed;
  • an analytics or billing pipeline keeps its workers up while new input stops arriving;
  • a deployment routes traffic somewhere unexpected, so the old service has clean metrics but the intended user path sees nothing useful;
  • the telemetry path fails, leaving the application and the monitor to disagree about what exists.

The first mistake is to treat absence as a universal failure. A low-traffic service may legitimately have no requests for an hour. A batch job may only be expected once per day. Kubernetes workloads can disappear because a deployment intentionally scaled down. The contract has to name the thing that should recur and the time in which it should recur.

“Zero requests” is not enough. “The reconciliation job that normally completes every fifteen minutes has not recorded a successful completion for thirty minutes” is an operational statement.

Expected work needs a freshness contract

For every path worth paging on, I want four answers written down:

  1. What event proves the path completed the useful work?
  2. Who or what emits that evidence?
  3. How late can it be before someone needs to decide something?
  4. What action should the alert receiver take first?

That evidence can be a timestamp gauge, a monotonically increasing completion counter, an externally observed heartbeat, a synthetic transaction, or a record in a downstream system. The mechanism matters less than the contract. The signal should describe completion of the work, not merely survival of the process assigned to do it.

A readiness endpoint might prove that a web process can accept a request. It does not prove that a daily export still reaches its destination. A successful Kubernetes rollout shows that the workload reached its configured rollout conditions, such as updated and available replicas. It does not prove that the business event after that rollout still travels through every dependency.

Google’s SRE workbook makes the related point about monitoring-data freshness: stale data slows incident response and can lead people to act on incorrect information. That applies to a dashboard, but it also applies to the system being observed. If the evidence of a required activity is stale, the operator should not have to infer health from a lack of complaints.

A small Docker experiment

I built a disposable Docker Compose experiment with two containers: a tiny HTTP exporter and Prometheus. For this test, the exporter served a heartbeat gauge whose value was the current Unix time on every scrape:

job_heartbeat_last_success_unixtime{job="nightly-report"} <current-unix-time>

Despite the metric name, this was deliberately only a scrape-path signal. It was not evidence that a nightly report completed. A real completion metric must update only after the job has completed its meaningful work.

Prometheus scraped it every second. The alert rule was deliberately about absence, not a numeric threshold:

absent_over_time(
  job_heartbeat_last_success_unixtime{exported_job="nightly-report"}[5s]
) == 1

The rule waited another two seconds before firing. The timings are intentionally unrealistically short so the behavior fits in a small test; they are not a production recommendation.

The healthy state produced one scrape target with health=up and no firing alert:

target_health=up
healthy_firing_alerts=0

Then I stopped only the exporter container. Prometheus remained running. Once no matching sample remained in the five-second range, the next rule evaluation made the alert pending. After the condition had remained true for at least for: 2s across subsequent evaluations, it fired. Exact wall-clock timing depends on scrape and rule-evaluation alignment:

missing_firing_alerts=1
alert_state=firing
alertname=ExpectedHeartbeatMissing
severity=page

This selector was intentionally scoped so the absence function yielded one synthetic alert. It was not a per-target missing-series detector.

That proves a narrow thing: for this scrape-path metric and this rule, Prometheus can turn missing scrape-visible data into an alert. It does not prove that a real nightly report completed, that its output was correct, or that a production notification channel would wake the right person. Those require a stronger, path-specific check.

The experiment also surfaced an easy-to-miss detail. With Prometheus’s default honor_labels: false, the scrape configuration owns the target’s job label. Because the exporter also exposed job="nightly-report", Prometheus renamed the scraped label to exported_job="nightly-report". A configuration using honor_labels: true behaves differently. That is exactly the kind of label collision that should be caught in a test before an absence alert earns trust.

No data is not one state

Missing evidence has several meanings, and collapsing them into one generic “no data” alert creates noise.

Grafana distinguishes at least three cases: a query error, no data at all, and a missing series where other series still exist. Those need different responses. A datasource failure may be a monitoring incident. One missing regional series might be a workload problem. A deliberately scaled-down job could be expected behavior.

Cloud Monitoring has a useful guardrail for metric-absence policies: it requires at least one successful measurement after the policy is installed or changed. Otherwise a never-emitted metric would look exactly like a broken metric. Prometheus does not enforce that guard. An absent_over_time rule can become true when a matching series has never existed. In this experiment, the healthy pre-check established prior observation manually. In production, pair the detector with an expected-membership source, an initialization process, or an alerting policy that accounts for newly introduced signals.

Prometheus’s absent() and absent_over_time() are useful building blocks, but they do not replace that reasoning. A broad expression can tell you that nothing matched. It cannot decide whether the missing series was intentionally removed, whether a new label should exist, or whether an upstream event actually reached the customer-facing outcome.

Where absence checks earn their place

I would add an absence or freshness check when the missing work has a clear owner and a bounded recovery action. Some examples:

Expected activity Useful evidence First response
Scheduled backup Timestamp of a verified successful backup Check scheduler, credentials, storage destination, and restore metadata
Payment/webhook ingestion Recent accepted event counter or end-to-end synthetic event Check provider delivery, queue/subscription, and consumer lag
Reconciliation job Last successful completion timestamp and record count Check source availability, job logs, and output freshness
Telemetry pipeline Recent collector/exporter heartbeat plus ingestion signal Separate application health from observability-path health
Customer route after a deploy Bounded synthetic request with an expected semantic response Check DNS, TLS, routing, auth boundary, and backend selection

The table is not a template for paging on everything. It is a prompt to state the unit of useful work. A low-volume workflow may need a scheduled expectation. A high-volume request path may need a rate floor with a seasonality-aware baseline. An external dependency may need an independent probe because a heartbeat emitted from inside the same failure domain can lie with the rest of the system.

Do not turn freshness into another noisy threshold

Absence alerts get bad quickly when they ignore operating context.

Start with a generous window. A job expected every fifteen minutes should not page at minute sixteen if the scheduler occasionally starts late and the business can tolerate that delay. Make the window reflect the actual deadline, not the nominal schedule. Include a warning before a page when that helps someone intervene early.

Keep labels that identify the owner and expected cadence in the alert design, but be precise about the PromQL behavior. For absent_over_time alerts, put stable identity labels needed for routing in equality matchers or static rule labels. Labels that existed only on the vanished series, such as an ephemeral pod or instance, cannot be preserved by the absence result. A broad absence query produces one synthetic alert, not one alert for each missing target.

For dynamic fleets, use a source of expected membership: a deployment desired-replica signal, a durable job registry, or a service-level aggregation. Otherwise scale-down and churn become indistinguishable from loss.

Most importantly, keep the alert tied to a decision. “No data from metric X” is a debugging clue. “Nightly report did not complete by 06:30 UTC; customer statements may be stale; inspect run ID, source freshness, and destination write” gives the responder somewhere to begin.

The check after the fix matters

When the path is repaired, the dashboard often returns to green before the contract has been met. The process restarts. The consumer reconnects. The queue drains a little. None of that proves the missing work has completed.

The recovery check should ask for fresh evidence:

  • Did a new completion timestamp arrive?
  • Did a known synthetic transaction cross the boundary and produce the expected downstream record?
  • Did the next scheduled run finish within its window?
  • Did the telemetry signal return independently of the application’s own health endpoint?

That is also the reason I prefer “last successful completion” over a generic process heartbeat for scheduled work. A process can be alive all day without doing the job it exists to do.

A green dashboard is a claim with limits

Threshold alerts catch bad values. Absence checks catch missing evidence. Synthetic checks catch whether an important path still behaves as a user or dependency expects. None is sufficient on its own.

The practical goal is smaller than complete observability. Pick the work whose silence would matter, give it an explicit freshness contract, and test the detector by making that evidence disappear in a safe environment. Then make the alert useful enough that the person receiving it knows what decision belongs to them.

No errors is not a health signal. Sometimes it only means nobody is left to report the failure.

Sources