Part 3 of Operating AI Agents Like Production Systems. A green HTTP check can tell you that an agent endpoint answered. It cannot tell you whether the agent completed any useful work, whether it is stuck behind approval, or whether the tool runner stopped executing actions.
I built the last fixture in this series around that distinction. It counts completed executions, review-queue items, and total requests. The input is intentionally tiny: synthetic event records, not model traces or production telemetry. The goal was to make one quiet failure visible: every request can receive an answer while the execution rate stays at zero.
That failure is easy to miss if the dashboard only tracks request success and latency. The endpoint is alive. Users may even see polished answers. The business action never happened.
A status code is the wrong finish line
The synthetic health function marks a stream as unhealthy when it has events but none completed execution. It marks a stream degraded when work completes but the review queue has outstanding items.
The demo had one completed billing action and one pending-review action:
{
"executed": 1,
"execution_rate": 0.5,
"review_queue": 1,
"status": "degraded",
"total": 2
}
That is not a universal threshold. A support agent may correctly send many requests to review. A deployment agent may execute rarely by design. The useful point is that execution and review backlog need their own signals. Neither can be inferred from HTTP success.
The isolated test
I added two health tests to the same local fixture used in the earlier posts. One feeds two answered-but-not-executed events and expects unhealthy. The other feeds one execution and one review item and expects degraded.
test_silent_nonexecution_is_unhealthy_even_when_requests_succeed ... ok
test_completed_work_with_one_review_queue_item_is_degraded ... ok
Ran 8 tests in 0.001s
OK
The test ran in python:3.12-alpine with no network, read-only root filesystem, no Linux capabilities, no-new-privileges, a 16 MiB temporary filesystem, 128 MiB memory, one CPU, a 64-PID cap, and UID/GID 65532.
Signals I would add first
Start with a request-to-outcome view rather than a token or latency view:
- count proposed actions by operation and policy result;
- count approvals, denials, expiries, and review age;
- count tool executions and tool failures separately from model responses;
- record the ratio of executed actions to eligible actions;
- sample evidence bundles for missing or stale references.
Keep the model's response, policy decision, approval, and execution as linked events under one request ID. Otherwise operators end up correlating a chat log, approval queue, and tool log by timestamp during an incident.
The boundary of this result
This experiment does not recommend alert thresholds or prove a monitoring stack. It has no distributed traces, no retries, no queue, no real tool integration, and no model. It only demonstrates that an operational health rule can catch a silent nonexecution state that request success misses.
Real systems also need to distinguish intentional waiting from blocked work. A review queue may be healthy if its oldest item is five minutes old and unhealthy if it is five days old. An idempotent retry may be safe for one tool and dangerous for another. Instrument the state transitions before deciding which counter should page someone.
Closing the loop
The series began with evidence because a confident answer is not a reviewable action. It added approval binding because a generic approval is not authorization for an arbitrary payload. Health comes last because both controls can still fail quietly in production.
I would ship the small contracts first, then make their transitions observable. Fancy agent orchestration can wait. It is much harder to retrofit proof of what happened after the system has already learned how to act.
Continue the series
- Part 1: Build an evidence gate
- Part 2: Bind approvals to the exact action
Test notes: The health rules are local synthetic Python code in experiments/agent-operations/; all eight tests ran in a restricted Docker container. No model, telemetry backend, credentials, or external API was used.
Sources: OpenTelemetry semantic conventions (instrumentation context) · No errors is not a health signal: monitoring the work that stopped · An agent needs an evidence bundle, not a confident answer.