Part 1 of Operating AI Agents Like Production Systems. A helpful agent answer and an operable agent action are different artifacts. Before an agent can change an external system, a reviewer needs the input references, policy result, exact proposed action, approval state, and execution state.
I built a tiny synthetic refund workflow because it makes the gap obvious. A model can say, “The $240 refund was submitted,” yet leave no way to tell which rule it applied, whether a person approved the charge, or whether the billing tool ran. That sentence may be useful to a customer. It is useless in a review.
The post that prompted this series argued for evidence bundles. This one turns the idea into a small contract and a test. I ran it in a locked-down Docker container against synthetic JSON only. It did not call a model, payment provider, approval service, or production API.
The contract is deliberately boring
The synthetic record needs five fields besides its request ID and final answer: input_refs, policy_decision, proposed_action, approval, and execution. The validator does not decide whether a refund is morally or commercially correct. It answers a narrower question: can someone inspect the state needed to challenge the claim?
A response-only record fails the check because it cannot answer that question:
{
"decision": "needs-review",
"missing": [
"input_refs",
"policy_decision",
"proposed_action",
"approval",
"execution"
]
}
A complete bundle with an approval still pending also returns needs-review. Structure is necessary, but it is not permission to execute.
The Docker experiment
I wrote eight unit tests and ran them inside python:3.12-alpine. The container had no network, a read-only root filesystem, a 16 MiB temporary filesystem, no Linux capabilities, no-new-privileges, a 64-PID cap, 128 MiB of memory, one CPU, and UID/GID 65532.
Ran 8 tests in 0.001s
OK
The demonstration used a synthetic $240 refund. Its policy requires human approval above $100. The same fixture is used for the rest of this series, so the boundary stays clear: these are record and policy tests, not payment-system tests.
What this proves
The validator caught the missing review fields and refused to treat a pending approval as executable. That is the useful part. A chat response can remain short while the system keeps a separate record for an operator.
It does not prove that the policy is correct, that references are fresh, that approval identity is trustworthy, or that an execution record came from a real provider. Those are separate controls. A JSON schema is not an audit system just because it has an execution key.
Use the gate before the tool call
Put this check between agent reasoning and the side-effecting tool. If the bundle is incomplete, return a review state rather than making the tool call. Store immutable references where possible, including the policy version and the exact action payload.
The next post tightens the other half of the boundary: an approval must bind to the action that will run. A general “looks good” comment should not authorize a different refund or a destructive operation.
Continue the series
Test notes: The code is a local synthetic Python fixture in experiments/agent-operations/, tested in Docker with no network or external credentials. It validates small in-memory records only.
Sources: An agent needs an evidence bundle, not a confident answer · OpenAI Presence announcement (context for approved actions; not exercised by this experiment).