delivery — heyimusa

I built a small linter for health checks that disagree with rollouts

Mon, 20 Jul 2026 13:26:02 +0000

Original diagram for this Docker-only build note. It shows a configuration disagreement, not a production incident.

I kept tripping over the same sentence while writing about health checks: “we gate the rollout on readiness.”

It sounds reassuring. Then you open the deployment file and find a healthcheck that calls /healthz, because that endpoint was easy to add and it returns 200 as long as the process is alive.

Neither file is necessarily wrong on its own. Together, they can tell two different stories.

So I made a small CLI called probe-contract. It compares a declared health contract with a Docker Compose file and reports the disagreements that are easy to miss in review.

The first release is deliberately small. It does not call a live endpoint. It does not attempt to infer whether a checkout flow, queue, or database is truly healthy. It reads configuration and asks a narrower question: does the probe in the deployment file support the health assumption you wrote down?

The smallest useful contract

The tool takes a tiny YAML file alongside Compose:

services:
  api:
    liveness: /healthz
    readiness: /readyz
    user_path: /checkout
    rollout_gate: readiness

Then it compares that declaration with the Compose healthcheck.

For the mismatch case, the contract said readiness should gate the rollout, while the Compose probe called only /healthz:

WARNING  api: rollout_gate is readiness, but Compose healthcheck does not reference /readyz (READINESS_NOT_PROBED)

That warning is the whole point of the first version. It does not prove /readyz is a good readiness endpoint. It makes the disagreement visible before somebody treats a green container as proof that a rollout is safe.

I tested the tool in a container too

I ran the released CLI against its fixtures inside a Docker container with a read-only filesystem, a non-root user, dropped capabilities, and no-new-privileges.

The valid fixture produced empty JSON:

{
  "diagnostics": []
}

A Compose file with its healthcheck explicitly disabled produced an error and exit code 1:

ERROR  api: service has no Compose healthcheck (HEALTHCHECK_MISSING)

That second case mattered. An early review of the tool found that healthcheck: { disable: true } could look like an active healthcheck to a naive YAML parser. The release now treats both disable: true and Docker’s test: [NONE] form as missing checks.

The review also caught a less obvious false pass: /ready should not match /readyz just because one string contains the other. That is fixed too. A health contract is already an approximation; the checker should not add accidental ambiguity on top.

What it checks today

probe-contract v0.1.0 checks a few things and stops there:

a service declared in the contract is present in Compose;
that service has an active healthcheck;
a readiness rollout gate actually probes the declared readiness path;
liveness and readiness are not declared as the same endpoint by accident;
interval, timeout, and retries are present on the healthcheck.

It emits human-readable text by default and JSON with --format json, which is enough to start using it in CI without making every warning a release blocker.

What it intentionally does not do

I do not want this to become another linter that promises too much.

It does not inspect Kubernetes manifests yet. It does not send traffic to a live service. It does not infer a business transaction from a URL. And it cannot tell whether a dependency should be part of readiness for a particular application.

Those choices belong to the team running the service. The tool only asks them to make the choice explicit, then checks whether the Compose file agrees.

That scope is small enough to be useful. It is also small enough that a reviewer can understand what a warning means without trusting a black box.

Release and next steps

The project is public under MIT and includes a Linux amd64 binary with a checksum:

The next likely steps are Kubernetes manifest support, SARIF output, and a GitHub Action. I am deliberately not calling those features until the Compose contract is useful enough to earn them.

Test notes: I built and ran probe-contract v0.1.0 only in Docker for this note. The demonstration container used a read-only filesystem, non-root UID 65532, dropped Linux capabilities, no-new-privileges, and read-only mounted fixture files. The results are configuration checks against local fixtures, not a benchmark or a production deployment.

I made a healthy service page on purpose

Mon, 20 Jul 2026 10:12:13 +0000

Original diagram for a Docker-only experiment in this note. It models a failed dependency, not a production system.

I made a service look healthy on purpose.

Not healthy to a user. Healthy to the one check that only cared whether the process was still running.

It is an easy trap. A container is up, the liveness endpoint returns 200, the dashboard stays green, and everyone gets to feel better for a few minutes. Meanwhile the dependency the service needs has gone away, requests are timing out, and the thing users came for is unavailable.

I built a tiny version of that failure in a disposable Docker container. The service had three endpoints:

/healthz   process liveness
/readyz    dependency-aware readiness
/checkout  a small user-facing path

With the simulated dependency available, all three were fine:

/healthz   http=200 body=process=up
/readyz    http=200 body=dependency=reachable
/checkout  http=200 body=checkout=accepted

Then I restarted the same demo with one environment flag that simulated a dependency failure. The Python process did not crash. That is important.

/healthz   http=200 body=process=up
/readyz    http=503 body=dependency=unavailable
/checkout  http=503 body=checkout=unavailable: dependency timeout

The liveness check was not lying. It answered the question it was given: is the process alive? The problem is that it was the wrong question for deciding whether to send a user more traffic, or whether a rollout was safe to keep.

I find it useful to separate health checks by the decision they support.

/healthz asks whether restarting the process is likely to help. A process that cannot accept a TCP connection, is deadlocked, or has stopped responding belongs here. This check should be narrow. If every temporary dependency wobble makes it fail, an orchestrator can turn a recoverable outage into a restart loop.

/readyz asks whether the instance should receive work. In the experiment, it included the dependency state. That made it a reasonable signal for traffic admission and rollout progress. A failing readiness check can take an instance out of rotation without pretending that the process needs to be killed.

/checkout asks the least convenient question: can someone complete the thing they actually came to do? It is usually more expensive to measure and should not become a noisy probe that hits every dependency every second. But some form of user-path signal belongs in monitoring. Otherwise a green fleet can hide a useless product.

A green container is a weak promise

Container state is still useful. It tells you whether a workload exists and whether the runtime can keep it alive. It does not tell you that the application has a connection pool, that the queue is moving, or that an important request succeeds.

That distinction matters most during a deployment. If a new version starts successfully but cannot talk to its required dependency, a liveness-only check may let the rollout continue. The failure then moves from deployment time to user time, where it is noisier and harder to unwind.

This is also why I do not like treating readiness as a cosmetic endpoint added late in a project. It is part of the contract between the application and the platform. The platform needs an honest answer before it decides to route traffic or declare a revision ready.

Do not turn readiness into a dependency census

There is a bad version of this pattern too: make /readyz call every downstream service, every time, and fail on any brief hiccup.

That can create its own outage. If a shared dependency has a short blip, hundreds of instances may all become unready at once. A probe that was meant to reduce risk becomes a traffic switch with no damping.

The useful questions are smaller:

Which dependency makes this instance incapable of doing its primary job?
How long must that dependency be unavailable before traffic should stop?
Can the check use a bounded timeout and cached result instead of adding load during an incident?
What signal tells us the user path is degraded even if the process remains alive?

The answers will differ by service. A worker may be alive and intentionally disconnected while a queue is paused. A checkout API with no database connection is not in the same situation.

What I would wire into a real rollout

For a non-critical environment, I would start with a deliberately boring drill:

deploy a candidate revision;
make one required dependency unavailable in a controlled way;
confirm that readiness fails while liveness stays stable;
confirm that the traffic or rollout controller reacts to readiness, not just process survival;
restore the dependency and verify the user-facing path, not only the pod state.

That drill has a nice side effect: it makes the rollback criteria concrete. In my previous note, I argued that a rollout needs a return address. This is part of the address book. If the only evidence after a rollback is “the pods are running,” there is still a lot left to guess.

Test notes: I ran this in two disposable Docker containers using Python 3.12. Each container used a read-only root filesystem, dropped Linux capabilities, no-new-privileges, a PID limit of 64, 0.25 CPU, and 128 MiB memory. The dependency failure was a local simulation. No host service, production endpoint, cluster, or external dependency was accessed.