Original diagram for a Docker-only experiment in this note. It models a failed dependency, not a production system.
I made a service look healthy on purpose.
Not healthy to a user. Healthy to the one check that only cared whether the process was still running.
It is an easy trap. A container is up, the liveness endpoint returns 200, the dashboard stays green, and everyone gets to feel better for a few minutes. Meanwhile the dependency the service needs has gone away, requests are timing out, and the thing users came for is unavailable.
I built a tiny version of that failure in a disposable Docker container. The service had three endpoints:
/healthz process liveness
/readyz dependency-aware readiness
/checkout a small user-facing path
With the simulated dependency available, all three were fine:
/healthz http=200 body=process=up
/readyz http=200 body=dependency=reachable
/checkout http=200 body=checkout=accepted
Then I restarted the same demo with one environment flag that simulated a dependency failure. The Python process did not crash. That is important.
/healthz http=200 body=process=up
/readyz http=503 body=dependency=unavailable
/checkout http=503 body=checkout=unavailable: dependency timeout
The liveness check was not lying. It answered the question it was given: is the process alive? The problem is that it was the wrong question for deciding whether to send a user more traffic, or whether a rollout was safe to keep.
Three questions that should not share one endpoint
I find it useful to separate health checks by the decision they support.
/healthz asks whether restarting the process is likely to help. A process that cannot accept a TCP connection, is deadlocked, or has stopped responding belongs here. This check should be narrow. If every temporary dependency wobble makes it fail, an orchestrator can turn a recoverable outage into a restart loop.
/readyz asks whether the instance should receive work. In the experiment, it included the dependency state. That made it a reasonable signal for traffic admission and rollout progress. A failing readiness check can take an instance out of rotation without pretending that the process needs to be killed.
/checkout asks the least convenient question: can someone complete the thing they actually came to do? It is usually more expensive to measure and should not become a noisy probe that hits every dependency every second. But some form of user-path signal belongs in monitoring. Otherwise a green fleet can hide a useless product.
A green container is a weak promise
Container state is still useful. It tells you whether a workload exists and whether the runtime can keep it alive. It does not tell you that the application has a connection pool, that the queue is moving, or that an important request succeeds.
That distinction matters most during a deployment. If a new version starts successfully but cannot talk to its required dependency, a liveness-only check may let the rollout continue. The failure then moves from deployment time to user time, where it is noisier and harder to unwind.
This is also why I do not like treating readiness as a cosmetic endpoint added late in a project. It is part of the contract between the application and the platform. The platform needs an honest answer before it decides to route traffic or declare a revision ready.
Do not turn readiness into a dependency census
There is a bad version of this pattern too: make /readyz call every downstream service, every time, and fail on any brief hiccup.
That can create its own outage. If a shared dependency has a short blip, hundreds of instances may all become unready at once. A probe that was meant to reduce risk becomes a traffic switch with no damping.
The useful questions are smaller:
- Which dependency makes this instance incapable of doing its primary job?
- How long must that dependency be unavailable before traffic should stop?
- Can the check use a bounded timeout and cached result instead of adding load during an incident?
- What signal tells us the user path is degraded even if the process remains alive?
The answers will differ by service. A worker may be alive and intentionally disconnected while a queue is paused. A checkout API with no database connection is not in the same situation.
What I would wire into a real rollout
For a non-critical environment, I would start with a deliberately boring drill:
- deploy a candidate revision;
- make one required dependency unavailable in a controlled way;
- confirm that readiness fails while liveness stays stable;
- confirm that the traffic or rollout controller reacts to readiness, not just process survival;
- restore the dependency and verify the user-facing path, not only the pod state.
That drill has a nice side effect: it makes the rollback criteria concrete. In my previous note, I argued that a rollout needs a return address. This is part of the address book. If the only evidence after a rollback is “the pods are running,” there is still a lot left to guess.
Test notes: I ran this in two disposable Docker containers using Python 3.12. Each container used a read-only root filesystem, dropped Linux capabilities, no-new-privileges, a PID limit of 64, 0.25 CPU, and 128 MiB memory. The dependency failure was a local simulation. No host service, production endpoint, cluster, or external dependency was accessed.
Related reading: A rollout needs a return address · Delivery systems case study