<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>topicplatformengineering &amp;mdash; heyimusa</title>
    <link>https://heyimusa.blog/tag:topicplatformengineering</link>
    <description></description>
    <pubDate>Tue, 01 Sep 2026 06:56:41 +0000</pubDate>
    <item>
      <title>Refactoring is bonsai work, but the shears are tests</title>
      <link>https://heyimusa.blog/refactoring-is-bonsai-work</link>
      <description>&lt;![CDATA[A mature codebase rarely announces that it needs refactoring. It makes a small change feel oddly expensive. A new route requires touching three modules whose names no longer describe their responsibilities. A dependency update turns into archaeology. An incident fix needs a second person on the call because nobody is certain which branch of the conditional protects the customer path.&#xA;&#xA;That is the moment I find the bonsai metaphor useful, with one important correction: the job is not to make code look pretty. The job is to preserve a living system while shaping it so the next change has somewhere safe to go.&#xA;&#xA;!--more--&#xA;&#xA;The metaphor is useful because growth is the problem&#xA;&#xA;Bonsai is not a miniature tree that happens by accident. Its shape comes from repeated, deliberate interventions: pruning, wiring, repotting, and deciding which growth to keep. The tree stays alive, but it does not stay untouched.&#xA;&#xA;Software has the same tension. A healthy service keeps changing because product requirements, dependencies, traffic, and operators change. Left alone, the code can still compile and serve requests while its internal shape becomes costly to understand. The first symptom is not necessarily a defect. Often it is hesitation: the team starts treating a local edit as if it might wake something elsewhere.&#xA;&#xA;Martin Fowler&#39;s definition adds the part that is easy to lose: refactoring changes internal structure to make software easier to understand and cheaper to modify, without changing observable behavior. That excludes feature work, a migration that changes a contract, and a rewrite dressed up as cleanup. Those can all be worthwhile. They simply need different planning and different evidence.&#xA;&#xA;Calling everything &#34;refactoring&#34; is how a small maintenance task acquires an unlimited blast radius.&#xA;&#xA;A bonsai is not a weekly haircut&#xA;&#xA;The weak version of the metaphor produces cosmetic work: rename a few variables, reorder imports, split a long file, declare victory. Those changes can help, but they do not automatically make future changes safer.&#xA;&#xA;A better question is: what recurring change does this shape make expensive?&#xA;&#xA;If every new HTTP endpoint requires copying authorization checks, the useful work might be extracting an explicit policy boundary. If a feature flag has spread through handlers, workers, and templates, the issue is probably lifecycle ownership rather than the length of any one function. If an on-call engineer cannot identify the request path during an outage, improving observability or module boundaries may matter more than satisfying a linter.&#xA;&#xA;Refactoring earns its keep when a real next change has exposed a structural constraint.&#xA;&#xA;The shears are tests, not confidence&#xA;&#xA;There is a dangerous part of the gardening analogy. Pruning a branch is visible. A behavior regression may be invisible until production traffic finds it.&#xA;&#xA;That is why tests are not an optional polish step around refactoring. They are the evidence that the system still behaves as intended after an internal change. The refactoring literature repeatedly emphasizes small, safe mechanics. In a 2020 case study on an oversized class, Clare Sudbery describes keeping the code compiling and the tests running at every step, with coverage in place before the refactor starts.&#xA;&#xA;For a service, I would want the narrowest test that protects the behavior being moved, plus the checks that make the deployment credible:&#xA;&#xA;a unit or component test for the rule being extracted;&#xA;a contract or integration test where the code crosses a boundary; and&#xA;where proportionate, a post-deployment synthetic probe or smoke check for the important path.&#xA;&#xA;The exact stack differs. If the available evidence cannot support the claim that the intended behavior stayed stable, describe the change more modestly or add the missing check.&#xA;&#xA;This also gives a useful stop condition. When a proposed cleanup cannot be protected by a reasonably scoped test, the first task may be characterization: document and test what the system does today, including the awkward behavior nobody wants to rediscover during an incident. Only then is the pruning safe.&#xA;&#xA;AI makes overgrowth faster&#xA;&#xA;AI-assisted coding changes the economics here. It can produce a working branch, a duplicate helper, or another configuration path in seconds. That is useful when the team is deliberate. It also means structural debt can arrive at a rate that manual coding rarely achieved.&#xA;&#xA;A 2025 paper, The Shift from Writing to Pruning Software, proposes a bonsai-inspired interface for AI-generated code. Its core observation is sensible: when generation becomes cheap, the human task shifts toward navigating, evaluating, and reshaping a larger body of code. The paper proposes an interface; it is not evidence that the approach improves production reliability. Still, the direction matches a practical concern: generation can reduce the cost of adding code without reducing the cost of owning it.&#xA;&#xA;The response should not be &#34;never use AI.&#34; It should be to give generated code the same ownership questions as handwritten code:&#xA;&#xA;What responsibility does this new code own?&#xA;Which existing path does it duplicate or bypass?&#xA;What behavior proves it belongs here?&#xA;If it needs to change later, where would an engineer look first?&#xA;&#xA;If those questions have no good answer, the code is not ready just because it passes the current build.&#xA;&#xA;An infrastructure engineer prunes more than application code&#xA;&#xA;For an infrastructure or platform engineer, the tree extends beyond a service repository: delivery pipelines; infrastructure definitions and environment overlays; identity, network, and secret boundaries; runtime configuration; observability; and the runbooks and recovery paths that connect them.&#xA;&#xA;The same failure mode appears at every layer: a workaround becomes a pattern, a copied pipeline step diverges, an exception disappears into an overlay, or an alert keeps firing because no one owns the decision it is meant to support. Declarative infrastructure can accumulate structural debt just as application code can.&#xA;&#xA;That makes refactoring part of the platform engineer&#39;s job, not a side activity reserved for developers. The target is not maximal abstraction. It is a platform whose next user, deploy, incident, and security change can be made with a clear owner, a small review surface, and evidence.&#xA;&#xA;A practical sweep across the field looks like this:&#xA;&#xA;Platform and developer experience: turn tribal deployment knowledge into paved paths, understandable templates, and documented interfaces. Remove a platform abstraction when it only hides the system without removing work.&#xA;CI/CD and GitOps: consolidate duplicated release logic, make promotion and rollback paths explicit, and keep generated or rendered configuration reviewable. A pipeline refactor is incomplete if an operator cannot tell what will reach production.&#xA;Cloud and infrastructure as code: separate reusable modules from environment-specific intent, name ownership boundaries, and avoid copy-pasted resources that drift independently. Treat plans and policy checks as evidence, while remembering that they do not prove the live system is unchanged.&#xA;Kubernetes and runtime operations: simplify workload, ingress, Gateway, policy, and configuration overlays while preserving the routing and rollout contract. Rendered manifests and a bounded runtime check are stronger evidence than a tidy directory tree.&#xA;Security and identity: reduce broad permissions, clarify trust boundaries, remove obsolete access paths, and make secret rotation and break-glass procedures testable. A cleaner IAM policy is not safer unless the required workload still functions and the unwanted path is actually closed.&#xA;Networking and reliability: replace undocumented routing exceptions with explicit contracts; preserve DNS, TLS, timeout, retry, and failover behavior as you simplify. For a customer-facing path, verification must extend beyond a syntax check.&#xA;Observability and incident response: prune dashboards, alerts, and runbooks that no longer map to an operational decision. Keep the signals that let an engineer answer what changed, who is affected, and how to roll back.&#xA;Cost and capacity: remove idle or duplicated resources only after measuring dependency and demand. Cost reduction is a production change, not janitorial work.&#xA;&#xA;None of these areas needs a giant “platform cleanup” programme to improve. The valuable unit is still a small, explainable structural change backed by the evidence appropriate to that layer: an IaC plan, a rendered manifest diff, a policy evaluation, a synthetic request, a rollout observation, or a recovery exercise. The appropriate checks vary by layer, but each should support a specific preservation claim.&#xA;&#xA;Prune on the path of work&#xA;&#xA;A bonsai owner does not reshape the entire tree every time they notice one awkward branch. The same restraint matters in production code.&#xA;&#xA;I prefer refactoring that is adjacent to a real change: improve a boundary before extending it, remove duplication when the second copy appears, name a concept when the team needs to discuss it, and leave unrelated cleanups alone. This keeps the diff reviewable and lets the pull request tell a coherent story.&#xA;&#xA;It also makes verification more honest. A narrowly scoped refactor can state what it preserved, which tests exercised that claim, and what it deliberately did not touch. A broad &#34;cleanup&#34; commit usually cannot.&#xA;&#xA;There are times to fund a larger structural effort. Repeated incidents, a blocked security upgrade, an untestable critical path, or a migration that keeps failing for the same architectural reason are all stronger signals than aesthetic discomfort. Even then, split the work into behavior-preserving slices where possible. The system does not care that the end state looked tidy in a diagram if the path there breaks customers.&#xA;&#xA;A small operating loop&#xA;&#xA;Before calling a change refactoring, I would write down four things:&#xA;&#xA;The behavior to preserve. Describe it from the caller or user perspective, not from the internal class name.&#xA;The structural pressure. Name the repeated cost: duplicated policy, hidden dependency, impossible test setup, ambiguous ownership, or something equally concrete.&#xA;The evidence. Identify the test or probe that should remain green before and after the change.&#xA;The boundary. State what is not being redesigned in this pull request.&#xA;&#xA;Then make one structural move, run the evidence, and review the diff as if you were the next person debugging it at an inconvenient hour.&#xA;&#xA;That is less romantic than trimming a miniature tree. It is also more useful. Refactoring is routine maintenance of the structure through which future changes must travel.&#xA;&#xA;Limits of the metaphor&#xA;&#xA;Bonsai suggests a single careful owner and a stable aesthetic goal. Real systems have many owners, competing requirements, and behavior that must change. Refactoring will not replace migration planning, incident response, performance work, or product decisions. A green test suite also cannot prove every production property, especially across external dependencies and real traffic.&#xA;&#xA;So use the image lightly. Keep the discipline.&#xA;&#xA;Preserve the behavior you can state. Improve the structure that is making real work harder. Keep the change small enough to understand. And do not put the shears down until you have evidence that the tree still lives.&#xA;&#xA;Sources and further reading&#xA;&#xA;Martin Fowler, Refactoring. Fowler defines refactoring as changing internal structure to make software easier to understand and cheaper to modify without changing observable behavior.&#xA;Clare Sudbery, Refactoring: This class is too large (2020). A practical case study of incremental refactoring, covering tests, compiling at every step, and small commits.&#xA;Ulam Labs, Why is refactoring your code important? (2020). A useful example of the bonsai analogy, though this article intentionally avoids treating cleanliness or security as automatic outcomes of refactoring.&#xA;The Shift from Writing to Pruning Software: A Bonsai-Inspired IDE for Reshaping AI Generated Code, arXiv:2503.02833 (2025). A research proposal about navigating and reshaping generated code, not a production-effectiveness study.&#xA;&#xA;!-- taxonomy: #topicplatformengineering #topicgitopsdelivery --&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>A mature codebase rarely announces that it needs refactoring. It makes a small change feel oddly expensive. A new route requires touching three modules whose names no longer describe their responsibilities. A dependency update turns into archaeology. An incident fix needs a second person on the call because nobody is certain which branch of the conditional protects the customer path.</p>

<p>That is the moment I find the bonsai metaphor useful, with one important correction: the job is not to make code look pretty. The job is to preserve a living system while shaping it so the next change has somewhere safe to go.</p>



<h2 id="the-metaphor-is-useful-because-growth-is-the-problem">The metaphor is useful because growth is the problem</h2>

<p>Bonsai is not a miniature tree that happens by accident. Its shape comes from repeated, deliberate interventions: pruning, wiring, repotting, and deciding which growth to keep. The tree stays alive, but it does not stay untouched.</p>

<p>Software has the same tension. A healthy service keeps changing because product requirements, dependencies, traffic, and operators change. Left alone, the code can still compile and serve requests while its internal shape becomes costly to understand. The first symptom is not necessarily a defect. Often it is hesitation: the team starts treating a local edit as if it might wake something elsewhere.</p>

<p>Martin Fowler&#39;s definition adds the part that is easy to lose: refactoring changes internal structure to make software easier to understand and cheaper to modify, without changing observable behavior. That excludes feature work, a migration that changes a contract, and a rewrite dressed up as cleanup. Those can all be worthwhile. They simply need different planning and different evidence.</p>

<p>Calling everything “refactoring” is how a small maintenance task acquires an unlimited blast radius.</p>

<h2 id="a-bonsai-is-not-a-weekly-haircut">A bonsai is not a weekly haircut</h2>

<p>The weak version of the metaphor produces cosmetic work: rename a few variables, reorder imports, split a long file, declare victory. Those changes can help, but they do not automatically make future changes safer.</p>

<p>A better question is: <em>what recurring change does this shape make expensive?</em></p>

<p>If every new HTTP endpoint requires copying authorization checks, the useful work might be extracting an explicit policy boundary. If a feature flag has spread through handlers, workers, and templates, the issue is probably lifecycle ownership rather than the length of any one function. If an on-call engineer cannot identify the request path during an outage, improving observability or module boundaries may matter more than satisfying a linter.</p>

<p>Refactoring earns its keep when a real next change has exposed a structural constraint.</p>

<h2 id="the-shears-are-tests-not-confidence">The shears are tests, not confidence</h2>

<p>There is a dangerous part of the gardening analogy. Pruning a branch is visible. A behavior regression may be invisible until production traffic finds it.</p>

<p>That is why tests are not an optional polish step around refactoring. They are the evidence that the system still behaves as intended after an internal change. The refactoring literature repeatedly emphasizes small, safe mechanics. In a 2020 case study on an oversized class, Clare Sudbery describes keeping the code compiling and the tests running at every step, with coverage in place before the refactor starts.</p>

<p>For a service, I would want the narrowest test that protects the behavior being moved, plus the checks that make the deployment credible:</p>
<ul><li>a unit or component test for the rule being extracted;</li>
<li>a contract or integration test where the code crosses a boundary; and</li>
<li>where proportionate, a post-deployment synthetic probe or smoke check for the important path.</li></ul>

<p>The exact stack differs. If the available evidence cannot support the claim that the intended behavior stayed stable, describe the change more modestly or add the missing check.</p>

<p>This also gives a useful stop condition. When a proposed cleanup cannot be protected by a reasonably scoped test, the first task may be characterization: document and test what the system does today, including the awkward behavior nobody wants to rediscover during an incident. Only then is the pruning safe.</p>

<h2 id="ai-makes-overgrowth-faster">AI makes overgrowth faster</h2>

<p>AI-assisted coding changes the economics here. It can produce a working branch, a duplicate helper, or another configuration path in seconds. That is useful when the team is deliberate. It also means structural debt can arrive at a rate that manual coding rarely achieved.</p>

<p>A 2025 paper, <em>The Shift from Writing to Pruning Software</em>, proposes a bonsai-inspired interface for AI-generated code. Its core observation is sensible: when generation becomes cheap, the human task shifts toward navigating, evaluating, and reshaping a larger body of code. The paper proposes an interface; it is not evidence that the approach improves production reliability. Still, the direction matches a practical concern: generation can reduce the cost of adding code without reducing the cost of owning it.</p>

<p>The response should not be “never use AI.” It should be to give generated code the same ownership questions as handwritten code:</p>
<ul><li>What responsibility does this new code own?</li>
<li>Which existing path does it duplicate or bypass?</li>
<li>What behavior proves it belongs here?</li>
<li>If it needs to change later, where would an engineer look first?</li></ul>

<p>If those questions have no good answer, the code is not ready just because it passes the current build.</p>

<h2 id="an-infrastructure-engineer-prunes-more-than-application-code">An infrastructure engineer prunes more than application code</h2>

<p>For an infrastructure or platform engineer, the tree extends beyond a service repository: delivery pipelines; infrastructure definitions and environment overlays; identity, network, and secret boundaries; runtime configuration; observability; and the runbooks and recovery paths that connect them.</p>

<p>The same failure mode appears at every layer: a workaround becomes a pattern, a copied pipeline step diverges, an exception disappears into an overlay, or an alert keeps firing because no one owns the decision it is meant to support. Declarative infrastructure can accumulate structural debt just as application code can.</p>

<p>That makes refactoring part of the platform engineer&#39;s job, not a side activity reserved for developers. The target is not maximal abstraction. It is a platform whose next user, deploy, incident, and security change can be made with a clear owner, a small review surface, and evidence.</p>

<p>A practical sweep across the field looks like this:</p>
<ul><li><strong>Platform and developer experience:</strong> turn tribal deployment knowledge into paved paths, understandable templates, and documented interfaces. Remove a platform abstraction when it only hides the system without removing work.</li>
<li><strong>CI/CD and GitOps:</strong> consolidate duplicated release logic, make promotion and rollback paths explicit, and keep generated or rendered configuration reviewable. A pipeline refactor is incomplete if an operator cannot tell what will reach production.</li>
<li><strong>Cloud and infrastructure as code:</strong> separate reusable modules from environment-specific intent, name ownership boundaries, and avoid copy-pasted resources that drift independently. Treat plans and policy checks as evidence, while remembering that they do not prove the live system is unchanged.</li>
<li><strong>Kubernetes and runtime operations:</strong> simplify workload, ingress, Gateway, policy, and configuration overlays while preserving the routing and rollout contract. Rendered manifests and a bounded runtime check are stronger evidence than a tidy directory tree.</li>
<li><strong>Security and identity:</strong> reduce broad permissions, clarify trust boundaries, remove obsolete access paths, and make secret rotation and break-glass procedures testable. A cleaner IAM policy is not safer unless the required workload still functions and the unwanted path is actually closed.</li>
<li><strong>Networking and reliability:</strong> replace undocumented routing exceptions with explicit contracts; preserve DNS, TLS, timeout, retry, and failover behavior as you simplify. For a customer-facing path, verification must extend beyond a syntax check.</li>
<li><strong>Observability and incident response:</strong> prune dashboards, alerts, and runbooks that no longer map to an operational decision. Keep the signals that let an engineer answer what changed, who is affected, and how to roll back.</li>
<li><strong>Cost and capacity:</strong> remove idle or duplicated resources only after measuring dependency and demand. Cost reduction is a production change, not janitorial work.</li></ul>

<p>None of these areas needs a giant “platform cleanup” programme to improve. The valuable unit is still a small, explainable structural change backed by the evidence appropriate to that layer: an IaC plan, a rendered manifest diff, a policy evaluation, a synthetic request, a rollout observation, or a recovery exercise. The appropriate checks vary by layer, but each should support a specific preservation claim.</p>

<h2 id="prune-on-the-path-of-work">Prune on the path of work</h2>

<p>A bonsai owner does not reshape the entire tree every time they notice one awkward branch. The same restraint matters in production code.</p>

<p>I prefer refactoring that is adjacent to a real change: improve a boundary before extending it, remove duplication when the second copy appears, name a concept when the team needs to discuss it, and leave unrelated cleanups alone. This keeps the diff reviewable and lets the pull request tell a coherent story.</p>

<p>It also makes verification more honest. A narrowly scoped refactor can state what it preserved, which tests exercised that claim, and what it deliberately did not touch. A broad “cleanup” commit usually cannot.</p>

<p>There are times to fund a larger structural effort. Repeated incidents, a blocked security upgrade, an untestable critical path, or a migration that keeps failing for the same architectural reason are all stronger signals than aesthetic discomfort. Even then, split the work into behavior-preserving slices where possible. The system does not care that the end state looked tidy in a diagram if the path there breaks customers.</p>

<h2 id="a-small-operating-loop">A small operating loop</h2>

<p>Before calling a change refactoring, I would write down four things:</p>
<ol><li><strong>The behavior to preserve.</strong> Describe it from the caller or user perspective, not from the internal class name.</li>
<li><strong>The structural pressure.</strong> Name the repeated cost: duplicated policy, hidden dependency, impossible test setup, ambiguous ownership, or something equally concrete.</li>
<li><strong>The evidence.</strong> Identify the test or probe that should remain green before and after the change.</li>
<li><strong>The boundary.</strong> State what is not being redesigned in this pull request.</li></ol>

<p>Then make one structural move, run the evidence, and review the diff as if you were the next person debugging it at an inconvenient hour.</p>

<p>That is less romantic than trimming a miniature tree. It is also more useful. Refactoring is routine maintenance of the structure through which future changes must travel.</p>

<h2 id="limits-of-the-metaphor">Limits of the metaphor</h2>

<p>Bonsai suggests a single careful owner and a stable aesthetic goal. Real systems have many owners, competing requirements, and behavior that must change. Refactoring will not replace migration planning, incident response, performance work, or product decisions. A green test suite also cannot prove every production property, especially across external dependencies and real traffic.</p>

<p>So use the image lightly. Keep the discipline.</p>

<p>Preserve the behavior you can state. Improve the structure that is making real work harder. Keep the change small enough to understand. And do not put the shears down until you have evidence that the tree still lives.</p>

<h2 id="sources-and-further-reading">Sources and further reading</h2>
<ul><li>Martin Fowler, <a href="https://refactoring.com/">Refactoring</a>. Fowler defines refactoring as changing internal structure to make software easier to understand and cheaper to modify without changing observable behavior.</li>
<li>Clare Sudbery, <a href="https://martinfowler.com/articles/class-too-large.html">Refactoring: This class is too large</a> (2020). A practical case study of incremental refactoring, covering tests, compiling at every step, and small commits.</li>
<li>Ulam Labs, <a href="https://ulam.io/blog/why-refactoring-code-important/">Why is refactoring your code important?</a> (2020). A useful example of the bonsai analogy, though this article intentionally avoids treating cleanliness or security as automatic outcomes of refactoring.</li>
<li><em>The Shift from Writing to Pruning Software: A Bonsai-Inspired IDE for Reshaping AI Generated Code</em>, <a href="https://arxiv.org/abs/2503.02833">arXiv:2503.02833</a> (2025). A research proposal about navigating and reshaping generated code, not a production-effectiveness study.</li></ul>


]]></content:encoded>
      <guid>https://heyimusa.blog/refactoring-is-bonsai-work</guid>
      <pubDate>Tue, 11 Aug 2026 16:02:41 +0000</pubDate>
    </item>
    <item>
      <title>A Gateway migration is a routing change, not a YAML conversion</title>
      <link>https://heyimusa.blog/a-gateway-migration-is-a-routing-change</link>
      <description>&lt;![CDATA[A Docker-only smoke test of ingress2gateway against a synthetic Ingress with TLS, HTTP-to-HTTPS redirect, a regex route, and a capture-group rewrite.&#xA;&#xA;ingress-nginx reached retirement in March 2026. The obvious response is to convert manifests to Gateway API and move on.&#xA;&#xA;I do not think that is enough.&#xA;&#xA;An ingress migration changes the way a public request reaches an application. The YAML is different, but the thing worth preserving is a routing contract: which hostnames exist, which paths match, which backend receives them, whether HTTP redirects, where TLS terminates, and which exceptions were carried by annotations.&#xA;&#xA;!--more--&#xA;&#xA;The Kubernetes SIG Network project ingress2gateway is the right kind of starting point. It converts Ingress and supported provider resources into Gateway API resources, and it reports fields it cannot translate. Its current v1.2.0 release was published on July 7, 2026.&#xA;&#xA;I wanted to see what that warning surface looks like in a small, bounded migration—not to test a live controller.&#xA;&#xA;The fixture was deliberately awkward&#xA;&#xA;I made one synthetic Ingress for shop.example.test:&#xA;&#xA;TLS with a named Secret;&#xA;a / prefix route to a storefront service;&#xA;an /api(/|$)(.) regular-expression route to an API service;&#xA;an HTTP-to-HTTPS redirect; and&#xA;the common ingress-nginx rewrite annotation nginx.ingress.kubernetes.io/rewrite-target: /$2.&#xA;&#xA;That last field is the important one. It changes the request URI forwarded to the application. Losing it can turn a migration that looks correct in a manifest review into an application-level break.&#xA;&#xA;I built ingress2gateway from the signed v1.2.0 source tag in a disposable Docker build stage. The final run used a non-root UID, no network, a read-only filesystem, a small temporary filesystem, dropped Linux capabilities, no-new-privileges, one CPU, 256 MiB memory, and a PID limit of 64. The fixture was baked into the image, so the run did not mount a host directory.&#xA;&#xA;Gateway/nginx&#xA;HTTPRoute/store-shop-example-test&#xA;HTTPRoute/store-shop-example-test-http&#xA;&#xA;The generated Gateway preserved the hostname and produced separate HTTP and HTTPS listeners:&#xA;&#xA;listeners:&#xA;hostname: shop.example.test&#xA;  name: shop-example-test-http&#xA;  port: 80&#xA;  protocol: HTTP&#xA;hostname: shop.example.test&#xA;  name: shop-example-test-https&#xA;  port: 443&#xA;  protocol: HTTPS&#xA;  tls:&#xA;    certificateRefs:&#xA;    kind: Secret&#xA;      name: shop-tls&#xA;&#xA;It also generated a port-80 HTTPRoute that returns a 308 redirect to HTTPS. That is a concrete routing behavior worth checking, not just a generated object worth counting.&#xA;&#xA;The converter gave me the review list&#xA;&#xA;The conversion did not silently declare success. It emitted these warnings:&#xA;&#xA;INFO  Using case-insensitive regex path matches. You may want to change this.&#xA;&#xA;WARN  Path rewrites with capture group references are not supported&#xA;      source: STANDARD_EMITTER&#xA;      object: HTTPRoute: demo/store-shop-example-test&#xA;&#xA;WARN  Gateway API does not support configuring URL normalization&#xA;      Please check if this matters for your use case.&#xA;&#xA;That is the most useful result of the experiment.&#xA;&#xA;The generated HTTPS route still matched the API pattern:&#xA;&#xA;matches:&#xA;path:&#xA;    type: RegularExpression&#xA;    value: (?i)/api(/|$)(.).&#xA;backendRefs:&#xA;name: api&#xA;  port: 8080&#xA;&#xA;But the $2 rewrite was absent because the standard conversion cannot represent capture-group references. The route exists; the request URI semantics may not. A simple resource-by-resource diff would make that easy to miss.&#xA;&#xA;Conversion is not equivalence&#xA;&#xA;ingress2gateway documents an important boundary: it translates supported Ingress fields and provider-specific resources into Gateway API resources; it is not meant to copy annotations blindly. That is sensible. Many annotations are controller-specific behavior, not portable API.&#xA;&#xA;The consequence is that every untranslatable annotation is a migration-review item. It should have an owner and a decision:&#xA;&#xA;warning -  affected route -  expected behavior -  replacement or explicit acceptance&#xA;&#xA;For this fixture, the follow-up would be to decide how the API should receive /api/... requests under the target Gateway implementation. That might require a supported rewrite mechanism, a backend change, or a decision to preserve controller-specific behavior outside the standard conversion. The converter cannot safely make that product decision.&#xA;&#xA;The check I would put beside a migration PR&#xA;&#xA;I would review an Ingress-to-Gateway change as a table of observable behavior, not as an object count:&#xA;&#xA;| Contract | Before | After | Evidence |&#xA;|---|---|---|---|&#xA;| Hostname | shop.example.test | listener and HTTPRoute hostname | rendered manifests |&#xA;| TLS | Secret-backed HTTPS | certificateRefs on port 443 | rendered manifests + target controller status |&#xA;| HTTP redirect | enabled | explicit 308 route filter | rendered manifests + request test |&#xA;| API path match | regex | case-insensitive regex | converter warning + request test |&#xA;| URI rewrite | /$2 | absent | converter warning; remediation required |&#xA;| Backend | api:8080 | api:8080 | rendered manifests |&#xA;&#xA;Only the final column can establish that a controller accepts and serves the intended configuration. My Docker run did not install Gateway API CRDs, start a controller, contact a cluster, resolve DNS, or send an external request. It proved that the converter can surface an important portability gap in a repeatable synthetic input. It did not prove a production migration safe.&#xA;&#xA;That distinction matters. A migration has passed the conversion stage when its generated resources parse and its warnings are understood. It has passed the routing stage only when the target controller reports accepted resources and the expected requests behave correctly.&#xA;&#xA;What I would build next&#xA;&#xA;The gap is not another converter. ingress2gateway already has a focused job, supports several providers and emitters, and reports unsupported behavior.&#xA;&#xA;The useful companion would be a small, offline routing-contract diff*: read the source Ingress plus the generated Gateway and Routes, then list hosts, path matching, backends, redirects, TLS references, and converter warnings side by side. It should flag a missing rewrite or a changed match mode as review required, never claim behavioral equivalence.&#xA;&#xA;That would make the handoff from conversion output to migration review explicit—the same way a good GitOps diff turns a subtle configuration change into a question someone can answer deliberately.&#xA;&#xA;---&#xA;&#xA;Test notes: This was a smoke test in a disposable Docker image built from the ingress2gateway v1.2.0 source tag. The final container ran as UID 65532 with --network none, a read-only filesystem, /tmp as a limited tmpfs, all capabilities dropped, no-new-privileges, --pids-limit 64, --memory 256m, and --cpus 1. It used one sanitized synthetic Ingress. No Kubernetes cluster, kubeconfig, DNS provider, Secret value, or live hostname was accessed. The output YAML SHA-256 was 11aa5ed72087027043e299a6b78e9f1efd944f26b3883127cdcc375e469dbaa5.&#xA;&#xA;Sources: Kubernetes on the retirement of ingress-nginx · Ingress2Gateway 1.0 announcement · ingress2gateway documentation&#xA;&#xA;Related work: I built a security-impact diff for GitOps · A rollout needs a return address&#xA;&#xA;!-- taxonomy: #topicplatformengineering --&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><em>A Docker-only smoke test of <code>ingress2gateway</code> against a synthetic <code>Ingress</code> with TLS, HTTP-to-HTTPS redirect, a regex route, and a capture-group rewrite.</em></p>

<p><code>ingress-nginx</code> reached retirement in March 2026. The obvious response is to convert manifests to Gateway API and move on.</p>

<p>I do not think that is enough.</p>

<p>An ingress migration changes the way a public request reaches an application. The YAML is different, but the thing worth preserving is a routing contract: which hostnames exist, which paths match, which backend receives them, whether HTTP redirects, where TLS terminates, and which exceptions were carried by annotations.</p>



<p>The Kubernetes SIG Network project <a href="https://github.com/kubernetes-sigs/ingress2gateway"><code>ingress2gateway</code></a> is the right kind of starting point. It converts Ingress and supported provider resources into Gateway API resources, and it reports fields it cannot translate. Its current <a href="https://github.com/kubernetes-sigs/ingress2gateway/releases/tag/v1.2.0"><code>v1.2.0</code> release</a> was published on July 7, 2026.</p>

<p>I wanted to see what that warning surface looks like in a small, bounded migration—not to test a live controller.</p>

<h2 id="the-fixture-was-deliberately-awkward">The fixture was deliberately awkward</h2>

<p>I made one synthetic <code>Ingress</code> for <code>shop.example.test</code>:</p>
<ul><li>TLS with a named Secret;</li>
<li>a <code>/</code> prefix route to a storefront service;</li>
<li>an <code>/api(/|$)(.*)</code> regular-expression route to an API service;</li>
<li>an HTTP-to-HTTPS redirect; and</li>
<li>the common ingress-nginx rewrite annotation <code>nginx.ingress.kubernetes.io/rewrite-target: /$2</code>.</li></ul>

<p>That last field is the important one. It changes the request URI forwarded to the application. Losing it can turn a migration that looks correct in a manifest review into an application-level break.</p>

<p>I built <code>ingress2gateway</code> from the signed <code>v1.2.0</code> source tag in a disposable Docker build stage. The final run used a non-root UID, no network, a read-only filesystem, a small temporary filesystem, dropped Linux capabilities, <code>no-new-privileges</code>, one CPU, 256 MiB memory, and a PID limit of 64. The fixture was baked into the image, so the run did not mount a host directory.</p>

<pre><code class="language-text">Gateway/nginx
HTTPRoute/store-shop-example-test
HTTPRoute/store-shop-example-test-http
</code></pre>

<p>The generated Gateway preserved the hostname and produced separate HTTP and HTTPS listeners:</p>

<pre><code class="language-yaml">listeners:
- hostname: shop.example.test
  name: shop-example-test-http
  port: 80
  protocol: HTTP
- hostname: shop.example.test
  name: shop-example-test-https
  port: 443
  protocol: HTTPS
  tls:
    certificateRefs:
    - kind: Secret
      name: shop-tls
</code></pre>

<p>It also generated a port-80 <code>HTTPRoute</code> that returns a <code>308</code> redirect to HTTPS. That is a concrete routing behavior worth checking, not just a generated object worth counting.</p>

<h2 id="the-converter-gave-me-the-review-list">The converter gave me the review list</h2>

<p>The conversion did not silently declare success. It emitted these warnings:</p>

<pre><code class="language-text">INFO  Using case-insensitive regex path matches. You may want to change this.

WARN  Path rewrites with capture group references are not supported
      source: STANDARD_EMITTER
      object: HTTPRoute: demo/store-shop-example-test

WARN  Gateway API does not support configuring URL normalization
      Please check if this matters for your use case.
</code></pre>

<p>That is the most useful result of the experiment.</p>

<p>The generated HTTPS route still matched the API pattern:</p>

<pre><code class="language-yaml">matches:
- path:
    type: RegularExpression
    value: (?i)/api(/|$)(.*).*
backendRefs:
- name: api
  port: 8080
</code></pre>

<p>But the <code>$2</code> rewrite was absent because the standard conversion cannot represent capture-group references. The route exists; the request URI semantics may not. A simple resource-by-resource diff would make that easy to miss.</p>

<h2 id="conversion-is-not-equivalence">Conversion is not equivalence</h2>

<p><code>ingress2gateway</code> documents an important boundary: it translates supported Ingress fields and provider-specific resources into Gateway API resources; it is not meant to copy annotations blindly. That is sensible. Many annotations are controller-specific behavior, not portable API.</p>

<p>The consequence is that every untranslatable annotation is a migration-review item. It should have an owner and a decision:</p>

<pre><code class="language-text">warning -&gt; affected route -&gt; expected behavior -&gt; replacement or explicit acceptance
</code></pre>

<p>For this fixture, the follow-up would be to decide how the API should receive <code>/api/...</code> requests under the target Gateway implementation. That might require a supported rewrite mechanism, a backend change, or a decision to preserve controller-specific behavior outside the standard conversion. The converter cannot safely make that product decision.</p>

<h2 id="the-check-i-would-put-beside-a-migration-pr">The check I would put beside a migration PR</h2>

<p>I would review an Ingress-to-Gateway change as a table of observable behavior, not as an object count:</p>

<table>
<thead>
<tr>
<th>Contract</th>
<th>Before</th>
<th>After</th>
<th>Evidence</th>
</tr>
</thead>

<tbody>
<tr>
<td>Hostname</td>
<td><code>shop.example.test</code></td>
<td>listener and <code>HTTPRoute</code> hostname</td>
<td>rendered manifests</td>
</tr>

<tr>
<td>TLS</td>
<td>Secret-backed HTTPS</td>
<td><code>certificateRefs</code> on port 443</td>
<td>rendered manifests + target controller status</td>
</tr>

<tr>
<td>HTTP redirect</td>
<td>enabled</td>
<td>explicit <code>308</code> route filter</td>
<td>rendered manifests + request test</td>
</tr>

<tr>
<td>API path match</td>
<td>regex</td>
<td>case-insensitive regex</td>
<td>converter warning + request test</td>
</tr>

<tr>
<td>URI rewrite</td>
<td><code>/$2</code></td>
<td>absent</td>
<td>converter warning; remediation required</td>
</tr>

<tr>
<td>Backend</td>
<td><code>api:8080</code></td>
<td><code>api:8080</code></td>
<td>rendered manifests</td>
</tr>
</tbody>
</table>

<p>Only the final column can establish that a controller accepts and serves the intended configuration. My Docker run did not install Gateway API CRDs, start a controller, contact a cluster, resolve DNS, or send an external request. It proved that the converter can surface an important portability gap in a repeatable synthetic input. It did not prove a production migration safe.</p>

<p>That distinction matters. A migration has passed the conversion stage when its generated resources parse and its warnings are understood. It has passed the routing stage only when the target controller reports accepted resources and the expected requests behave correctly.</p>

<h2 id="what-i-would-build-next">What I would build next</h2>

<p>The gap is not another converter. <code>ingress2gateway</code> already has a focused job, supports several providers and emitters, and reports unsupported behavior.</p>

<p>The useful companion would be a small, offline <em>routing-contract diff</em>: read the source Ingress plus the generated Gateway and Routes, then list hosts, path matching, backends, redirects, TLS references, and converter warnings side by side. It should flag a missing rewrite or a changed match mode as <strong>review required</strong>, never claim behavioral equivalence.</p>

<p>That would make the handoff from conversion output to migration review explicit—the same way a good GitOps diff turns a subtle configuration change into a question someone can answer deliberately.</p>

<hr>

<p><strong>Test notes:</strong> This was a smoke test in a disposable Docker image built from the <code>ingress2gateway</code> <code>v1.2.0</code> source tag. The final container ran as UID <code>65532</code> with <code>--network none</code>, a read-only filesystem, <code>/tmp</code> as a limited tmpfs, all capabilities dropped, <code>no-new-privileges</code>, <code>--pids-limit 64</code>, <code>--memory 256m</code>, and <code>--cpus 1</code>. It used one sanitized synthetic Ingress. No Kubernetes cluster, kubeconfig, DNS provider, Secret value, or live hostname was accessed. The output YAML SHA-256 was <code>11aa5ed72087027043e299a6b78e9f1efd944f26b3883127cdcc375e469dbaa5</code>.</p>

<p><strong>Sources:</strong> <a href="https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/">Kubernetes on the retirement of ingress-nginx</a> · <a href="https://kubernetes.io/blog/2026/03/20/ingress2gateway-1-0-release/">Ingress2Gateway 1.0 announcement</a> · <a href="https://github.com/kubernetes-sigs/ingress2gateway">ingress2gateway documentation</a></p>

<p><strong>Related work:</strong> <a href="/i-built-a-security-impact-diff-for-gitops">I built a security-impact diff for GitOps</a> · <a href="/a-rollout-needs-a-return-address">A rollout needs a return address</a></p>


]]></content:encoded>
      <guid>https://heyimusa.blog/a-gateway-migration-is-a-routing-change</guid>
      <pubDate>Sat, 25 Jul 2026 03:25:00 +0000</pubDate>
    </item>
    <item>
      <title>I built a small linter for health checks that disagree with rollouts</title>
      <link>https://heyimusa.blog/i-built-a-small-linter-for-health-checks</link>
      <description>&lt;![CDATA[A declared readiness rollout gate disagrees with a Docker Compose healthcheck that calls only liveness&#xA;&#xA;Original diagram for this Docker-only build note. It shows a configuration disagreement, not a production incident.&#xA;&#xA;I kept tripping over the same sentence while writing about health checks: “we gate the rollout on readiness.”&#xA;&#xA;It sounds reassuring. Then you open the deployment file and find a healthcheck that calls /healthz, because that endpoint was easy to add and it returns 200 as long as the process is alive.&#xA;&#xA;Neither file is necessarily wrong on its own. Together, they can tell two different stories.&#xA;&#xA;!--more--&#xA;&#xA;So I made a small CLI called probe-contract. It compares a declared health contract with a Docker Compose file and reports the disagreements that are easy to miss in review.&#xA;&#xA;The first release is deliberately small. It does not call a live endpoint. It does not attempt to infer whether a checkout flow, queue, or database is truly healthy. It reads configuration and asks a narrower question: does the probe in the deployment file support the health assumption you wrote down?&#xA;&#xA;The smallest useful contract&#xA;&#xA;The tool takes a tiny YAML file alongside Compose:&#xA;&#xA;services:&#xA;  api:&#xA;    liveness: /healthz&#xA;    readiness: /readyz&#xA;    userpath: /checkout&#xA;    rolloutgate: readiness&#xA;&#xA;Then it compares that declaration with the Compose healthcheck.&#xA;&#xA;For the mismatch case, the contract said readiness should gate the rollout, while the Compose probe called only /healthz:&#xA;&#xA;WARNING  api: rolloutgate is readiness, but Compose healthcheck does not reference /readyz (READINESSNOTPROBED)&#xA;&#xA;That warning is the whole point of the first version. It does not prove /readyz is a good readiness endpoint. It makes the disagreement visible before somebody treats a green container as proof that a rollout is safe.&#xA;&#xA;I tested the tool in a container too&#xA;&#xA;I ran the released CLI against its fixtures inside a Docker container with a read-only filesystem, a non-root user, dropped capabilities, and no-new-privileges.&#xA;&#xA;The valid fixture produced empty JSON:&#xA;&#xA;{&#xA;  &#34;diagnostics&#34;: []&#xA;}&#xA;&#xA;A Compose file with its healthcheck explicitly disabled produced an error and exit code 1:&#xA;&#xA;ERROR  api: service has no Compose healthcheck (HEALTHCHECKMISSING)&#xA;&#xA;That second case mattered. An early review of the tool found that healthcheck: { disable: true } could look like an active healthcheck to a naive YAML parser. The release now treats both disable: true and Docker’s test: [NONE] form as missing checks.&#xA;&#xA;The review also caught a less obvious false pass: /ready should not match /readyz just because one string contains the other. That is fixed too. A health contract is already an approximation; the checker should not add accidental ambiguity on top.&#xA;&#xA;What it checks today&#xA;&#xA;probe-contract v0.1.0 checks a few things and stops there:&#xA;&#xA;a service declared in the contract is present in Compose;&#xA;that service has an active healthcheck;&#xA;a readiness rollout gate actually probes the declared readiness path;&#xA;liveness and readiness are not declared as the same endpoint by accident;&#xA;interval, timeout, and retries are present on the healthcheck.&#xA;&#xA;It emits human-readable text by default and JSON with --format json, which is enough to start using it in CI without making every warning a release blocker.&#xA;&#xA;What it intentionally does not do&#xA;&#xA;I do not want this to become another linter that promises too much.&#xA;&#xA;It does not inspect Kubernetes manifests yet. It does not send traffic to a live service. It does not infer a business transaction from a URL. And it cannot tell whether a dependency should be part of readiness for a particular application.&#xA;&#xA;Those choices belong to the team running the service. The tool only asks them to make the choice explicit, then checks whether the Compose file agrees.&#xA;&#xA;That scope is small enough to be useful. It is also small enough that a reviewer can understand what a warning means without trusting a black box.&#xA;&#xA;Release and next steps&#xA;&#xA;The project is public under MIT and includes a Linux amd64 binary with a checksum:&#xA;&#xA;Source and README&#xA;v0.1.0 release&#xA;&#xA;The next likely steps are Kubernetes manifest support, SARIF output, and a GitHub Action. I am deliberately not calling those features until the Compose contract is useful enough to earn them.&#xA;&#xA;---&#xA;&#xA;Test notes: I built and ran probe-contract v0.1.0 only in Docker for this note. The demonstration container used a read-only filesystem, non-root UID 65532, dropped Linux capabilities, no-new-privileges, and read-only mounted fixture files. The results are configuration checks against local fixtures, not a benchmark or a production deployment.&#xA;&#xA;Related work: Delivery systems case study · I made a healthy service page on purpose&#xA;&#xA;!-- taxonomy: #topicplatformengineering #topicgitopsdelivery --&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="/img/articles/probe-contract-makes-health-assumptions-visible.svg" alt="A declared readiness rollout gate disagrees with a Docker Compose healthcheck that calls only liveness"></p>

<p><em>Original diagram for this Docker-only build note. It shows a configuration disagreement, not a production incident.</em></p>

<p>I kept tripping over the same sentence while writing about health checks: “we gate the rollout on readiness.”</p>

<p>It sounds reassuring. Then you open the deployment file and find a healthcheck that calls <code>/healthz</code>, because that endpoint was easy to add and it returns <code>200</code> as long as the process is alive.</p>

<p>Neither file is necessarily wrong on its own. Together, they can tell two different stories.</p>



<p>So I made a small CLI called <a href="https://github.com/heyimusa/probe-contract"><code>probe-contract</code></a>. It compares a declared health contract with a Docker Compose file and reports the disagreements that are easy to miss in review.</p>

<p>The first release is deliberately small. It does not call a live endpoint. It does not attempt to infer whether a checkout flow, queue, or database is truly healthy. It reads configuration and asks a narrower question: does the probe in the deployment file support the health assumption you wrote down?</p>

<h2 id="the-smallest-useful-contract">The smallest useful contract</h2>

<p>The tool takes a tiny YAML file alongside Compose:</p>

<pre><code class="language-yaml">services:
  api:
    liveness: /healthz
    readiness: /readyz
    user_path: /checkout
    rollout_gate: readiness
</code></pre>

<p>Then it compares that declaration with the Compose healthcheck.</p>

<p>For the mismatch case, the contract said readiness should gate the rollout, while the Compose probe called only <code>/healthz</code>:</p>

<pre><code class="language-text">WARNING  api: rollout_gate is readiness, but Compose healthcheck does not reference /readyz (READINESS_NOT_PROBED)
</code></pre>

<p>That warning is the whole point of the first version. It does not prove <code>/readyz</code> is a good readiness endpoint. It makes the disagreement visible before somebody treats a green container as proof that a rollout is safe.</p>

<h2 id="i-tested-the-tool-in-a-container-too">I tested the tool in a container too</h2>

<p>I ran the released CLI against its fixtures inside a Docker container with a read-only filesystem, a non-root user, dropped capabilities, and <code>no-new-privileges</code>.</p>

<p>The valid fixture produced empty JSON:</p>

<pre><code class="language-json">{
  &#34;diagnostics&#34;: []
}
</code></pre>

<p>A Compose file with its healthcheck explicitly disabled produced an error and exit code <code>1</code>:</p>

<pre><code class="language-text">ERROR  api: service has no Compose healthcheck (HEALTHCHECK_MISSING)
</code></pre>

<p>That second case mattered. An early review of the tool found that <code>healthcheck: { disable: true }</code> could look like an active healthcheck to a naive YAML parser. The release now treats both <code>disable: true</code> and Docker’s <code>test: [NONE]</code> form as missing checks.</p>

<p>The review also caught a less obvious false pass: <code>/ready</code> should not match <code>/readyz</code> just because one string contains the other. That is fixed too. A health contract is already an approximation; the checker should not add accidental ambiguity on top.</p>

<h2 id="what-it-checks-today">What it checks today</h2>

<p><code>probe-contract v0.1.0</code> checks a few things and stops there:</p>
<ul><li>a service declared in the contract is present in Compose;</li>
<li>that service has an active healthcheck;</li>
<li>a readiness rollout gate actually probes the declared readiness path;</li>
<li>liveness and readiness are not declared as the same endpoint by accident;</li>
<li>interval, timeout, and retries are present on the healthcheck.</li></ul>

<p>It emits human-readable text by default and JSON with <code>--format json</code>, which is enough to start using it in CI without making every warning a release blocker.</p>

<h2 id="what-it-intentionally-does-not-do">What it intentionally does not do</h2>

<p>I do not want this to become another linter that promises too much.</p>

<p>It does not inspect Kubernetes manifests yet. It does not send traffic to a live service. It does not infer a business transaction from a URL. And it cannot tell whether a dependency should be part of readiness for a particular application.</p>

<p>Those choices belong to the team running the service. The tool only asks them to make the choice explicit, then checks whether the Compose file agrees.</p>

<p>That scope is small enough to be useful. It is also small enough that a reviewer can understand what a warning means without trusting a black box.</p>

<h2 id="release-and-next-steps">Release and next steps</h2>

<p>The project is public under MIT and includes a Linux amd64 binary with a checksum:</p>
<ul><li><a href="https://github.com/heyimusa/probe-contract">Source and README</a></li>
<li><a href="https://github.com/heyimusa/probe-contract/releases/tag/v0.1.0">v0.1.0 release</a></li></ul>

<p>The next likely steps are Kubernetes manifest support, SARIF output, and a GitHub Action. I am deliberately not calling those features until the Compose contract is useful enough to earn them.</p>

<hr>

<p><strong>Test notes:</strong> I built and ran <code>probe-contract v0.1.0</code> only in Docker for this note. The demonstration container used a read-only filesystem, non-root UID <code>65532</code>, dropped Linux capabilities, <code>no-new-privileges</code>, and read-only mounted fixture files. The results are configuration checks against local fixtures, not a benchmark or a production deployment.</p>

<p><strong>Related work:</strong> <a href="/portfolio/work/#delivery-systems">Delivery systems case study</a> · <a href="/i-made-a-healthy-service-page-on-purpose">I made a healthy service page on purpose</a></p>


]]></content:encoded>
      <guid>https://heyimusa.blog/i-built-a-small-linter-for-health-checks</guid>
      <pubDate>Mon, 20 Jul 2026 13:26:02 +0000</pubDate>
    </item>
    <item>
      <title>I made a healthy service page on purpose</title>
      <link>https://heyimusa.blog/i-made-a-healthy-service-page-on-purpose</link>
      <description>&lt;![CDATA[One service, three health signals: liveness stays green while readiness and the checkout path fail&#xA;&#xA;Original diagram for a Docker-only experiment in this note. It models a failed dependency, not a production system.&#xA;&#xA;I made a service look healthy on purpose.&#xA;&#xA;Not healthy to a user. Healthy to the one check that only cared whether the process was still running.&#xA;&#xA;!--more--&#xA;&#xA;It is an easy trap. A container is up, the liveness endpoint returns 200, the dashboard stays green, and everyone gets to feel better for a few minutes. Meanwhile the dependency the service needs has gone away, requests are timing out, and the thing users came for is unavailable.&#xA;&#xA;I built a tiny version of that failure in a disposable Docker container. The service had three endpoints:&#xA;&#xA;/healthz   process liveness&#xA;/readyz    dependency-aware readiness&#xA;/checkout  a small user-facing path&#xA;&#xA;With the simulated dependency available, all three were fine:&#xA;&#xA;/healthz   http=200 body=process=up&#xA;/readyz    http=200 body=dependency=reachable&#xA;/checkout  http=200 body=checkout=accepted&#xA;&#xA;Then I restarted the same demo with one environment flag that simulated a dependency failure. The Python process did not crash. That is important.&#xA;&#xA;/healthz   http=200 body=process=up&#xA;/readyz    http=503 body=dependency=unavailable&#xA;/checkout  http=503 body=checkout=unavailable: dependency timeout&#xA;&#xA;The liveness check was not lying. It answered the question it was given: is the process alive? The problem is that it was the wrong question for deciding whether to send a user more traffic, or whether a rollout was safe to keep.&#xA;&#xA;Three questions that should not share one endpoint&#xA;&#xA;I find it useful to separate health checks by the decision they support.&#xA;&#xA;/healthz asks whether restarting the process is likely to help. A process that cannot accept a TCP connection, is deadlocked, or has stopped responding belongs here. This check should be narrow. If every temporary dependency wobble makes it fail, an orchestrator can turn a recoverable outage into a restart loop.&#xA;&#xA;/readyz asks whether the instance should receive work. In the experiment, it included the dependency state. That made it a reasonable signal for traffic admission and rollout progress. A failing readiness check can take an instance out of rotation without pretending that the process needs to be killed.&#xA;&#xA;/checkout asks the least convenient question: can someone complete the thing they actually came to do? It is usually more expensive to measure and should not become a noisy probe that hits every dependency every second. But some form of user-path signal belongs in monitoring. Otherwise a green fleet can hide a useless product.&#xA;&#xA;A green container is a weak promise&#xA;&#xA;Container state is still useful. It tells you whether a workload exists and whether the runtime can keep it alive. It does not tell you that the application has a connection pool, that the queue is moving, or that an important request succeeds.&#xA;&#xA;That distinction matters most during a deployment. If a new version starts successfully but cannot talk to its required dependency, a liveness-only check may let the rollout continue. The failure then moves from deployment time to user time, where it is noisier and harder to unwind.&#xA;&#xA;This is also why I do not like treating readiness as a cosmetic endpoint added late in a project. It is part of the contract between the application and the platform. The platform needs an honest answer before it decides to route traffic or declare a revision ready.&#xA;&#xA;Do not turn readiness into a dependency census&#xA;&#xA;There is a bad version of this pattern too: make /readyz call every downstream service, every time, and fail on any brief hiccup.&#xA;&#xA;That can create its own outage. If a shared dependency has a short blip, hundreds of instances may all become unready at once. A probe that was meant to reduce risk becomes a traffic switch with no damping.&#xA;&#xA;The useful questions are smaller:&#xA;&#xA;Which dependency makes this instance incapable of doing its primary job?&#xA;How long must that dependency be unavailable before traffic should stop?&#xA;Can the check use a bounded timeout and cached result instead of adding load during an incident?&#xA;What signal tells us the user path is degraded even if the process remains alive?&#xA;&#xA;The answers will differ by service. A worker may be alive and intentionally disconnected while a queue is paused. A checkout API with no database connection is not in the same situation.&#xA;&#xA;What I would wire into a real rollout&#xA;&#xA;For a non-critical environment, I would start with a deliberately boring drill:&#xA;&#xA;deploy a candidate revision;&#xA;make one required dependency unavailable in a controlled way;&#xA;confirm that readiness fails while liveness stays stable;&#xA;confirm that the traffic or rollout controller reacts to readiness, not just process survival;&#xA;restore the dependency and verify the user-facing path, not only the pod state.&#xA;&#xA;That drill has a nice side effect: it makes the rollback criteria concrete. In my previous note, I argued that a rollout needs a return address. This is part of the address book. If the only evidence after a rollback is “the pods are running,” there is still a lot left to guess.&#xA;&#xA;---&#xA;&#xA;Test notes: I ran this in two disposable Docker containers using Python 3.12. Each container used a read-only root filesystem, dropped Linux capabilities, no-new-privileges, a PID limit of 64, 0.25 CPU, and 128 MiB memory. The dependency failure was a local simulation. No host service, production endpoint, cluster, or external dependency was accessed.&#xA;&#xA;Related reading: A rollout needs a return address · Delivery systems case study&#xA;&#xA;!-- taxonomy: #topicplatformengineering --&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="/img/articles/health-signals-are-not-the-same.svg" alt="One service, three health signals: liveness stays green while readiness and the checkout path fail"></p>

<p><em>Original diagram for a Docker-only experiment in this note. It models a failed dependency, not a production system.</em></p>

<p>I made a service look healthy on purpose.</p>

<p>Not healthy to a user. Healthy to the one check that only cared whether the process was still running.</p>



<p>It is an easy trap. A container is up, the liveness endpoint returns <code>200</code>, the dashboard stays green, and everyone gets to feel better for a few minutes. Meanwhile the dependency the service needs has gone away, requests are timing out, and the thing users came for is unavailable.</p>

<p>I built a tiny version of that failure in a disposable Docker container. The service had three endpoints:</p>

<pre><code class="language-text">/healthz   process liveness
/readyz    dependency-aware readiness
/checkout  a small user-facing path
</code></pre>

<p>With the simulated dependency available, all three were fine:</p>

<pre><code class="language-text">/healthz   http=200 body=process=up
/readyz    http=200 body=dependency=reachable
/checkout  http=200 body=checkout=accepted
</code></pre>

<p>Then I restarted the same demo with one environment flag that simulated a dependency failure. The Python process did not crash. That is important.</p>

<pre><code class="language-text">/healthz   http=200 body=process=up
/readyz    http=503 body=dependency=unavailable
/checkout  http=503 body=checkout=unavailable: dependency timeout
</code></pre>

<p>The liveness check was not lying. It answered the question it was given: is the process alive? The problem is that it was the wrong question for deciding whether to send a user more traffic, or whether a rollout was safe to keep.</p>

<h2 id="three-questions-that-should-not-share-one-endpoint">Three questions that should not share one endpoint</h2>

<p>I find it useful to separate health checks by the decision they support.</p>

<p><code>/healthz</code> asks whether restarting the process is likely to help. A process that cannot accept a TCP connection, is deadlocked, or has stopped responding belongs here. This check should be narrow. If every temporary dependency wobble makes it fail, an orchestrator can turn a recoverable outage into a restart loop.</p>

<p><code>/readyz</code> asks whether the instance should receive work. In the experiment, it included the dependency state. That made it a reasonable signal for traffic admission and rollout progress. A failing readiness check can take an instance out of rotation without pretending that the process needs to be killed.</p>

<p><code>/checkout</code> asks the least convenient question: can someone complete the thing they actually came to do? It is usually more expensive to measure and should not become a noisy probe that hits every dependency every second. But some form of user-path signal belongs in monitoring. Otherwise a green fleet can hide a useless product.</p>

<h2 id="a-green-container-is-a-weak-promise">A green container is a weak promise</h2>

<p>Container state is still useful. It tells you whether a workload exists and whether the runtime can keep it alive. It does not tell you that the application has a connection pool, that the queue is moving, or that an important request succeeds.</p>

<p>That distinction matters most during a deployment. If a new version starts successfully but cannot talk to its required dependency, a liveness-only check may let the rollout continue. The failure then moves from deployment time to user time, where it is noisier and harder to unwind.</p>

<p>This is also why I do not like treating readiness as a cosmetic endpoint added late in a project. It is part of the contract between the application and the platform. The platform needs an honest answer before it decides to route traffic or declare a revision ready.</p>

<h2 id="do-not-turn-readiness-into-a-dependency-census">Do not turn readiness into a dependency census</h2>

<p>There is a bad version of this pattern too: make <code>/readyz</code> call every downstream service, every time, and fail on any brief hiccup.</p>

<p>That can create its own outage. If a shared dependency has a short blip, hundreds of instances may all become unready at once. A probe that was meant to reduce risk becomes a traffic switch with no damping.</p>

<p>The useful questions are smaller:</p>
<ul><li>Which dependency makes this instance incapable of doing its primary job?</li>
<li>How long must that dependency be unavailable before traffic should stop?</li>
<li>Can the check use a bounded timeout and cached result instead of adding load during an incident?</li>
<li>What signal tells us the user path is degraded even if the process remains alive?</li></ul>

<p>The answers will differ by service. A worker may be alive and intentionally disconnected while a queue is paused. A checkout API with no database connection is not in the same situation.</p>

<h2 id="what-i-would-wire-into-a-real-rollout">What I would wire into a real rollout</h2>

<p>For a non-critical environment, I would start with a deliberately boring drill:</p>
<ol><li>deploy a candidate revision;</li>
<li>make one required dependency unavailable in a controlled way;</li>
<li>confirm that readiness fails while liveness stays stable;</li>
<li>confirm that the traffic or rollout controller reacts to readiness, not just process survival;</li>
<li>restore the dependency and verify the user-facing path, not only the pod state.</li></ol>

<p>That drill has a nice side effect: it makes the rollback criteria concrete. In my previous note, I argued that a rollout needs a return address. This is part of the address book. If the only evidence after a rollback is “the pods are running,” there is still a lot left to guess.</p>

<hr>

<p><strong>Test notes:</strong> I ran this in two disposable Docker containers using Python 3.12. Each container used a read-only root filesystem, dropped Linux capabilities, <code>no-new-privileges</code>, a PID limit of 64, 0.25 CPU, and 128 MiB memory. The dependency failure was a local simulation. No host service, production endpoint, cluster, or external dependency was accessed.</p>

<p><strong>Related reading:</strong> <a href="/a-rollout-needs-a-return-address">A rollout needs a return address</a> · <a href="/portfolio/work/#delivery-systems">Delivery systems case study</a></p>


]]></content:encoded>
      <guid>https://heyimusa.blog/i-made-a-healthy-service-page-on-purpose</guid>
      <pubDate>Mon, 20 Jul 2026 10:12:13 +0000</pubDate>
    </item>
  </channel>
</rss>