Deployment health: prove a business operation before moving traffic

Choose the result that makes this release useful

An inventory service can answer its health endpoint while reservations remain stuck in a queue. The HTTP handler is running. The product still cannot promise stock to a buyer. A release decision needs an observation of that promise being kept.

Consider a reservation service with asynchronous processing. A request accepts one test reservation, a worker records the result, and a status endpoint exposes it. The release probe should follow that operation across those boundaries. This is a worked design, not a report of production measurements.

Google's SRE monitoring guidance distinguishes externally observed behavior from internal instrumentation. It also describes incorrect content returned with HTTP 200 as a possible error. Use the business result to decide whether the release worked; use queue, database and worker observations to investigate why it did not.

Write the probe's contract before its request

Give the probe a dedicated test tenant and a reserved inventory fixture. Its identity should authorize only the intended operation on that fixture. The path must exercise the components being changed, while every downstream effect stays within the test boundary. It must not send a customer notification, charge a payment method or reserve stock needed by a real order.

Define completion as a persisted reservation for the expected tenant, fixture and quantity, retrievable through the buyer-facing status path. Bind the result to an opaque command identity and the unchanged request content. A response saying accepted establishes admission; it does not establish that the worker completed the reservation.

Choose an observation deadline from the service's delivery promise and the release decision. Record the maximum admitted operations, total runtime, and cleanup responsibility. Keep the command identity and result evidence after cleanup so an interrupted probe can be reconciled. Cleanup is a separately recorded operation, not deletion of the evidence that explains the test.

Prove that the candidate handled the work

A successful probe served entirely by the previous release says little about the candidate. Retain the version that accepted the command, the worker version that processed it and the version that served the result. When the system cannot expose a required identity, record that attribution gap in the release decision.

Traffic percentages do not establish which version handled an individual operation. Cloudflare Workers, for example, supports dividing requests between deployed versions. Its gradual-deployment control is a routing mechanism; a team still has to choose observations that establish whether the changed behavior works.

If the candidate and previous release share a queue or database, name that shared state explicitly. Run the coexistence paths the deployment will permit. A candidate request processed by an older worker may be a useful compatibility observation, but it cannot replace a probe of a changed worker implementation.

Keep failure, uncertainty and missing checks distinct

A matching persisted result within the deadline passes this probe. A terminal rejection for an operation that should have succeeded is a failure. A result with the wrong tenant, command or reserved quantity is an integrity failure, even if the endpoint returns success. Stop the promotion and retain the smallest useful diagnostic record.

An accepted command whose result is still absent at the deadline has an unknown outcome. Stop promotion and query by the original identity. A transport timeout does not establish that no reservation exists. Any retry must use the service's explicit reconciliation or idempotency contract; a fresh command identity can create additional work.

A check that never started is another state. Retain when it was expected, whether a run began and whether that run produced a terminal observation. Do not turn a missing result into a pass or label it a reservation failure before identifying which component stopped progressing. A watchdog for missing checks needs a clock and delivery path that do not rely solely on the same scheduler.

Rehearse the decisions in a controlled environment

Before using the gate for a rollout, demonstrate its decisions against controlled fixtures. A normal reservation should produce the expected persisted result. A deliberately delayed worker should leave the gate waiting, then stop promotion when its observation deadline passes. A mismatched fixture result should fail even when its HTTP response is successful.

Exercise a lost acknowledgement after acceptance. The probe must retain its original command identity and investigate that operation before another admission. Restart the probe runner and verify that its retained state still identifies the outstanding command. These are test requirements for the design; this article does not claim those rehearsals have run.

Also disable the probe's trigger in the controlled environment. The operator should see a missing-check condition with the expected time and responsible scheduler. That rehearsal tests whether anyone can distinguish an unavailable business path from an observation system that has stopped checking it.

Use the probe inside a broader promotion decision

One successful reservation cannot establish capacity, rare-error frequency or every tenant's configuration. Google's canary guidance calls for a representative population and observation duration, with metrics that distinguish the candidate from the control. Choose those conditions from the workload and the failures the release could introduce; a quiet interval is weak evidence when the relevant work never occurred.

Combine the bounded probe with observations of real completed operations, error outcomes, queue age and resource pressure where those signals change the promotion decision. State the population and denominator for each measure. Keep synthetic reservations identifiable so they do not inflate commercial transactions or hide an absence of real traffic.

The gate should identify the action for each failure state and the operator who owns it. Stopping traffic movement does not undo reservations already accepted or changes already persisted. Keep their reconciliation and cleanup procedures available through recovery.

Make the release decision reviewable

For this reservation path, retain the candidate and previous release identities, fixture scope, completion rule, observation deadline, actual command identity and terminal result. Include the versions observed at each stage and any attribution gap. Link that record to the controlled rehearsal and the proposed recovery action.

An assessment can start with one disputed release signal: the service appears healthy, but the business result is uncertain. Trace that operation from admission to its persisted result, identify what the current checks miss, and specify the evidence required before more traffic moves. The deliverable is a gate the release owner can execute and interpret for that path.