Skip to content
atom®technologiesBook a call
Engineering notes

Production AI model fallback: preserve the application's decision

A backup model can return valid JSON and still change what the application does. Define semantic compatibility, abstention, and a shared latency budget before enabling provider fallback.

A packed auditorium during a technology keynote

Define what a fallback is allowed to decide

An order-intake application extracts requested delivery dates from customer messages. Its primary model becomes unavailable, so traffic moves to a backup provider. Both can return a date-shaped field. The application still needs to know whether that date was supplied by the customer, inferred from an ambiguous phrase, or absent from the message.

For this design exercise, the application accepts three outcomes. An explicit, unambiguous calendar date becomes extracted, with the source text that supports it. A complete message containing no delivery-date request becomes not_requested. Ambiguity becomes needs_review. The application decides what those outcomes permit; neither model can submit an order or choose a delivery promise.

A message saying 'next Friday' requires review under this contract because relative dates are outside the accepted input rules. If a backup returns not_requested, a downstream default could erase the customer's stated preference. That is a semantic compatibility failure even when the response satisfies the JSON schema.

Keep the application contract above provider adapters

Google's Gemini structured-output documentation explicitly calls for application validation of values and handling of semantically incorrect output. A provider's schema constraint helps with representation. It does not establish that the application should accept the interpretation.

Define the domain result once, then adapt each provider's response into it. Require a real calendar date for extracted, reject unsupported status values, and verify that its evidence references text in the submitted message. Keep not_requested distinct from an extraction failure. These checks reject malformed answers; they cannot prove that a model noticed every date request or interpreted the evidence correctly.

Bind both attempts to the same complete message and policy version. Record the model identifier and adapter configuration used by each. Check that the backup can accept the input size and supported language. Silently truncating a long message can remove the customer's date request and turn missing context into an apparently valid not_requested result.

The alternate provider also needs approval to process that class of customer data. Its retention and regional processing requirements belong in fallback eligibility. An outage should not expand the set of recipients authorized to receive the message.

Classify the first result before attempting another

Separate transport unavailability from a completed domain result. An accepted needs_review outcome ends automatic extraction for this request. Trying providers until one supplies a date would change the abstention policy. Invalid output can enter a bounded recovery path, but the backup must satisfy the same acceptance rules.

Anthropic's structured-output documentation describes refusals that arrive with HTTP 200 and output that may not follow the requested schema. It also documents incomplete output when max_tokens stops generation. Inspect the provider's completion status before parsing the domain result. An HTTP success alone cannot establish that extraction finished.

Keep a provider refusal distinguishable from technical unavailability and send it through the application's documented refusal policy. Do not convert it into not_requested. If no eligible provider can return an acceptable result, expose the review or unavailable outcome that the caller can handle. Preserve the technical reason internally so an outage does not disappear into ordinary ambiguity.

One application decision across provider attemptsEach attempt uses the same message and semantic contract. Abstention remains a valid outcome, and the original request deadline never resets.
  1. Bind
    Fix the work and deadline

    Retain the complete input, policy version, permitted providers, and request deadline.

  2. Classify
    Interpret the first outcome

    Accept a valid result or abstention. Separate unavailable, refused, truncated, and invalid responses.

  3. Admit
    Check fallback eligibility

    Require an allowed failure reason, compatible provider, and remaining time, cost, and capacity budgets.

  4. Decide
    Apply the same acceptance rules

    Validate the backup result or return the documented review or unavailable outcome.

Spend the remaining deadline deliberately

A fallback adds another attempt inside the customer's original waiting time. As illustrative budget arithmetic, allow 3,000 milliseconds for the whole request. After 1,400 milliseconds have elapsed, reserving 200 for validation and response delivery leaves 1,400 for a backup attempt. A configured 1,200-millisecond attempt fits. After 2,000 milliseconds have elapsed, only 800 remain, so that same attempt no longer fits.

Use a monotonic deadline and include connection setup and provider waiting in the attempt budget. Recheck time before accepting a result. Cancellation should end local waiting, and late responses must not replace a result already returned. A timeout setting and a reserve are planning controls; measure the complete path to establish whether the service meets its latency objective.

Limit attempts across the SDK and application together. Google's SRE guidance describes both request-level retry limits and aggregate retry budgets because retries add load during failures. Apply that reasoning to the backup lane: check its available capacity and rate limits before sending an outage's displaced traffic there.

Price the behavior the backup actually preserves

Account for input and output charges from both attempts where billed, including work that ultimately returns review or unavailable. A cheaper backup can increase manual review if it cannot preserve the accepted interpretation rules. Compare cost per accepted extraction alongside review volume and deadline misses. Keep the mix of customer messages fixed when comparing these outcomes.

Rehearse fallback with an explicit date, an ambiguous relative date, and a complete message with no requested date. Add a message whose relevant sentence would be lost by truncation. Force the primary to time out and check the resulting application decision, then exhaust the deadline and confirm that no new attempt starts. Retain both provider responses and the application outcome under the approved data-handling policy.

Before enabling fallback, the owner needs evidence that the backup preserves those distinctions within the agreed cost and latency limits. Assign someone to disable the backup lane when review volume or deadline misses exceed the approved bounds. Keep the documented unavailable outcome usable while the team investigates compatibility.