Skip to content
atom®technologiesBook a call
Production AI

Fallback-served LLM requests at release: separate passes, cost per task, and rate-limit headroom

Before approving a production LLM feature, count fallback-served requests separately from primary-model passes. The release record should show the fallback step, cost per contract-satisfying task for each path, and rate-limit headroom for each model.

A packed auditorium during a technology keynote
The short answer

No, not as primary passes. Approve only when the record separates primary success from fallback success, shows which step served each request, reports cost per contract-satisfying task for each path, and documents rate-limit headroom per model, with thresholds set independently for each path.

The decision, answered directly

Approve the release only when fallback-served requests are counted separately from primary-model passes. A request that succeeded after a fallback is a real success for that user, but it is not evidence that the primary model meets the bar; if your pass metric aggregates both, a primary regression can hide behind fallback success. The release record needs the fallback step per request, cost per contract-satisfying task for each path, and rate-limit headroom for each model.

What actually changes when a request is served by a fallback

Cloudflare's AI Gateway fallback documentation, last updated April 20, 2026, states that Cloudflare can trigger your fallback provider in response to request errors or predetermined request timeouts, and that by default a fallback fires if a model request returns an error. The timeout-triggered case is documented as a capability, not a default, so treat timeout-triggered fallback as something to configure and verify rather than assume. The documentation also shows that you can add as many fallbacks as you need by adding another object in the array.

The telemetry that makes this gate possible is the response header cf-aig-step, which indicates which step successfully processed the request: 0 means the primary model was used successfully, 1 means the request fell back to the second model, and each further fallback increments the step number. One scope caveat matters here: Cloudflare documents this header specifically when using the Universal endpoint with fallbacks. If your requests go through a different endpoint or provider, this header is not documented as available, and the gate cannot run until you verify an equivalent path marker exists or add one yourself. The following is an inference, not a documented guarantee: separating primary from fallback success is only possible if your application captures cf-aig-step or an equivalent marker and joins it to the task outcome in your own logs. If the header is dropped at a proxy or never persisted, the release record cannot distinguish the paths, and that gap itself is a finding to fix before approval.

Cost per contract-satisfying task, computed per path

Cloudflare's Workers AI pricing page, last updated November 12, 2026, states that Workers AI is priced at 0.011 dollars per 1,000 Neurons, with a free allocation of 10,000 Neurons per day, charges above that allocation on the Workers Paid plan, and limits that reset daily at 00:00 UTC; exceeding a limit makes further operations fail with an error. The page lists materially different prices per model. For example, @cf/meta/llama-3.1-8b-instruct-fp8-fast is listed at 0.045 dollars per million input tokens and 0.384 dollars per million output tokens, while @cf/meta/llama-3.3-70b-instruct-fp8-fast is listed at 0.293 dollars per million input and 2.253 dollars per million output. That is roughly six and a half times the input price and just under six times the output price between those two models on the same page. Some deepseek models list separate cached-input prices, such as 0.006 dollars per million cached input tokens against 0.300 dollars per million uncached input for deepseek-v4.1-flash-1017.

This supports a reasoned inference: because per-model prices differ, a fallback to a different model changes the cost of the same task, in this comparison by several times rather than a rounding difference. The proposed metric is cost per contract-satisfying task, computed per path: sum the spend attributable to requests served by the primary model, divide by the count of those requests that met the product contract, then repeat for each fallback step. Counting only contract-satisfying tasks prevents a cheap but useless path from looking efficient. This is a proposed diligence metric, not a provider requirement, and the figures above are Cloudflare's listed prices for its own models, not a measured cost for any workload.

Rate-limit headroom for each path

Cloudflare's Workers AI limits page, last updated September 17, 2026, distinguishes two kinds of ceilings, and the release record should keep them apart. Default limits are set per task type: text generation carries a default of 300 requests per minute unless the model requires the Workers Paid plan, while other task types range from 720 to 3,000 requests per minute. Separately, models that require the Workers Paid plan are limited per account and per model, at 20 requests per minute under standard Workers AI billing or 50 requests per minute when paid via prepaid AI Gateway credits with the gateway's billing setting set to Unified billing. The page also notes that beta models may have lower rate limits while performance and scale work continues.

The inference for the release gate: a fallback shifts traffic to a model whose ceiling may be far below the primary's, and for paid models that ceiling is per account and per model, so two models in the same chain do not share a pool. Headroom must therefore be recorded per path, using whichever limit applies to that model, at the billing mode you actually run, and compared against peak requests per minute observed in the release test. This is a proposed check; no source establishes your traffic shape or your provider's capacity beyond the documented limits, and vendor limits can change.

Side effects when fallback or retry re-executes a tool call

The Amazon Builders' Library article "Making retries safe with idempotent APIs," by Malcolm Featonby, states that retrying a service call as mitigation for a transient fault rests on the simplifying assumption that the operation can be retried without side effects, and discusses idempotent API operations as the way to mitigate undesirable side effects of retrying. The article's own example is the risk of retrying an instance launch and ending up with two instances.

This is an analogy, not an AI-specific requirement: the article is about service retries, not model fallbacks. But if your LLM feature calls tools that write, charge or send, a fallback that re-executes a partially completed call has the same shape of problem. The proposed check before approval is that every side-effecting tool call carries an idempotency key or deduplication rule, so a fallback-served retry cannot double-apply the effect.

The release record and who owns each part

This section is a proposal, assembled from the evidence above rather than a standard. The feature team owns outcome telemetry: task result joined to the fallback step, so primary and fallback success rates are separately reportable. The platform or infrastructure owner owns the cost and capacity lines: cost per contract-satisfying task per path, and rate-limit headroom per model at the current billing mode, distinguishing task-type defaults from per-account per-model paid limits. The approver, whether that is the CTO, VP Engineering or Head of Data, owns the thresholds, and the proposed rule is that they be set independently for the primary and fallback paths, with fallback success not compensating for primary failure unless the product decision explicitly accepts degraded fallback.

When the approach changes

Three conditions change the recommendation. First, if the product decision explicitly treats fallback output as equivalent to primary output, document that decision and the evidence behind it, and the separate thresholds can collapse into one; that is a product call, not an engineering default. Second, if your provider or endpoint offers no documented equivalent of cf-aig-step, the gate cannot be run as described until you add path marking yourself, and the honest release record says so. Third, treat all Cloudflare figures here as vendor documentation for Cloudflare products, not independent proof of general behavior; prices and limits may change, so verify current values and your own provider's equivalents before relying on them. For broader context, NIST's page on the AI Risk Management Framework describes the framework as intended for voluntary use and aimed at improving the trustworthiness of AI, with a January 26, 2023 announcement of guidance seeking to cultivate trust while mitigating risk; that is useful framing for the approval conversation, but it is not a release gate and does not substitute for the path-specific evidence above.

Checks

  1. Capture the fallback step per request, such as the cf-aig-step header on Cloudflare's Universal endpoint, and join it to the task outcome in storage you control; verify an equivalent marker exists if you use another endpoint or provider.
  2. Report primary and fallback success rates against separate thresholds; do not let fallback passes offset primary failures.
  3. Compute cost per contract-satisfying task for each path using current per-model prices, including any cached-input pricing that applies.
  4. Record rate-limit headroom for each model in the chain at your actual billing mode, distinguishing task-type default limits from per-account per-model paid limits, against peak observed requests per minute.
  5. Verify every side-effecting tool call is idempotent or deduplicated before any fallback or retry can re-execute it.

Questions

How do I compute cost per contract-satisfying task for a fallback path?

Take the spend attributable to requests served at that fallback step, divide by the number of those requests that met the product contract, and report it beside the same figure for the primary path. Use your provider's current per-model prices; Cloudflare's Workers AI pricing page, for example, lists per-model token and Neuron prices that differ widely between models.

Does a fallback success count toward the pass rate at all?

Yes, but in its own column. Cloudflare's fallback documentation provides cf-aig-step, on the Universal endpoint with fallbacks, so you can tell which model served each request; the proposed rule is that fallback-served successes report against the fallback threshold, never silently into the primary pass rate.

What if the primary model is fine but the fallback path is over its rate limit?

That is still a release blocker under this gate. Cloudflare's limits page shows paid models can be capped at 20 requests per minute per account and per model depending on billing mode, so a fallback path can saturate while the primary has headroom; record headroom for both.