The same model is often sold by several providers. fable-5 is available from Anthropic directly, on Amazon Bedrock, and through OpenRouter. Each of those routes has its own rate limits, its own credits, its own availability, its own latency, and its own error formats. So a provider outage should slow you down at worst. It should never make the model unavailable. This post describes how our gateway does that, with the code.
Retries are not enough
When a provider call fails, there are three possible responses. You can retry the same provider, which works for a blip and does nothing for an outage: your retry dies with the provider. You can fail over to the same model on a different provider, which is what this post is about. Or you can silently switch to a different model, which is the dangerous one: the caller asked for a specific model, tested against it, and tuned prompts for it. Swapping it under them changes behavior without telling anyone. The gateway does the second and is built so the third cannot happen by accident.
Which failures are allowed to fail over
The rule the code enforces: a request spills to the next provider only when the failure belongs to the provider binding, never when it belongs to the caller's request. Provider-binding failures are 429s, 5xx, timeouts, connection failures, and streams that end without a terminal event. They also include provider-credential 401s and 403s and provider-side 404s. That is broader than capacity failures on purpose: a misconfigured credential or a renamed model ID on one provider says nothing about your request, and another rung can serve it. Caller failures, meaning every other 4xx, return a corrective 400 and are never counted against the provider's health. From exp/runtime/models/providers/errors.py:
def _transport_failure(status_code: int | None) -> GatewayFailure: """Classify one sanitized HTTP or connection failure by status only. Remaining 4xx statuses are the caller's request being rejected, so they map to INVALID_REQUEST: the caller gets a corrective 400 and the deployment health circuit never counts the attempt against the deployment.""" if status_code in {401, 403}: return GatewayFailure( failure_class=GatewayFailureClass.PROVIDER_AUTHENTICATION, safe_message=( "provider authentication failed; ask the gateway operator " "to verify the provider connection credential" ), failover_eligible=True, ) if status_code == 429: return GatewayFailure( failure_class=GatewayFailureClass.THROTTLED, safe_message=( "provider throttled the request; retry after the delay " "in the Retry-After header" ), failover_eligible=True, )
Refusals do not fail over by default. A model declining a request is model behavior rather than a provider fault, and the next rung serves the same model. There is an opt-in flag for teams that want refusal failover anyway; it is covered below.
The waterfall
Each rung of a model's chain is a deployment: one way to reach one exact model. For fable-5 the chain is Anthropic on your key, then Bedrock on pooled accounts, then OpenRouter on your credits. An ordered set of deployments is an ExactModelPool, and the pool is where model identity is enforced. Every deployment in a pool carries the same exact_model_id, and the schema refuses to build a multi-deployment pool unless an operator has certified the deployments as equivalent. From exp/common/models/gateway_catalog.py:
class ExactModelPool(ContractModel): """An ordered set of deployments certified as one exact logical model.""" pool_id: ExactModelPoolId exact_model_id: ExactModelId deployment_ids: tuple[DeploymentId, ...] = Field(min_length=1) equivalence: GatewayEquivalenceCertification | None = None @model_validator(mode="after") def _require_unique_deployments(self) -> ExactModelPool: if len(set(self.deployment_ids)) != len(self.deployment_ids): raise ValueError("exact-model pool deployments must not repeat") if len(self.deployment_ids) > 1 and self.equivalence is None: raise ValueError( "multi-deployment pools require operator equivalence certification" )
Failover picks the next deployment from the same pool, so a rung cannot point at a different model. The routing code states the contract in its own words: choose a safe retry or later exact deployment without changing logical model.
Failover during streaming
Streaming constrains failover, and the boundary is the first semantic event. Until a text delta, a tool call, or a visible refusal has reached the client, the gateway can retry the same rung or advance to the next one freely. Sequence numbers are rewritten into one public sequence, so a provider swap before that point is invisible on the wire. The first semantic event commits the route. After that, failover is off: if the provider dies mid stream, the request ends as a truncated stream. There is no resume and no silent restart, because a restart could replay or reorder tokens the client already has.
The opt-in refusal flag works inside the same boundary. With it set, refusal deltas are buffered rather than sent, bounded at 64KB or 256 events, and the route advances to the next rung without the refused output ever reaching the client.
Billing through failures
Every physical dispatch is reserved in a durable ledger before the provider is called, at a conservative worst-case cost, and settled exactly once afterward. The settle-once path is tested end to end against the failure cases that produce double charges elsewhere: an empty completion settles at zero, a truncated completion settles at exactly the delivered tokens, a provider error after retries settles at zero with the reservation released, and a post-commit crash with no usage received settles at zero.
Caller retries have their own contract. /v1/chat/completions and /v1/responses accept an Idempotency-Key header. The same key with different content is a typed conflict. The same key with the same body replays the completed response exactly, from a bounded in-process replay store; concurrent callers wait on the owner and get the same result, and a failed owner is abandoned fail-closed rather than retried under the caller's key. The durable ledger itself stores a hash of the request and no content. Replays never touch the ledger or the budget, so a replay cannot double a charge. Each rung also derives a stable idempotency key for its upstream call, reused on same-rung retries and fresh for each new rung; whether the provider honors it is up to the provider.
Tool calls across providers
Failover assumes the next provider produces the same wire behavior, and by default it does not: tool calls and structured outputs differ across the OpenAI, Anthropic, Bedrock, and Gemini wire formats. The engine normalizes every provider into one internal event stream, with per-provider mappers on the way in, and re-encodes that stream into the caller's dialect on the way out, whether the caller speaks OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages. A dated certification matrix records which capabilities are proven per provider, and its cells admit unproven states, including untested for lack of credentials.
Knowing a provider is down
Health tracking runs in process with no I/O, so consulting it costs nothing on the request path. Two consecutive operational failures open a deployment's circuit for thirty seconds, followed by one half-open probe. Throttling gets its own separate thirty second window, so a 429 storm does not open the failure circuit. Provider-credential failures and 404s suppress a deployment immediately. And a fully suppressed route still admits bounded last-resort probes, so a model never fails purely on circuit state while a provider has already recovered.
When credits run out
Running out of credits looks like a failure but routes differently, and the scope decides. Exhaustion scoped to one deployment advances the waterfall to the next rung. Exhaustion scoped to the organization, a key, or a model stops routing and returns 429 insufficient_quota: the gateway never silently moves your traffic onto a paid lane you did not choose. Bring-your-own-key traffic never reaches the credit gate at all.
What the ledger keeps
All of the above is observable without retaining content. Both ledgers are content-free by contract: per-attempt rows carry the provider, the exact model, the route depth, the failure class, token counts, and cost, and there is no prompt or response column anywhere in the schema. The idempotency store keeps only a hash of the request. A test plants secret canaries in requests and asserts they persist nowhere.
The gaps
Five things are weaker than the design above suggests. Equivalence across providers is operator-attested configuration, dated and carrying an evidence hash; nothing samples two providers and compares outputs. Customers see the final provider and an attempt count, while the full ordered attempt trail is recorded internally but not yet exposed. Quota rejections lose their specific reason at the HTTP boundary, so the response carries a generic quota message even though the ledger knows which limit fired. Non-streaming requests currently ride the streaming path, which re-pays the provider's inter-token pacing; that is a known performance cost. And the chaos suite faults one deployment at a time; multi-provider failover under load has functional coverage and no load drill.
Hardening the rungs
Failover shipped, and running it in production taught us one lesson twice: a waterfall is only as strong as the metadata on its rungs. The health circuits above pull a rung that errors or throttles. Two consecutive operational failures open a deployment's circuit for thirty seconds, and a single half-open probe earns it back. But a rung can be up and healthy and still be one the router must not use, because it cannot be priced. That is the failure mode this section is about.
Billing through failures, above, reserves a worst-case cost before each dispatch. That reservation needs a number. The old behavior, when a rung had no known price, was to treat its worst case as zero. That is fail-open, and it is dangerous: a route priced at zero slips every gate, because zero is under the balance, under the daily cap, and under every scope budget, so it can overspend real money. We changed it to fail closed. A host-funded attempt with no bounded cost is refused before any budget check runs:
if p_maximum_cost_micro_usd is null then raise exception using errcode = 'P1013', message = 'deployment_price_unknown: this route has no known ' || 'price, so its spend cannot be bounded against the credit ' || 'balance, a daily cap, or a scope budget; it is ineligible ' || 'and another route may serve the request';end if;
The message says what happens next. The route is ineligible, and another route may serve the request. An unpriced rung is not an error the caller sees; it is a rung the waterfall skips, exactly like a dead one, and the request falls over to the next priced rung. Only when no rung can be priced does the caller get an error. Skipping a rung costs a little coverage; guessing its cost and being wrong overspends a customer's budget, and that asymmetry is why the unknown-price case now refuses instead of assuming zero.
Fail-closed also made a quiet gap loud. The worst-case reservation is a model's output-token ceiling times its output rate. When a request does not set max_tokens and the model carries no output ceiling of its own, the worst case is null: there is nothing to bound, so P1013 skips the rung. Under the old fail-open rule that rung would have routed at a phantom cost of zero; failing closed turned the gap into a visible skip and pointed straight at the models with no ceiling. The fix is to coalesce a missing ceiling to a conservative upper bound, the model's context window, since output can never exceed context, with any real and smaller provider output cap winning. That work is in progress.
When we backfill a model's ceiling or correct a rung's price, the running gateway picks it up without a restart. It polls a watermark every fifteen seconds, one small query over the catalog tables that returns their row counts and newest update time. If nothing changed it does nothing; if the watermark moved it reloads the affected rows, rebuilds an immutable snapshot, and swaps it in with a single assignment, and a failed refresh keeps the last good snapshot:
class GatewayCatalogRefresher: def __init__(self, *, poll_interval_seconds: float = 15.0): self._poll_interval = poll_interval_seconds def refresh_now(self) -> bool: watermark = read_catalog_watermark(self._connection) if self._state is not None and watermark == self._watermark: return False # nothing changed; no work, no swap ... # reload rows, rebuild the snapshot, swap it in return True
So a backfilled ceiling, a corrected price, or a new rung added to a waterfall, goes live on the next poll with no deploy. One rung failure we still catch only slowly: a provider that accepts the connection and then never sends a token is bounded today only by the overall request deadline, around thirty-five seconds, because the health circuits key on errors and throttling, not silence. A first-byte bound to fail those lanes over quickly is rolling out. It is not done.
The failover engine described here, the classification, the pools, the streaming boundary, the ledger, the health circuits, is open source in experientiallabs/experiential, first shipped in release 0.5.0 with 0.5.2 current, and it is the same engine we host: since 0.5.1 the hosted data plane runs the repo's native Rust implementation, with the Python engine as its fallback.