A model upgrade needs a regression register
A model upgrade can fix some cases and break others, so keep a paired record of both before replacing the model in a working application.
The provider recommends a new model, and the old endpoint has a retirement date. That raises two separate questions: when you must move, and what changes when you do. They need separate fields in your own records. A deadline makes the test urgent without supplying the result.
Anthropic's deprecation guide, read on 14 November 2026, distinguishes deprecated models that still work from retired models whose requests fail. It also notes that partner-operated platforms can have different schedules, so the date that counts is the one for the route your application uses, not one borrowed from another platform.
What changed case by case?
The test starts from a fixed set of representative inputs and one acceptance rule, meaning the written definition of a pass. The existing configuration and the proposed replacement both run on it. Each result keeps the model identifier, route, prompt, settings and date. If the provider requires a prompt or parameter change, that goes in the record too, because the test is then of a replacement configuration and not only a model swap.
The table is an invented paired test of 100 cases, small enough to inspect. Each case has an old result and a new result, then falls into one of four groups.
| Outcome in the fictional paired test | Cases |
|---|---|
| Both pass | 88 |
| Old only passes | 7 |
| New only passes | 4 |
| Neither passes | 1 |
The old candidate passes 95 cases and the new one 92, a change of -3 percentage points. For the migration plan, the split matters more: seven formerly passing cases now fail, while four formerly failing cases improve. Those are different sets of work.
Naming each regression and who fixes it
A regression is a case that used to pass and now fails. For each one, the input, the expected result and what changed are worth saving, with the failure labelled in terms of the job: wrong field, missing evidence, invalid tool argument, unfinished answer or unacceptable delay. A label like “less intelligent” gives the person fixing the workflow little to act on.
Some losses should block the release and some should not. A minor wording change and a wrong amount do not belong in the same average, and the blocking examples should stay visible even if a later run produces a better overall pass rate.
The gains deserve the same scrutiny. A new answer that merely looks better may still miss the acceptance rule. If the rule changes after you have seen the results, the earlier candidate has to be rerun under the new rule before anyone claims an improvement.
Tuning the prompt is a different experiment
Tuning the prompt for the new candidate means the original test result stays on file, and the next run counts as a new configuration with its own version. Some cases are held back from tuning so they can show whether the repair extends beyond the examples you worked on.
Variation needs repeats on the important cases. One pass or failure on one attempt is an observation, not a promise about the next request, so the repetitions and their outcomes should be reported instead of a single favourable answer.
Cost belongs in the same record, counted as spend for the complete test, failed attempts and recovery included. A replacement with a lower token rate may still write more or need more attempts on your task. That has to be measured, because the old token counts will not necessarily carry across.
Getting ready for the retirement date
A migration checklist holds the last verified deadline, the affected application, the replacement configuration and the unresolved regressions. The provider page is worth rechecking as the date approaches. An old screenshot can explain your plan but cannot show what is available today.
A rollback route helps only while it stays available. Once the old model retires, recovery may need a different provider or a reduced feature set, so that route should be chosen in advance and the part you intend to rely on tested. The register itself should end in a dated decision with evidence, not a model name copied from a release announcement.