Debugging AI in productionPractice question (not reported)hardChecked 2026-11-17
After upgrading to a newer model version, the customer's workflow got worse on some tasks. How do you handle it?
Answer it out loud first, then compare. That's what makes it stick.
30-second answer
I treat the upgrade as a change that needs evidence. I run the same golden set on both models, side by side, and diff results by task type to see exactly what regressed. Usually prompts tuned for the old model need adjusting, so I fix those, and for anything still worse I keep the old pinned version for that route. I roll out with a canary and monitor quality metrics before moving all traffic.
Framework to remember
- Reproduce: same inputs, both models
- Diff by task type
- Adapt prompts; re-run suite
- Route: new model where better, old where not
- Canary, monitor, then migrate
Deep answer
Newer is not automatically better for this customer's tasks.
1. Reproduce
- Replay a frozen set of real production inputs through old and new models with identical prompts, tools and settings. Same temperature, same retrieved context. Now the model is the only variable.
2. Diff
- Score both with the same graders. Break results by task type, input length, language and customer segment. A 2-point average gain can hide a 15-point loss on one critical task.
- Read 20-30 side-by-side failures. Common causes: different verbosity, a changed output format, stricter or looser instruction following, different tool-calling habits, different refusal behaviour.
3. Adapt
- Prompts tuned on the old model often carry workarounds the new one doesn't need or reacts badly to. Simplify, make format requirements explicit, use structured outputs, then re-run the suite.
- Check parameters: some newer models restrict sampling settings or change defaults, so confirm in the current docs.
4. Decide per route
- Keep the old model where it still wins. Anthropic's model-ID docs say dateless IDs for recent models point to a fixed snapshot and weights are not changed under an existing ID, so pinning is a real control. Pinned versions are retired on a published schedule though, so keep a migration date on the calendar.
5. Roll out safely
- Shadow traffic first, then a canary at 5-10%, with auto-rollback thresholds on quality, error rate, latency and cost per task. Keep the old path live until the canary is clean.
6. Own the story
- Show the customer a before/after table on their own data. Add every regression to the golden set so the next upgrade is a one-command check.
Strong answer
Replays identical inputs on both models, diffs by task type, adapts prompts, pins or routes per task, and rolls out with a canary and rollback thresholds.Weak answer
Rolls back and refuses to upgrade ever, or blames the provider without data.The trap
Judging the new model on an overall average, which hides regressions on the customer's most important tasks.Likely follow-ups
- The old model is being retired in 60 days. What is your plan?
- How large must your golden set be to trust the comparison?
- Costs went up 30% with the new model. How does that change the decision?
Sources (2) · confidence high
- Model IDs and versions (Claude Platform Docs) (docs, unknown) Explains that recent dateless model IDs map to a fixed snapshot and that existing model IDs are not updated.
- The Model Deprecation Treadmill (prep, 2026-04-27) Discusses preparing for model retirements before the sunset notice arrives.