"The model update that changed everything"

Nothing in my codebase changed on the day one of my automations started behaving like a different system. No deploy, no config change, no prompt edit. The provider had upgraded the model behind the API alias I was pointing at, and the component at the heart of the workflow had been swapped out from under me overnight.

The system in question is an LLM automation that processes incoming email for my own business, classifying messages, extracting fields, and drafting short summaries that I read every day. Those summaries were how I noticed. They had always been terse, almost curt, because that is what the prompt asked for. One morning they arrived warmer, longer, and sprinkled with helpful little caveats. Pleasant, arguably. But I had not asked for pleasant, and if the tone had shifted, I had no idea what else had.

The honest answer is that I would have had no way to find out, except that by this point I had an eval suite: a few hundred real, labelled inputs with known correct outputs, runnable against any model version on demand. I ran it against the new model that morning. Overall accuracy looked fine, down a fraction of a point, nothing you would notice in a demo. The per-case diff told a different story. Three distinct regressions, none of which overlapped with the others.

The first was the most dangerous. My extraction prompt instructs the model to return UNKNOWN for any field it cannot find in the source text. The old model obeyed this reliably. The new one was noticeably more eager to please: on a chunk of the cases where the correct answer was UNKNOWN, it now produced a confident guess instead. The guesses were plausible, which is precisely what made them worse than errors. That single behaviour change converted honest gaps into silent misinformation.

The second regression was a classification boundary that moved. Two of my categories are genuinely adjacent, and the old model had settled into a particular reading of the borderline cases, a reading my downstream routing had been tuned around. The new model read the same borderline cases differently. Neither reading is objectively wrong; the point is that the boundary shifted and everything built on top of it did not.

The third was mechanical. The new model had a slightly stronger habit of wrapping JSON output in markdown code fences and prefacing it with a friendly sentence. My parser handled fences, but the combination pushed a small percentage of responses into the retry path, which meant more latency and more cost for identical work. Small, but it compounds at volume.

The fixes were straightforward once the problems were visible. I moved every workflow off convenience aliases, the pointers a provider quietly re-aims at the newest snapshot shortly after a release, and onto pinned, dated model versions, so an upgrade is something I choose rather than something that happens to me. It takes thirty seconds to check what your model string actually resolves to, and in the codebases I have reviewed since, it is wrong surprisingly often. I tightened the extraction prompt against guessing and added an eval case for each regression. And I wrote down an upgrade ritual: when a new version appears, run the full eval suite against it side by side with the current one, read the per-case diff rather than the headline number, fix or accept each behavioural change deliberately, then switch, with the old version held in reserve for a fast rollback.

Pinning is not a way to avoid upgrades, to be clear. Providers deprecate and retire versions on their own schedule, and once a version is marked deprecated the guaranteed notice before it stops answering can be as short as sixty days, depending on the provider, so every pin has an expiry date and the upgrade will eventually be mandatory. What pinning buys you is the ability to do that upgrade on a day you chose, with a test suite in hand, instead of discovering it in production through a change of tone. It converts a surprise into a release.

The transferable lesson is that a model version is a dependency, and it deserves at least the care you would give a major library upgrade, nobody bumps a database driver in production without running the tests. The difference is that a library changes behaviour when you update it; a floating model alias changes behaviour when someone else does. Aggregate accuracy will not warn you, because aggregate numbers hide boundary shifts and instruction drift. Only labelled examples run against both versions, and a human reading the diff, will show you what actually changed.

If you have LLM features in production and no way to answer "what would break if the model changed tomorrow", that is exactly the work I do, building the eval suites and upgrade discipline that make model changes boring. Get in touch via the contact form.