"Regression testing prompts: catching the fix that breaks everything else"
You fix the reported failure, ship the prompt change, and three other cases quietly break. Prompts are global configuration: every edit changes behaviour everywhere. The only defence is re-running the full golden set on every change, and reading the per-example diff.

A user reports that the system mishandles a particular supplier's invoices. An engineer reproduces it, adds a line to the prompt, confirms the failing case now passes, and ships. Two weeks later three other suppliers' invoices are being mishandled in a new way, and nobody connects it to the fix. I've watched this cycle, fix one, break three, fix three, break two, run for months on teams that would never tolerate it in ordinary code.
The root cause is that a prompt isn't code; it's global configuration. When you change a function, the blast radius is roughly the function's callers, and your tooling can tell you who they are. When you change a prompt, every instruction in it shapes every output, and there is no static analysis to warn you. Add "always include the VAT breakdown" to fix one case, and you've changed what the model does with receipts that have no VAT, with documents where VAT is implied, and with edge cases you've never listed.
The habit of testing only the failing case is inherited from code, where locality mostly holds. With prompts, locality doesn't exist, it's the first instinct I have to talk developers out of when they move onto LLM work. The fix that looks surgical is a global edit, every time, including the changes that feel obviously safe: reordering instructions, tightening a sentence, bumping a model version. In my experience those "can't possibly affect anything else" changes are precisely the ones that bite, because nobody watches them.

The defence is blunt and effective: re-run the entire golden set on every change to the model boundary. Not the failing case, not a smoke-test subset, everything. This feels wasteful until you price it. Fifty examples at a fraction of a penny each, run in parallel, is a couple of minutes and small change in CI. One regression reaching production costs more than a year of full-set runs.
The output that matters is not the aggregate score but the per-example diff. An aggregate that moves from 94 to 92 per cent tells you almost nothing and invites rationalisation. What you want on every run is the list: which examples changed status, from what to what, in which fields. "Fixed INV-0231, broke DOC-0117, DOC-0242 and DOC-0388" is a sentence an engineer can act on before shipping, instead of an incident to reconstruct afterwards.
This is the pattern in my public repo, github.com/afareed007/docextract: golden-document evals run in CI, and a regression fails the build. The part that earns its keep isn't the pass/fail bit, it's that a failed run names the documents that changed status, so the whack-a-mole becomes visible at review time rather than in production.
Sometimes the full-set run reveals a genuine trade rather than a bug: the change fixes a common case and breaks a rare one, and no prompt wording satisfies both. That's fine, but it should be a decision, taken by a person looking at both cases, not a surprise discovered by whichever user hits the rare case next. The eval doesn't remove the judgement; it makes sure the judgement happens.
Recurring trades are also an architectural signal worth taking seriously. When you repeatedly can't make one prompt satisfy the whole golden set, the set is telling you the task has a real boundary in it, two document families, two distinct intents, and the answer is a classifier in front of two specialised prompts, not another round of wordsmithing. I've seen weeks of prompt-tuning frustration dissolve into an afternoon of routing.
None of this requires a big harness. A script that runs the set, writes one JSON result per example, and diffs against the last accepted run is an honest day's work. If you'd rather not write it, open-source tools do the same job off the shelf, promptfoo is the one I see most in the wild, with per-case assertions and a CI exit code built in. Either way, the effect is the same: prompt edits go from the most dangerous change in the codebase to one of the safest.
Building this kind of regression harness around prompts, golden sets, per-example diffs, CI gates, is the work I do. If your team is playing whack-a-mole with prompt fixes, get in touch via the contact form.
Senior engineering roles — or production AI consulting?
I'm open to senior AI and software engineering roles and to selective consulting on agents, RAG, and LLM automation in production. If this post resonates — as a hiring manager or as someone shipping similar work — get in touch.
Get in touch