"Regression testing prompts: catching the fix that breaks everything else"
You fix the reported failure, ship the prompt change, and three other cases quietly break. Prompts are global configuration: every edit changes behaviour everywhere. The only defence is re-running the full golden set on every change — and reading the per-example diff.
A user reports that the system mishandles a particular supplier's invoices. An engineer reproduces it, adds a line to the prompt, confirms the failing case now passes, and ships. Two weeks later three other suppliers' invoices are being mishandled in a new way, and nobody connects it to the fix. I've watched this cycle — fix one, break three, fix three, break two — run for months on teams that would never tolerate it in ordinary code.
Have an AI feature stuck between demo and production?
The gap — reliability, evals, cost control, the plumbing that keeps it running unattended — is exactly the work I do. If that sounds familiar, a short conversation is usually enough to point you the right way.
Book a free consultation