"The automation that \\"worked\\" for three weeks: a silent failure story"

For three weeks, an LLM automation I run processed incoming email, extracted the wrong information from a chunk of it, and reported success on every single run. No errors. No failed executions. No alerts. The dashboards were green the entire time, and the only reason I found out at all was a ten-minute habit I very nearly skipped that week.

Some context. The automation reads incoming email for my own business, uses a model to extract structured fields, what the message is about, who it concerns, any dates or amounts, what action it needs, and routes each item to the right place. It has the guards you would expect: the model's output is validated against a schema before anything downstream touches it, failures retry with backoff, and anything that exhausts its retries lands in a dead-letter queue where I can see it. By the standards of most LLM systems I have looked at, it is well defended.

Then one of the upstream services that sends those emails redesigned its notification template. The details that used to sit in a plain paragraph moved into an HTML table, a summary line was reworded, and some boilerplate grew a date of its own. A human reading the new format would not have blinked. The model did not blink either, and that was exactly the problem.

It kept extracting. Asked for a date, it found one, in the boilerplate footer rather than the body. Asked for a category, it picked something defensible from the reworded summary, which happened to be wrong more often than not. Every one of those values was the right type, the right shape, the right length. The schema validation passed, because schema validation checks that an answer is well-formed, not that it is true. There is no Zod refinement for "this date belongs to the wrong sentence".

For three weeks the misrouted items flowed into a bucket that gets attention weekly rather than daily, so nothing downstream screamed. Each individual oddity looked like a one-off, a slightly strange categorisation, a deadline that seemed early. Wrong answers that arrive one at a time, each individually plausible, do not trigger anyone's suspicion. They just quietly accumulate.

What caught it was the weekly random sample: ten outputs pulled at random, checked against their source emails by eye. The third item in that week's sample had an extracted date that appeared nowhere in the actual content of the email, it had been lifted from the footer. I pulled every recent message from that sender and found the same defect in all of them, going back exactly to the day the template changed. Three weeks of confident, well-formed, wrong output, dated to the hour by a diff of two raw emails.

The immediate fix was easy: adjust the extraction prompt for the new layout and reprocess the affected items. The structural fix was the interesting part. First, I now fingerprint upstream formats, a rough structural hash of each sender's template, and alert when a sender's fingerprint changes, because that is the moment an extraction should be re-verified. Second, and more importantly, I added alerts on the business metrics themselves: the daily distribution of categories, the rate of items marked unknown, the typical range of extracted dates relative to the email's received time. When any of those drifts from its trailing baseline, I get told. That alert fires on this entire class of failure, whatever causes it next time.

It is worth being precise about why nothing else could have caught this. Exception monitoring watches whether the code ran; this code ran perfectly. Schema validation watches whether the output is well-formed; it was. A component that fails by being plausibly wrong, rather than loudly broken, is invisible to every check that looks at the mechanism instead of the answers. The only monitoring that works is monitoring that measures whether the answers, in aggregate, still make sense.

The transferable lesson is blunt: whatever feeds your automation will change without telling you, and your model will absorb the change without complaint. Assume both. Alert on the business numbers, distributions, rates, ranges, not just the error count, and keep a human sampling loop even when everything has been green for months. Especially when everything has been green for months.

If you are running an LLM system in production and you cannot currently say how you would know it had gone quietly wrong, that is a solvable problem, and it is exactly the work I do. Get in touch via the contact form.