All posts

"Running LLM automation unattended for months: what actually breaks"

An LLM automation that works in week one will find new ways to fail by month three. Here is what actually broke when I ran one unattended for months, from provider outages that arrive in bursts to a model that changed personality underneath me, and the small set of guards that earned their keep.

AI engineering5 min read8 July 2026by Ahmed
"Running LLM automation unattended for months: what actually breaks"

If you have an LLM feature in production, you have probably lived through the week where it quietly degraded and nobody noticed until a customer did. The demo was fine. The first month was fine. Then something shifted, the provider, the model, the inputs, and the thing that "just worked" started producing garbage with complete confidence. This post is a field report from the other side of that week.

I run an LLM automation, built on n8n, that processes incoming email for one of my own business operations. It has run for months with nobody babysitting it, which means I have a reasonable catalogue of what actually breaks over that timescale, as opposed to what people worry about in architecture reviews. The two lists overlap less than you would hope.

The first thing that breaks is the provider, and it does not break politely. Outages and rate limits do not arrive as a gentle background error rate; they arrive in bursts. Twenty minutes of every request failing, then normal service, then another burst at 2am. If your automation retries once and gives up, a burst becomes a pile of silently dropped work. If it retries forever, a burst becomes a thundering herd that makes the rate limiting worse. Neither default is acceptable, and neither is what most workflow tools give you out of the box.

"Running LLM automation unattended for months: what actually breaks"

The second thing that breaks is the model itself, not with an error, but with a personality change. Providers update models, and behaviour shifts underneath you. This is measured, not folklore: when Stanford and Berkeley researchers compared the March and June 2023 versions of GPT-4, its accuracy at identifying prime numbers fell from 97.6% to 2.4%, with no announcement and no error raised (arxiv.org/abs/2307.09009). That drop is extreme; the mechanism is routine. A prompt that reliably produced terse structured output starts adding preamble. Nothing in your code changed, your logs look healthy, and the system is now doing a slightly different job than the one you tested. Pinning a dated model snapshot rather than a floating alias buys you control, but snapshots get deprecated eventually, so pinning postpones the migration, it does not cancel it, and you still need a small evaluation set ready for the day you are forced to move.

Then there is volume. At ten requests a day you will never see the 1-in-200 bad output. At two thousand a week you see it ten times, and one of those ten lands somewhere that matters. The major providers now offer structured output modes that constrain generation to your JSON schema, which has largely killed the malformed-JSON class of failure, but what they guarantee is the shape, not the content. A syntactically perfect response with the wrong amount in the right field, or a confident summary of the thing it was told to ignore, still validates. Those failures are boringly inevitable once the numbers get large, and no amount of prompt wording eliminates them.

The failure mode that cost me most, though, was the quiet one: the queue backing up. A downstream step slowed, items accumulated, and the automation kept "working", every individual execution succeeded, while end-to-end latency crept from minutes to hours. No exceptions, no red dashboards. The system was healthy by every technical measure and useless by the only measure that counts, which is whether the work got done on time.

That points at the general lesson: the worst failures do not throw. An automation that crashes is annoying but honest. An automation that succeeds with the wrong output, extracts the wrong amount, routes the email to the wrong place, summarises the wrong thing, is the one that erodes trust, because you find out weeks later and you cannot tell how long it has been happening.

So, the guards that earned their keep. Retries with exponential backoff and jitter, the jitter matters, because a fleet of retries without it re-synchronises into a fresh burst, plus a dead-letter path, so anything that exhausts its retries lands somewhere visible instead of vanishing. Schema validation at every boundary: I check the model's output with Zod in TypeScript (Pydantic if you are in Python) before anything downstream is allowed to touch it, and a validation failure triggers a bounded re-ask, never an infinite loop. And alerting on the business metric rather than the exception count: I alert on emails processed per hour, because that number catches the backed-up queue and the silent drift that no stack trace ever will.

The last guard is the least glamorous: a weekly eyeball of a random sample of real outputs. Ten minutes, looking for the drift no automated check has caught yet. Every time I have skipped it for a few weeks, it has found something when I came back. It is also the first habit I push on developers I mentor into production AI work, because it encodes the real rule: unattended does not mean unmonitored. It means the monitoring is designed so a human touches the system on a schedule, not in a panic.

None of this is clever. It is retries, validation, dead-letter queues, business-level alerts and a human sampling loop, the same discipline that keeps any production system alive, applied to a component that fails in stranger ways than most. If you have an AI feature that works most of the time and you need it to work all of the time, this is exactly the work I do. Get in touch via the contact form.

Senior engineering roles — or production AI consulting?

I'm open to senior AI and software engineering roles and to selective consulting on agents, RAG, and LLM automation in production. If this post resonates — as a hiring manager or as someone shipping similar work — get in touch.

Get in touch

© 2026 Ahmed Fareed. All rights reserved.

LOADING