"Fail closed, not silent: what an AI system should do when it isn't sure it worked"

A quiet wrong answer is the most expensive thing an AI system can produce. A loud failure gets noticed: someone retries, files a ticket, fixes the cause. A quiet wrong answer sails downstream wearing the same formatting as a correct one, into an invoice, a CRM record, a customer reply, and by the time anyone notices, it has been acted on. The cost is not the error. The cost is everything built on top of it.

Traditional software fails loudly by default. Dereference a missing value and you get an exception; call a missing endpoint and you get a 404; the failure interrupts execution and demands attention. LLMs invert this. A model will return something for almost any input, fluent, well-structured, delivered with exactly the same confidence whether it is right or fabricated. Uncertainty does not throw. There is no stack trace for "I guessed". Your system gets a 200 OK with wrong content, which is the one failure mode the rest of your stack was never designed to catch.

"Fail closed" is a phrase borrowed from safety engineering. A fail-closed valve shuts when it loses power; a fail-open one keeps flowing. Translated to AI systems: when the system cannot establish that it worked, it should withhold the result and say so, not ship its best guess and hope. The default posture matters here more than anywhere else in software, because with an LLM the guess is always available and always looks presentable.

Which raises the practical question: how does the system know it is not sure? Confidence has to be constructed from signals, because the model's tone carries no information. Output that fails schema validation. A second pass over the same input that comes back different. Extracted line items that do not sum to the stated total. A date in the wrong century. Input that looks nothing like anything the prompt was designed for. A model-reported confidence below threshold, where you can get one. No single signal is reliable; a handful of cheap ones together are usefully hard to fool.

The design consequence is that success cannot be a boolean. Every AI step in a pipeline should return one of at least three outcomes, succeeded, failed, needs review, and every consumer of that step must handle all three. The needs-review state is the load-bearing one. It is the honest middle where the system says "I produced something, and I do not trust it enough to act on it". Leave that state out of the design and every uncertain case is forced to masquerade as success or failure. It will almost always choose success.

An LLM automation I run for my own operations classifies incoming documents before anything processes them. When classification confidence is low, the visible behaviour is that nothing happens: no record created, no downstream action triggered. What actually happens is that the item lands in a review queue with the input, the attempted classification and the reason for doubt attached. That "do nothing visible, escalate quietly" behaviour is a designed state, not an accident, and building it took about as long as building the classifier.

One caution: fail closed must not mean fail silent in the other direction. A system that quietly withholds results is accumulating a hidden backlog of undone work, which is its own slow-motion failure. Failing closed properly means the failure is visible somewhere someone actually looks, a queue with a named owner, a metric on a dashboard, an alert when the rate moves. The rate itself is diagnostic. A sudden rise means something upstream changed. A fall to zero almost never means the model became perfect; it usually means your uncertainty checks broke, and the quiet wrong answers are flowing again.

There is a real trade-off here and it deserves honesty. Every case that fails closed costs throughput: a human has to look at it, or the task simply is not done. Set the thresholds too conservatively and the review queue swamps the team, the reviewers start rubber-stamping, and you have rebuilt the manual process with extra steps. The threshold is not a philosophical position, it is a number you tune against measured error rates and the actual cost of a wrong answer in your domain. A slightly awkward marketing draft is not an invoice for the wrong amount.

What makes the investment worth it is trust, which is asymmetric. Users forgive "I couldn't process this one, a person will look at it" almost indefinitely. They do not forgive discovering that the confident answer they acted on last month was invented. And once an organisation stops trusting a system's outputs, it quietly starts re-checking everything by hand, at which point the system's value has dropped to zero without anyone ever declaring it dead.

Building AI systems that know when they are not sure, and do something defensible about it, is the work I do. If your system currently delivers every answer with the same confidence, get in touch via the contact form.