All posts

"Hallucination isn't a bug you can fix, it's a property you design around"

AI models make things up, confidently, and no amount of clever engineering makes the rate zero. The honest answer is not a fix, it is design: grounding, validation and review gates that keep a wrong answer from doing serious harm.

AI basics4 min read27 August 2026by Ahmed
"Hallucination isn't a bug you can fix, it's a property you design around"

If an AI supplier ever tells you they have eliminated hallucination, walk away. Hallucination (the model stating something false with complete confidence) is not a bug that a clever engineer patches out. It is a property of how these models work, and the honest way to handle it is to design your product so that a wrong answer cannot do serious harm.

Why do models make things up? Because of what they actually do. A language model does not look facts up in a database. It generates the most plausible next words, based on patterns learned from enormous amounts of text. Plausible is the key word. Most of the time, the most plausible continuation is also true. Sometimes it is not, and the model has no internal signal telling it the difference. Think of a brilliant improviser who has read half the internet but is never allowed to check a source. Fluent always; correct usually; and the delivery is identical either way.

That last part is what makes it dangerous for business use. A wrong answer arrives in the same confident tone as a right one. There is no stammer, no "I think". Your customer cannot tell. Often your staff cannot tell either.

"Hallucination isn't a bug you can fix, it's a property you design around"

You cannot prompt your way out of this. Adding "do not make things up" to the instructions helps at the margins, and newer models hallucinate less than older ones, but the rate never reaches zero. This is not just my opinion: OpenAI's own researchers published a paper in 2025, Why Language Models Hallucinate, showing the behaviour is baked in by how models are trained and tested, the standard scoring rewards a confident guess over an honest "I don't know", so guessing is what models learn. And it is worst exactly where your business lives: the specifics. Your prices, your refund policy, your product codes, the things the model never saw in its training and therefore fills in by pattern.

So the honest answer is design, and the first tool is grounding: instead of asking the model to recall facts, you hand it the facts inside the request and ask it to work with them. "Here is our actual refund policy, answer the customer using only this." Summarising a document it can see is a task these models do very well. Recalling your policy from memory is a task they do dangerously badly. (This hand-it-the-facts approach is usually built with retrieval, software that automatically looks up the right documents for each question.)

The second tool is validation: checking the output against reality before anyone acts on it. If the model quotes an order number, your code checks that the order exists. If it quotes a price, the price gets compared against the price list. These are cheap, mechanical checks, but they catch a whole class of invented detail before it reaches a customer.

The third tool is the review gate: a human checks the output before it goes out, wherever the stakes justify it. Not everywhere, that would kill the economics, but matched to risk. Drafting an internal summary? Let it flow. Sending a quote to a customer, or anything with legal or financial weight? A person clicks approve. Many of the best AI products are exactly this shape: the model drafts, the human decides.

There is also a quieter design move: give the model permission to say "I don't know". Left to its defaults, a model would rather produce a shaky answer than none, remember, it was trained to guess. If your instructions and your product make "I can't find that" an acceptable output, and route those cases to a person, you convert invisible errors into visible handovers.

For a founder, the practical shift is in the questions you ask. Not "is it accurate?", every vendor says yes, but "what happens when it is wrong?" How would we know? Who catches it? What is the worst thing a confident wrong answer could do here? A product with good answers to those questions is trustworthy. A product with a great accuracy claim and no answer to those questions is a demo.

Building AI features that stay useful when the model is wrong, grounding, validation, review gates in the right places, is precisely the work I do. If you are adding AI to something customers rely on, get in touch via the contact form.

Senior engineering roles — or production AI consulting?

I'm open to senior AI and software engineering roles and to selective consulting on agents, RAG, and LLM automation in production. If this post resonates — as a hiring manager or as someone shipping similar work — get in touch.

Get in touch

© 2026 Ahmed Fareed. All rights reserved.

LOADING