Anatomy of a production-grade extraction agent

Pulling structured data out of messy text is one of the most common things businesses want from an LLM: names and emails out of enquiry forms, amounts and dates out of invoices, fields out of PDFs. It is also one of the easiest things to demo. Paste the text in, ask the model for JSON, and it works — first try, in front of everyone.

Then it meets real traffic. The model returns an email address without an @ in it, confidently. It returns JSON that doesn't parse, occasionally. A prompt tweak that fixes one awkward document silently breaks ten others. And because the output looks plausible, nobody notices until wrong data is sitting deep in the CRM and someone asks why a customer was invoiced twice.

I built docextract as a small, open reference for the layer that prevents this — the production layer most extraction demos skip. It's a LangGraph agent behind a FastAPI service, MIT-licensed, on my GitHub at github.com/afareed007/docextract. It's deliberately small enough to read in an afternoon, but it demonstrates every guard I put around LLM output in real systems. Four of them do most of the work.

Guard one: validate, don't trust. Every field the model returns passes through a typed schema (Pydantic) with real validation — a malformed email or a phone number full of letters is caught at the boundary, not discovered three systems downstream. LLM output is input, and input gets validated. This single habit removes a whole class of silent data corruption.

Guard two: the bounded re-ask. When validation fails, the agent doesn't give up and doesn't guess — it shows the model what was wrong and asks again. But 'again' has a hard cap. At the cap, the loop fails closed instead of retrying forever, which means a weird document costs you a bounded number of tokens, not an unbounded bill.

Guard three: a human-review gate. The agent scores per-field confidence, and any record below the threshold is routed to needs_review instead of being accepted. A system that knows when not to act is the cheapest reliability feature you can build — a human reviewing five records a day is a far better outcome than a model quietly wrong fifty times a day.

Guard four: evals in CI. The repo carries a set of golden documents with known correct answers. Every push re-runs extraction against them and scores field-level precision and recall — and the build fails if accuracy drops below the threshold. That turns prompt changes into normal, boring engineering: you can refactor a prompt on Friday afternoon because the safety net is the same one you have for code.

One more piece worth stealing even if you ignore the rest: the provider seam. The agent runs against a deterministic mock provider by default, so the whole test suite runs offline, fast and free — and the real model is swapped in with an environment variable. If your AI feature can only be tested against a live API with a paid key, it will be under-tested. Guaranteed.

None of this is glamorous, and that's the point. The pattern generalises far beyond extraction — validate the output, bound the loops, gate low confidence to a human, eval every change. That's the difference between an AI feature you demo and an AI system you can leave running unattended.

If you've got an extraction pipeline — or any LLM feature — that works in the demo but you wouldn't trust on its own at 3am, that gap is exactly the work I do. Get in touch via the contact form below.