Engineering Truth in High-Stakes Document AI

LLMs are fundamentally designed to be helpful. In the world of AI engineering, that's a massive liability when your goal is accuracy over creativity. When you're dealing with public tenders, a 'plausible' completion that isn't backed by hard evidence isn't just a hallucination—it’s a legal landmine. If you want to build a system that produces compliant first drafts, you have to move past basic RAG and build a multi-layered validation architecture where the AI is explicitly taught when to shut up.

Validating Coverage Across Massive Contexts

Moving from a clean demo to a 99-page NHS pack exposes the cracks in standard retrieval. The primary challenge isn't just finding information; it's ensuring the model actually 'saw' everything. The Lucius system tackles this by measuring coverage against page spans and employing chunk-level result validation.

In their postmortem, they tracked 42 requirement rows and saw a massive reduction from 366 extracted requirements down to 184. This is a critical insight for builders: hallucination in these documents often starts with over-extraction. The model tries to be 'helpful' by finding requirements where none exist. To solve this in production, you can't just rely on a single pass. You need a dedicated second pass specifically for missed items to catch what the first pass overlooked, ensuring the scope is locked down before a single word of the draft is written.

Turning 'Refusal' Into a Core Product Feature

Here is where the engineering gets interesting: treating 'refusal' as a product feature rather than an error state. Instead of forcing an LLM to bridge gaps with creative writing, the system uses a capability-fit check. If a bidder can't meet a specific requirement, the system routes that unevidenced requirement to a visible partner slot and surfaces the count in a prominent banner.

This is an architectural choice to prioritize truth over completion. When you're dealing with £950,000 in potential contracts and £5,000,000 in Professional Indemnity Insurance, the goal isn't just to generate a draft in five minutes—it's to generate a compliant draft in five minutes. The system even declines to draft entirely if the initial analysis suggests the tender is a non-starter. You aren't just building a writer; you're building a gatekeeper.

The Plumbing of Truth-Measuring Instruments

The real story here isn't the LLM's ability to read a document; it's the engineering of the 'truth-measuring instrument.' If your verifier is wrong, the entire system is compromised. This is where most AI projects hit a wall.

If your verifier produces too many false alarms, your team will eventually get frustrated and start overriding them manually—effectively disabling your safety net. If you have too many false passes, your customer eventually realizes they're trusting fiction. To build grounded Document AI, you have to treat verifiers as measured components with their own evaluation sets. You can't just hope the model is 'honest.' You have to build the plumbing that measures honesty as a measurable, auditable component of the pipeline.