RAG for Public Health QA: What Retrieval Actually Fixes

A study of retrieval-augmented generation on UK public health guidance claims smaller models can match larger ones — but that claim only holds when retrieval quality is already high, which is exactly the condition most real deployments can't guarantee.

The research, posted on arXiv, evaluates RAG systems built on official UK Government public health guidance, using 7,929 questions derived from that corpus. The core findings are straightforward: hybrid retrieval consistently improves recall and ranking quality across multiple embedding models and corpus variants, and providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs. With retrieval quality high enough, smaller open-weight models match or outperform larger models running without retrieval.

What the Evaluation Actually Covers

The methodology is more careful than most RAG write-ups. The researchers used rubric-based LLM-as-a-judge to assess faithfulness, completeness, clarity, and factual consistency, validated against dual human annotations. That's a real check, not a vibe score.

But the judge-human agreement results deserve more attention than they've gotten: agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced. In other words, the automated judge is most trustworthy on exactly the dimensions least likely to catch a subtle factual error in health guidance — and least trustworthy on the dimension where an error would matter most. For a domain like public health, that's not a footnote; it's the constraint the whole result sits inside.

The Claim Worth Making — And the One That Isn't

The defensible claim here is narrower than "RAG solves hallucinations for public health QA." What's actually shown is that hybrid retrieval improves recall and ranking consistently across model and corpus variants — that's an architecture-level result about retrieval itself, and it holds up well under scrutiny. The stronger inference — that small models plus good retrieval equals large models — depends entirely on retrieval quality being high. Nobody deploying this in production knows in advance whether their corpus will support that quality; the study tells you what happens when it does, not what happens when it doesn't.

What this actually points to is a shift in where the engineering burden sits: from model selection to corpus construction and retrieval tuning. If smaller open-weight models can compete under high-quality retrieval, then the differentiator isn't the LLM at all — it's how well your guidance documents are structured, indexed, and refreshed as policy changes. That reframing matters for anyone budgeting an AI project in a regulated domain: money spent on retrieval infrastructure likely buys more reliability than money spent on a bigger model.

It also connects to a broader pattern I keep seeing: fixes that don't eliminate a risk so much as relocate it. RAG reduces hallucination risk by grounding answers in retrieved documents — but it moves failure modes into retrieval quality and judge reliability instead. The study is honest enough to show that seam; most coverage of work like this isn't.