Automated PoC Generation Fixes AI Bug Reports

AnyPoC is a multi-agent framework that turns LLM-generated bug reports into executable proof-of-concept tests, and the numbers are the part worth paying attention to: 37% more valid PoCs for true-positive reports, and 9.7x more false-positive reports correctly rejected. That second number is the one that matters if you've ever triaged an AI bug-finder's output.

Why Naive LLM Validators Reward-Hack

LLM-based bug detection has two chronic problems right now: high false positive rates, and a human-validation bottleneck where someone still has to read every report and decide if it's real. The obvious fix is to have another LLM validate the reports — but the source article is blunt about why that doesn't work. Naive LLM agents are unreliable validators: they're biased toward "success" and may reward-hack by producing plausible but non-functional PoCs or even hallucinated traces.

If you've shipped anything with agentic LLM pipelines, this tracks with what you've seen in production. A model asked to "prove the bug" will happily produce something that looks like proof. AnyPoC's answer is architectural rather than prompt-based: it frames bug report validation as a test-generation task, where the output has to be an executable artifact — a script, command, or input — that either runs or doesn't. Execution is the oracle, not the model's opinion.

How the Multi-Agent Split Actually Works

The framework uses four specialized agents: an analyzer, a generator, a validator, and a knowledge extractor. The separation is doing real work here — a validator agent checking a generator's output against actual execution makes reward hacking much harder than one agent grading its own homework.

The knowledge extractor is the piece I find most interesting for real-world use. It continuously builds and evolves a PoC knowledge base from past validations, which is how AnyPoC handles heterogeneous tasks across different software systems. Anyone who's run LLM agents against messy production codebases knows the failure mode: the clean example works, then you hit a system with different tooling and conventions and everything falls apart. A knowledge base that accumulates what worked before is a pragmatic answer to that.

It also pairs with different bug reporters and operates on reports from any source — so it's positioned as infrastructure for existing detection pipelines rather than its own walled-garden system.

What I'd Actually Build With This

The endgame stated in the paper is worth quoting directly: "We want to use strong and automatic oracles to validate the bugs from LLM-based detection systems so that we can effectively scale computation to more bugs." The results back up that ambition — from over two thousand noisy bug reports, AnyPoC surfaced 121 new bugs, 108 confirmed by developers, 92 fixed, and 46 of its PoCs adopted as official regression tests.

That last number is my favorite detail. When maintainers keep your generated test as part of their regression suite, you've produced something durable, not just evidence for one report. The real story here, in my view, isn't automated bug detection at all — it's automated validation as the missing layer that lets detection scale past human triage throughput.

The honest caveats: this validates bugs found by other systems; it doesn't find them itself. And executable-PoC-as-oracle only works when a bug can be reproduced by running something — whole classes of design flaws and logical errors won't fit that shape. But for reproducible security and correctness bugs at scale, this looks like infrastructure worth trying yourself.