RWGBench: Why Citation Decisions Beat Text Similarity

Most benchmarks that score LLMs on writing related work sections are measuring the wrong thing. They inherit metrics from summarization — lexical overlap, semantic similarity to a human-written reference section — and a model can ace them while citing papers that don't fit, omitting central prior work, or dumping citations into contexts that misrepresent what those papers actually say. RWGBench, described in a recent arXiv paper, is built around a different premise: related work generation is a citation decision-making task, not a text-similarity task, and evaluation should target the decisions.

Why Text Similarity Rewards The Wrong Output

The core problem the researchers identify is that conventional metrics reward outputs that resemble the reference text while failing to detect academically critical errors in citation behavior. A generated section can be fluent and semantically close to the real thing while over-relying on a narrow subset of citations, skipping foundational work entirely, or placing references where they distort their relevance. None of those failures show up in an n-gram or embedding score.

RWGBench reframes the task explicitly. Related work generation is anchored to a specific target paper — its title, abstract, and optionally introduction — and requires retrieving a relevant subset of papers from a large corpus, then organizing them into a narrative that positions the target paper against prior research. That's rhetorically different from survey generation, which aims for broad topic-level coverage; RWG has one job: situate this paper. The benchmark decomposes that job into three interdependent dimensions: citation selection (which works to cite given the paper's context and contributions), strategic placement (where citations sit in the discourse structure), and rhetorical framing (whether cited work is framed as building on or contrasting with the target).

What's Actually In The Benchmark

The construction is concrete rather than aspirational. It draws on 40,108 computer science papers pulled from arXiv between 2020 and 2025, with metadata, citation lists, and related work sections successfully extracted. The retrieval corpus contains 1.09 million candidate documents — built by aligning extracted citations with entries in the SurGE corpus and querying the arXiv API for gaps. The test set is small but deliberately high-fidelity: 100 papers accepted at ICLR, NeurIPS, or ICML whose author-written related work sections have all their citations covered by the retrieval corpus.

That last detail matters more than it looks. A benchmark where the ground-truth citations can't be found in the candidate pool would penalize models for retrieval failures that aren't their fault. By restricting to papers whose full reference sets are retrievable, RWGBench separates retrieval bottlenecks from generation-level positioning failures — which is exactly what experiments across representative and frontier generation settings reportedly expose as systematic limitations hidden by standard evaluations.

A blinded human study provides supplementary support for the diagnostic metrics. That's thin but appropriate; it's corroboration of construct validity rather than headline evidence.

What This Actually Points To

My read: this benchmark matters less as a leaderboard than as a diagnostic instrument — and its real value depends on adoption outside its own test set.

The 100-paper test set is peer-reviewed and carefully constructed, but it's small and confined to top-venue machine learning papers written between 2020 and 2025. A model could score well here and still fail on dense subfields where positioning conventions differ, or on older literatures where key prior work predates the corpus window entirely (the corpus covers 2020–2025). If you're evaluating an RWG system for your own domain, treat RWGBench as proof that citation-decision metrics are feasible — not as evidence any particular model handles your field's citation norms.

There's also an unaddressed gap worth naming: everything here assumes automated evaluation of citation decisions can be trusted at scale. The human study supports it directionally, but anyone who has shipped LLM-assisted writing tools knows that judges drift too — especially on judgment calls like whether a citation was framed fairly versus merely placed nearby. I'd want to see how stable these diagnostic scores are across evaluator models before treating single-point differences as signal.

What this actually points to is a shift in how we should audit writing assistants generally: stop scoring fluency against reference text and start scoring whether the substantive decisions underneath are defensible. That principle extends well beyond related work sections — to literature reviews generated by agents (a theme close to earlier coverage of AI-driven scientific discovery) or any pipeline where an LLM chooses sources rather than just summarizing them. The interesting failure mode isn't bad prose; it's confidently wrong scholarship that looks fine at first read.

For teams building RWG features now: run your system through both kinds of evaluation — similarity metrics for polish, citation-decision metrics for correctness — because they'll disagree more often than you'd expect.