The Cost of Better Reasoning: Why Agon’s Competitive RL Matters

Current reasoning models are hitting a wall where they prioritize verbosity over logic. While Reinforcement Learning from Verifiable Rewards (GRPO) has been the backbone of recent successes, it has a glaring blind spot: it only grades the final answer. Because the intermediate "thinking" trace isn't graded, models learn a shortcut—they write more to increase the probability of hitting the correct output. This is the "length pathology" that current benchmarks often fail to catch until they hit truly complex problems where "thinking" requires more than just a longer explanation.

Adversarial Training as a Logic Filter

Agon attempts to fix this by shifting from a single-model approach to a competitive one. Instead of a model simply trying to find the right answer in a vacuum, Agon uses two competing models to provide implicit rewards for reasoning quality. One model drafts a solution while the other reads it while simultaneously solving the problem. This creates a head-to-head game where a model's drafting is judged by its ability to withstand a critic's scrutiny. A step that leads to a "win" is reinforced, while filler that the opponent exploits is punished.

This adversarial dynamic forces the models to prune the fluff. Because the pair co-improves by facing progressively stronger rivals, it provides a training signal that single-model RL simply cannot replicate. The system is designed for efficiency, using two low-rank adapters over a single frozen base to keep memory overhead at about 2%. At inference, this manifests as a two-stage cascade: the first model drafts, and the second provides the final answer after reviewing that draft. This allows the models to reason together without the overhead of a full Mixture-of-Agents architecture.

The Production Trade-off: Quality vs. Volume

The numbers reveal a significant reality check for production teams. While Agon doubles GRPO’s pass@1 and provides roughly eight times the gain of an untrained Mixture-of-Agents setup on hard math problems, the length of the output inflates by an order of magnitude.

The real story here is that while Agon successfully punishes "filler" that an opponent can exploit, it hasn't yet solved the fundamental cost of reasoning. In a production environment, an order of magnitude increase in output length means a massive spike in inference costs and latency. We are seeing a move toward "adversarial" reasoning, which is a step in the right direction for model quality, but for now, it achieves that quality by demanding much more from the output's volume. The next frontier—reasoning in latent space—is where the real efficiency gains might finally happen, but until then, we're just watching models get much better at debating themselves into the right answer.