Solving the Verbosity Problem in Reasoning Models with Agon

Reinforcement learning from verifiable rewards, such as GRPO, has a glaring flaw: models often learn to pad their reasoning chains to increase the probability of hitting a correct answer without actually improving their underlying logic. This "length pathology" means that while a model might eventually arrive at the right conclusion, the path it takes is often bloated with filler. The Agon framework addresses this by shifting the training objective from a simple correctness check to a competitive, head-to-head game.

How Competitive RL Targets Reasoning Quality

Agon introduces a method where two models compete to solve the same task. Instead of a static reward for a correct answer, the system rewards a model for out-solving a rival that has already seen its work. This setup implicitly grades the reasoning trace; if a model includes filler or weak logic that an opponent can exploit, it is punished. If a specific step leads to a win, that logic is reinforced.

To keep the computational overhead manageable, the researchers optimize the pair of models as two low-rank adapters over a single frozen base, maintaining a memory overhead of only about 2%. During inference, the setup transitions into a two-stage cascade: one model drafts the reasoning, and the second model provides the final answer after reviewing that draft. This works best when the two models are comparably strong but behaviorally different, ensuring that a second model with different failure modes can effectively break the closed loop of a single model's errors.

The Tradeoff Between Accuracy and Verbosity

The results of the Agon method show a significant improvement over standard GRPO, doubling the pass@1 metric on hard math and competitive programming tasks. When compared to an untrained Mixture-of-Agents approach, Agon provides roughly eight times the gain over the same base model. These improvements were tested across multiple families, including Qwen3.5 and Gemma 4.

However, the data reveals a significant practical hurdle. While the accuracy rises, the length of the generated reasoning inflates by an order of magnitude. This suggests that while Agon is effective at forcing models to compete for the "win," it doesn't necessarily streamline the reasoning process into a concise format. It successfully identifies the winning path, but the model still tends to over-communicate to get there.

The Hidden Inference Tax of Competitive Reasoning

The real story here isn't just that competition improves accuracy; it's that we are still struggling to decouple "looking like reasoning" from "actually reasoning." The fact that length inflates by 10x while accuracy only sees modest gains points to a significant inference tax. In a production environment, where token costs and latency are primary constraints, an order of magnitude increase in output length is a massive overhead.

Agon is a clever technical workaround for the limitations of verifiable rewards, but it highlights a deeper problem in the current RL paradigm. We are rewarding the result, and because the chain of thought is never directly graded, the models are finding ways to pad their logic to satisfy the reward signal. For practitioners, the takeaway is that while competitive RL can push the boundaries of what a model can solve, it hasn't yet cracked the code on making that reasoning efficient. We've moved the needle on correctness, but the cost of that progress remains high in terms of raw token volume.