RL Doesn't Just Fine-Tune; It Architectures Strategy

Reinforcement learning (RL) post-training is often sold as a way to polish model outputs. However, new research into rewrite-grammar environments suggests the reality is much more structural. The study shows that RL isn't just reweighting existing behaviors; it is actively reorganizing primitive skills into higher-level, phased compositional strategies. While pretraining provides the raw materials, RL acts as the builder that assembles those materials into reliable procedures.

Selection vs. Synthesis: The RFT Trap

One of the most critical distinctions in this research is the difference between RL and rejection fine-tuning (RFT). In many production pipelines, RFT is the go-to for filtering high-quality outputs. But there's a fundamental flaw: RFT often produces "spurious shortcut-like rewrites." It selects for what looks correct on the surface without capturing the underlying logic.

RL operates on a different principle of selectivity. It concentrates on valid, reusable structures—specifically sequential and parallel compositions. Instead of just picking the best-looking answer from a pool of samples, RL identifies and consolidates procedures that the model can actually deploy across different problems. This is why RL solves held-out problems that even massive sampling budgets in pretraining fail to crack: it builds procedural competence rather than just memorizing high-probability paths.

The Pretraining Gate and the Substrate

The research reveals a phased procedural chunking mechanism during RL training. The model first strengthens its primitive reductions before it begins discovering and consolidating valid compositions. This confirms that pretraining serves as a "procedural substrate." It provides the basic ingredients, but those ingredients are often too weak to be useful in isolation.

For practitioners, this identifies a hard ceiling: the emergence of complex strategy is gated by how pretraining organizes primitive competence. If the base model doesn't have the right primitive building blocks, RL can't effectively assemble them into higher-level strategies. The model isn't learning a new trick; it's learning how to chain existing tricks into a reliable workflow.

The Production Reality

We need to stop viewing RL as a "vibe check" for model behavior and start seeing it as a structural organizer. If RL is reorganizing primitive competence into reusable strategies, then the quality of our RL results is directly capped by the diversity and reliability of the primitives in the base model.

In practice, if you're seeing a model fail on complex, multi-step reasoning despite heavy RL, the issue likely isn't your RL algorithm—it's that the pretraining substrate is missing the necessary primitive ingredients to be composed. Success depends on how well we can curate those underlying primitive skills before the RL phase even begins.