Direct-OPD: Reusing RL Signals to Train Stronger Models

A 1.7B model went from 48.3% to 58.3% on AIME 2024 in four hours on eight A100 GPUs — and the interesting part isn't the score, it's where the improvement came from. The method behind it, Direct On-Policy Distillation (Direct-OPD), doesn't run reinforcement learning on the target model at all. It borrows the RL signal from a smaller model and treats it as a dense reward for a stronger one.

How Direct-OPD Actually Works

The core move is a shift in what counts as reusable. Standard distillation takes a post-RL teacher and asks the student to imitate its outputs — which caps the student at the teacher's ceiling. If the teacher is weaker than the student, you're actively dragging the stronger model down.

Direct-OPD sidesteps that by comparing the post-RL teacher with its own pre-RL reference and using the log-ratio between those two policies as an implicit reward signal. The student isn't copying the teacher's answers; it's absorbing the policy shift that RL induced. That reframing is what lets the researchers describe weaker teachers consistently improving stronger target models — the student inherits the direction of improvement, not the teacher's final capability level.

There's a second consequence worth noting: because the signal is a shift rather than a snapshot, you can compose multiple policy shifts sequentially across model scales. Stack improvements from several small-model RL runs instead of treating each as a finished artifact.

What The Numbers Actually Show

The headline stat — Qwen3-1.7B jumping ten points on AIME 2024 in four hours on 8 A100 GPUs — is step-matched against direct RL, meaning the comparison isn't against an unlimited RL budget. That matters. The claim isn't "distillation beats RL forever"; it's "at equal compute on the student, reusing an existing RL signal beats running sparse-reward RL yourself."

That's the version of the claim that survives contact with real budgets. Sparse-reward RL on a math benchmark is notoriously expensive and unstable — most of the cost is in the exploration, not the final gradient steps. If someone has already paid that cost at a smaller scale, reusing it as a dense signal is exactly the kind of shortcut that holds up. The caveats I'd want tested before trusting it in my own pipeline: the method presumes a pre-RL reference checkpoint exists for the teacher, and it presumes the RL shift at a smaller scale transfers meaningfully at a larger one. Neither is guaranteed for your domain — a policy shift learned on competition math may not compose the same way for, say, code review or tool-use traces.

Why Reusable RL Signals Change The Economics

What this actually points to is a market structure change, not just a training trick. If RL outcomes become transferable implicit rewards rather than final models to imitate, then every well-funded RL run at a small model scale becomes an asset that stronger models can inherit. The researchers put it directly: RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

My read — and this is interpretation, not what the paper claims — is that the underappreciated part is the dense-vs-sparse framing. The hard part of RL on reasoning tasks has always been reward sparsity, and this converts someone else's sparse-reward struggle into a dense signal you can train against cheaply. The gap between the clean AIME setup and messy production domains is where this will get annoying: log-ratio rewards assume comparable policy distributions, and real fine-tuning divergences may be messier than a benchmark run. But as a pattern — pay for RL once at small scale, amortize it upward — this is the kind of technique worth prototyping the moment reference checkpoints are available for the models you actually use.