RoboDojo Tests Robot Manipulation Policies Where It Counts

Robot manipulation policies have had no shortage of leaderboards, but almost none of them test what actually matters: whether a policy survives contact with a physical robot. The RoboDojo paper changes that with a benchmark that spans both simulation and the real world - 42 simulation tasks and 18 real-world tasks - and evaluates generalist manipulation policies across five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.

Why Sim-Only Benchmarks Miss the Point

The paper states the problem plainly: simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. That tension is exactly why most published results in this space are hard to trust - a policy that looks strong in sim often says nothing about whether it can handle friction, latency, or an uncalibrated gripper.

RoboDojo's answer is RoboDojo-RealEval, a real-world evaluation system with standardized hardware and remote cloud access designed to produce reproducible conditions. Reproducibility is the part worth paying attention to here. Anyone who has tried to compare two robotics papers knows the usual problem: different rigs, different objects, different lighting, no way to tell whether a claimed improvement is real or just a luckier setup. Standardized hardware plus remote access doesn't guarantee fairness, but it removes the biggest source of noise.

What 30 Policies on One Leaderboard Actually Shows

The benchmark integrates with XPolicyLab so policies can be evaluated across both simulation and real-world settings with minimal adaptation. Thirty policies have already been integrated and evaluated, establishing a public leaderboard.

That's the part I'd watch rather than the task count. A leaderboard only matters if enough people submit to it, and 30 policies on day one is a real signal of buy-in. It also means cross-policy comparison finally happens under identical conditions - which is where generalist manipulation claims tend to fall apart. A policy that wins on generalization may quietly lose on precision or long-horizon execution; until now there was no shared place to see that tradeoff side by side.

The five evaluation dimensions are well chosen for exactly this reason. Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remained limited in systematically evaluating their capabilities - so papers could cherry-pick whichever metric flattered them. A benchmark that forces generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following into one scorecard makes cherry-picking harder.

The Gap That Still Isn't Closed

What this actually points to is a discipline shift: robotics evaluation moving toward the kind of shared infrastructure that made language model leaderboards meaningful. But there's a limit worth being honest about. Eighteen real-world tasks is small next to 42 in simulation, and no standardized rig eliminates every difference between labs - object wear alone will drift over time. The benchmark makes comparisons fairer; it doesn't make them perfect.

In practice this means you should treat RoboDojo rankings as stronger evidence than single-paper numbers, not as ground truth about which policy you'd deploy tomorrow. The right read is directional: if a policy holds up across sim tasks, real tasks on standardized hardware, and all five dimensions at once, that's more meaningful than any isolated demo video. The field has needed exactly this kind of unified sim-and-real benchmark for years - the question now is whether labs actually route their next results through it.