ARC-AGI-1 Without Fine-Tuning: What Orchestration Bought

An open-weight model running in non-thinking mode under a strict budget just hit 67.25% pass@2 on the ARC-AGI-1 evaluation set at $0.62 per task — no benchmark-specific training involved. That's the claim from a recent arXiv paper, and the interesting part isn't the headline number. It's how the system gets there: by decomposing the problem rather than hoping a single model call figures it out.

What The Numbers Actually Show

The paper introduces two architectures. The first is an Explorer-Definer Pipeline, which separates pattern discovery from executable transformation synthesis — instead of asking one model to both understand the puzzle and write working code for it, those stages are split. That pipeline alone reaches 57.50% pass@2 at $0.25 per task.

The second, a Reflective Orchestrator, adds autonomous re-exploration when hypotheses fail, pushing to 67.25% pass@2 at $0.62 per task. Both lift substantially over the 15.50% one-shot baseline — roughly 52 points of improvement from orchestration alone, with no fine-tuning on ARC-specific data.

The cost figures deserve attention too. At $0.25–$0.62 per task, this is cheap in absolute terms, but it's a multiplier on baseline inference cost: you're paying for multiple candidate generations plus orchestration overhead to buy reliability. That's a real tradeoff, not a free win.

The Part Worth Being Skeptical Of

The paper's own analysis is unusually honest about where the ceiling is: unbiased pass@k analysis suggests the pipeline is generation-bound, not selection-bound — meaning the system already captures roughly 95% of its candidate ceiling via selection using training-pair accuracy. In other words, better picking won't help much; better generating will.

That framing also tells you what to watch when these numbers get cited elsewhere. Pass@2 means you get two attempts — production systems rarely get that luxury on a fresh task, and selection accuracy measured on training pairs doesn't guarantee clean selection on unseen ones. The +9.81 pp unbiased pass@1 lift from adaptive re-exploration is the more production-relevant figure here.

One ablation stands out as genuinely useful signal: removing the think tool drops pass@2 by 5.75 pp. A single component accounting for that much movement is either a robust mechanism or a fragile dependency — and the paper can't fully distinguish which from these results alone.

What This Actually Points To

The real story here isn't that an open-weight model got good at ARC-AGI-1. It's that structured decomposition — splitting pattern discovery from program synthesis, then reflecting on failures — extracted ~52 points from a model that scores 15.50% raw. That's an argument about where capability lives right now: in orchestration scaffolding around models, not just in the models themselves.

My read as someone who has shipped pipelines like this: the approach holds up conceptually because it matches how abstraction problems actually behave — you fail cheaply and often before you succeed once, so architecture that makes failure informative beats raw model quality at this margin. But generation-bound means progress now depends on getting more diverse candidate programs out of each stage, and that's harder to buy with more compute than better selection was.

If you're evaluating this for anything real, treat it as evidence that decomposition-plus-reflection is worth prototyping on your own hard reasoning tasks — not as proof that ARC-AGI-1 is solved.