MirrorCode and the Reality of Long-Horizon AI Engineering

MirrorCode is a new benchmark designed to see if AI models can actually build software from the ground up rather than just patching existing code. Developed alongside METR, it moves past the standard 'fix this bug' or 'write this function' tasks that have dominated the evaluation landscape. Instead, it tasks models with end-to-end program reimplementation, requiring them to recreate entire tools—like a bioinformatics toolkit—without access to the original source code, the internet, or external lookup tables.

The Cost of Autonomy at Scale

The benchmark reveals a massive gap between 'chatting' with an AI and letting it work autonomously. To solve these long-horizon tasks, MirrorCode provides enormous inference budgets, allowing models to run for days or even weeks. For example, one of the largest tasks involved an AI working for 19 days without human intervention, costing $2,600 for a single run. In contrast, the Opus 4.7 model managed to solve a complex bioinformatics toolkit (gotree) in just 14 hours at a cost of $251, while passing 2000 out of 2001 tests.

These numbers highlight a shift from low-latency responses to high-latency, high-cost autonomous work. By sandboxing the models and using a 'memorization screen' to catch data contamination, the benchmark forces the AI to actually reason through the logic of a program rather than just retrieving it from its training data. It’s a move toward measuring persistence and architectural planning over simple pattern matching.

The Gap Between Benchmark Success and Production Reality

The real story here isn't just that an AI can now write 16,000 lines of Go code; it's what these results don't tell us about the reliability of that output in a production environment. While passing 2000/2001 tests on a specific toolkit is impressive, it doesn't account for the technical debt, maintainability, or edge-case stability of the generated code.

In practice, we are seeing the birth of 'agentic persistence'—the ability for a model to stay on task for 19 days—but we are still missing the framework for how these models handle iterative feedback from human engineers or how they manage state across massive, multi-file repositories. The benchmark proves that the compute is there to support long-horizon thinking, but the jump from a sandboxed 'reimplementation' to a live, evolving codebase remains the next hurdle. We’ve moved from asking if AI can write a line of code to asking if it can hold a project's logic in its head long enough to be useful, but we still need to see how that code survives a real-world deployment where the requirements aren't static.