JavaScript Framework Benchmarks Need More Than One Run

JavaScript framework benchmarks are only useful if the builds are comparable—and if one lucky run doesn’t decide the result. This experiment had coding agents build the same Pokémon app across JavaScript frameworks, then evaluated the output with a conformance suite and several measures of cost and quality.

What the JavaScript framework benchmarks measured

The evaluation tracked development time, token use, lines of code, and Lighthouse performance and accessibility scores. That mix matters: it looks beyond whether an agent produced a working app and asks how much effort it took, how much code it left behind, and how the result scored on the measured Lighthouse dimensions.

A shared app and conformance suite give the comparison a common target. But they do not make every run identical. The source article explicitly describes the experiment as imperfect: agent behavior can vary, and Lighthouse results can be stochastic too. The results were therefore reported as medians across repeated builds rather than as a single run.

That is the right instinct for a benchmark involving agents. A clean demo can show that a framework is possible to use; repeated attempts start to show how reliably an agent can get there. Those are different questions, and the second is closer to what I care about when deciding whether to build around an agent workflow.

Why the run count changes the comparison

Most frameworks were tested in five iterations. Angular was tested in three after the agent failed to create a working app twice. In other words, the Angular result uses fewer successful iterations, and the failures are part of the reported process rather than noise to quietly discard.

That difference is important when reading medians. A median over five runs and a median over three runs do not offer the same amount of evidence about run-to-run variation. The two failed attempts also raise a practical question that a single score cannot answer: how much of the experience is captured by the quality of successful builds, and how much by the chance that a build never becomes usable? The material provided does not supply enough results to quantify that distinction, but it is worth keeping in view.

My take is that reporting the failed runs makes the benchmark more useful, not less. In an integration, an agent that sometimes fails to produce a working app creates extra debugging and retry work. A benchmark should make that friction visible, while still being careful not to treat a small number of attempts as a stable failure rate.

What this benchmark can tell you—and what it can’t

The source describes the evaluation design and its limits, but the available summary does not give the individual framework scores or identify a winner. So there is no basis here for choosing a framework based on cost, code size, or Lighthouse performance. Treating the experiment as a ranking would go beyond the evidence at hand.

What it does offer is a useful template for evaluating coding agents: hold the task steady, check conformance, track both resource use and output, repeat builds, and disclose failures. If I were applying this to a project, I’d want more repeated runs before making a framework decision—especially where failed builds are part of the cost. The benchmark’s most actionable lesson is not which framework to pick; it’s to measure agent reliability alongside the polished result.