AI Training Data Needs Better Agent Evaluations

Handshake’s AI training data business reached $1 billion in gross annualized revenue roughly a year after launch, according to the source article. The more useful story is what the work involves: building expert data and reinforcement-learning environments for frontier labs, then testing whether agents can handle tasks that are harder than producing a plausible final answer.

AI training data is being tested through real work

The episode demonstrates three kinds of evaluation: financial work, scientific image interpretation, and safety. In ATLAS-Finance, an agent works across a simulated bank’s inboxes, documents, and financial models. VIALS asks models to interpret scientific images, including gel and brain-tissue samples. These are not interchangeable tests: each puts a different kind of applied judgment under pressure.

The demonstrations also show where models make mistakes. In finance, agents can fail to apply financial rules; in tissue-image interpretation, they can misread what they see; and in file handling, they can behave unsafely. The article does not provide benchmark scores or a broad comparison across models, so the demonstrations are better read as concrete examples of failure modes than as proof of a general performance level.

A final answer hides much of the evaluation

One point from the episode is especially relevant to anyone evaluating agents: “A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating.” That distinction matters. An agent can produce a convincing summary while making poor decisions along the way; for these tasks, the work process and handling of inputs can matter as much as the answer it returns.

That is my read of the benchmark examples, not a claim that they settle how agents perform in production. A simulated bank and selected scientific images can make evaluation more concrete, but the source material does not show how well results transfer to messy, changing workflows or how consistently the tests catch failures at scale. The useful shift is from asking only whether a model can answer to asking what it did to get there—and what it touched along the way.

Revenue signals demand; benchmarks show the harder question

Handshake’s reported revenue milestone suggests substantial demand for expert data and training environments. It does not, by itself, establish that the resulting models are reliable, or that the benchmarks predict performance outside the settings shown. Those are separate claims, and the demonstrations make the gap visible rather than closing it.

For teams choosing or building agents, the practical takeaway is to evaluate the full task: rule application, interpretation, and safe file handling—not just the final response. The source offers useful examples of what to test, but not enough evidence to declare which models are ready for deployment. The next signal worth watching is whether these evaluation approaches show consistent results beyond their demonstrated tasks.