DAYJOB: Tax Benchmark Puts AI Agents’ Judgment to Test

Surge AI’s DAYJOB: Tax benchmark gives AI agents complex tax assignments and scores whether they can do the work—and notice consequential issues clients may not know to ask about. Claude Sonnet 5.5 leads at a 10.8% pass rate, a result that says more about how far agents have to go than about readiness to handle client tax work.

What the DAYJOB: Tax benchmark measures

The benchmark covers 50 environments across eight areas of tax practice. Each workspace contains financial and tax records, and assignments test tax knowledge, calculations, evidence retrieval, instruction following, and professional judgment. Across the workspaces are 1,789 files: a median of 30 per assignment and as many as 123, including 1,487 PDFs, 117 Excel workbooks, and 38 Word documents.

That mix matters because tax work is not just a calculation problem. An agent may need to find evidence across documents, reconcile conflicting records, and decide which questions to raise before advising a client. DAYJOB: Tax explicitly tests whether agents investigate issues the client has overlooked—not only whether they answer the question they were given.

A 10.8% pass rate is a demanding signal

The leaderboard’s leading model, Claude Sonnet 5.5, has a 10.8% pass@1 rate. Claude Fable 5.1 scores 9.6%, and Claude Opus 5.5 scores 9.2%. An attempt counts as a pass only when it meets a 95% rubric threshold.

Those numbers are sobering, but they need the rubric alongside them: this is not a general measure of tax knowledge, but a strict test of complete performance on these assignments. My read is that the narrow gap between the listed models is less important than the low absolute pass rates. Even the leader rarely clears the benchmark’s bar on a single attempt. That is useful evidence against treating a plausible answer—or a strong result on a narrower task—as proof of dependable end-to-end tax work.

The overlooked question is part of the job

The condo example makes the benchmark’s standard concrete. Astra failed to identify a deadline that could affect eligibility for a tax exclusion; Opus identified it in all five attempts. The point is not simply that one model caught something another missed. The assignment asks whether an agent will investigate a consequential issue the client may not know to raise.

That is a harder and more practical bar than answering a well-formed prompt. In production, a tax agent’s mistake could be an omission hidden inside otherwise competent work. But the figures here do not establish how often these systems would succeed outside DAYJOB’s environments, or what review would be needed before a client relied on them. My takeaway: the benchmark tests a capability worth measuring, while its low pass rates argue for keeping human review firmly in the loop. The next useful signal is whether agents can improve on these complex assignments without losing the habit of checking evidence and surfacing the question nobody asked.