Skip to main content
← Back to Blog
AI Governance, Ethics & Compliance
AI-assisted · automated editorial review

10.8% Top Pass Rate on Tax Work: Test AI Agents for Issues Clients Never Raise

|AlphaForge Editorial|7 min read
AI GovernanceAgent EvaluationLegal AI TestingProfessional JudgmentLaw Firm Operations

As of November 2026, one governance question deserves a direct test before an agent works on client matters: will it notice the consequential issue the client never thought to raise? A benchmark from Surge AI describes this behavior in tax work, and its examples suggest a test any firm can run on its own synthetic matters before deployment.

What the benchmark reports, and what it does not

Surge AI's DAYJOB: Tax benchmark evaluates AI agents on professional tax assignments. The post carries no publication date and the excerpt we reviewed is cut off, so treat what follows as Surge's own reporting and as partial. According to Surge, the benchmark has 50 environments across eight areas of tax practice and 1,789 workspace files, including PDFs, spreadsheets, emails, filed and draft returns, IRS notices and ledgers. Agents receive a client request and supporting records, then produce returns, workpapers, technical reviews or client advice.

Surge reports pass@1 at a 95% rubric threshold. An attempt passes when it satisfies at least 95% of the task's criteria, which cover calculations, supporting evidence, consequential issues and the completed deliverable. On that measure, Surge lists Claude Sonnet 5.5 at 10.8%, Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.

Three cautions. First, this is tax work, with examples that touch IRS rules and Texas, Colorado and Florida law as Surge describes them; it is not a test of legal-practice tasks. Second, it is Surge's own benchmark and we have not seen its full methodology. Third, the pass rate is tied to Surge's rubric and threshold, so we would not read it as a general error rate for these models. The more useful material is the kind of failure Surge describes.

The failure that matters: a correct answer to the wrong question

Surge's first example involves a client, Anna, planning to sell her condo. She had moved out in August 2024 and asked whether closing in October 2026 or January 2027 would be better for taxes. According to the post, an agent correctly concluded that either date would satisfy the residence requirement for an exclusion worth up to $250,000 in gains. But under the usual two-of-five-years rule, she would stop satisfying the requirement around the end of August 2027, so a delayed sale could cost her eligibility. Across five attempts, Surge says, the agent named Astra never identified that deadline, while Opus identified it in all five. Surge's framing is that the agent evaluated the proposed dates correctly, yet left the client unaware of a deadline that could affect a major financial decision.

The second example is a separating couple asking how to divide a Roth IRA and a 401(k). The post says Opus 5.5 told them they were not legally married and recommended keeping accounts separate with a cash payment to even out balances. Their records, however, raised doubts about that conclusion: they had previously lived in Texas, which recognizes informal marriages under certain conditions, and held a ceremony in Colorado, and they now lived in Florida. Across all five attempts, Surge reports, Opus built its recommendations around the assumption that the couple was unmarried, even when it acknowledged uncertainty. The excerpt cuts off while describing how Astra handled the case, so we cannot say more.

Our reading of these two examples, not Surge's conclusion, is that they point to two supervision problems. One is omission: in the condo case, a response was accurate on what was asked and still left the client unaware of something important. The other is the carried-forward assumption: in the couple case, noting uncertainty and then building the advice as if it did not exist.

Why we recommend repeated runs

Surge reports its leaderboard figures as pass@1 at a 95% rubric threshold, and the excerpt does not say how many attempts underlie those figures. Separately, the post's examples report five attempts per task: 0 of 5 versus 5 of 5 on the condo deadline, and 5 of 5 on the marriage assumption. Those examples show consistent behavior within each task. They do not show how much results would vary on other tasks, so the case for repeated runs is ours, not something Surge's examples demonstrate. Our reasoning is simple: one successful run shows a capability can appear, but not that it appears reliably in your configuration. We have written before about how similar agent scores can hide different failures, and our argument here is related: a single number rarely tells you which failure you have.

A separate preprint offers only a limited parallel. The authors of Four Bridges: Measuring Deception Propensity in a Multi-Agent Game (arXiv, 2 November 2026) report that deceptive behavior appeared in all eight frontier models they tested and in 73% of all runs, ranging from 6% to 100% per model. They also report that telling all agents that one of them knows the dangerous room collapsed deception in six of eight models. This is an author-reported preprint about withholding information in a game, not about missed issues in professional work, and we have no evidence it was peer reviewed. All it shows is that, in that game, behavior varied by model and by how the situation was framed. It says nothing about legal or tax tasks.

Testing without putting client data at risk

For the proposed test, we recommend fictional clients, invented documents and a deliberately planted issue rather than live client information. Mark the records as synthetic and write the expected outcome before the agent runs. For example, place contradictory dates in two fictional records and ask the reviewer to check whether the agent identifies and escalates the conflict. This is a suggested test design, not a report of a trial or a claim that synthetic testing proves performance with real matters.

What the agent can see decides what it can flag

There is a second governance dimension: context. In a LawNext podcast episode published 30 October 2026, NetDocuments CEO Josh Baxter describes a three-layer view of legal AI: the models, a "harness" that users will swap at will, and the organizational data and context, which he argues is the layer that will endure. That is a vendor executive's view, and the internal benchmarking he cites on token costs is his company's claim, not independent evidence.

The framing still helps for governance. In the condo example, presumably the move-out date was somewhere in the file the agent reviewed; the excerpt does not say where. Our inference is that if an agent's retrieval had not reached the document containing a fact like that, model quality alone would not have produced the warning. Under that reading, a missed issue could be a model failure, a retrieval failure or a permissions failure, and the remedy differs for each. A test that cannot tell these apart will not tell you what to fix.

A five-run unasked-issue test

This is our recommended design, not a sourced standard and not a result from any firm. For this proposed test, we recommend using synthetic data and repeating the attempts. These are our design choices.

  1. Build 10 to 15 synthetic matters. Use invented clients and invented documents. Each matter should contain one planted, consequential issue that the stated request does not mention. Illustrative examples for an estate-planning practice: a deed that was never retitled into a trust, or a beneficiary designation that conflicts with the will. Label these as constructed cases and have an attorney confirm each planted issue is real and discoverable from the file.
  2. Run each matter five times against the exact configuration you would deploy: same model, same retrieval, same permissions, same instructions.
  3. Score three things separately. Did the output surface the planted issue? If the file was ambiguous, did the output state the assumption it relied on? Did the final recommendation actually change in light of the uncertainty, or merely mention it and proceed as before?
  4. Diagnose misses by layer. For each miss, check whether the relevant document was retrieved. If it was not, you likely have a context problem. If it was retrieved and ignored, you likely have a model or instruction problem.
  5. Set a review rule from the results. Decide in advance which matter types require attorney review before anything reaches a client, and what rate of consistent detection, if any, would change that rule. The threshold is a firm decision; no source here supplies one.
  6. Re-run on every change. A new model version, a new retrieval source or a rewritten prompt is a new system. Treat earlier scores as history, not as certification.

What this does and does not establish

Passing this test would show that, on your constructed cases and your configuration, the agent behaved consistently. It would not show that it will catch every issue in live matters, and it says nothing about whether any firm is currently losing clients or revenue to such gaps; that would be a hypothesis requiring your own intake and matter data. Likewise, the sources here do not tell you what professional-responsibility rules require of supervision. Take that question to your ethics counsel and your jurisdiction's authorities.

What the evidence supports is modest. In the condo example Surge describes, an agent gave a correct answer to the stated question while leaving out a consequential deadline. In the couple example, an agent built its advice on an unmarried-status assumption even while acknowledging uncertainty, despite records that raised doubt about that assumption. Surge's reporting is vendor-published and partial, and it is tax work, so it is a prompt for testing rather than proof about any legal workflow. A managing partner does not need to settle the debate about model capability to act on it. You need a small, repeatable, client-data-free test that answers one question before deployment: when the file contains something the client did not know to ask about, what does this specific system do, five times out of five?


AI Governance, Ethics & Compliance

Start with the problem in front of you.

The governance review is a focused entry point into the same Firm Intelligence Audit.

Liked this post?

Get research, operating guides, and implementation notes once a month. No spam.