Recommended check: the useful question for a managing partner is not which AI tool scored best on someone else's benchmark. It is what evidence the firm holds before it lets a tool touch intake, document review or client communication. Our answer is that the firm needs its own evaluation and feedback loop: a failure ledger, a per-task rubric, a regression set, and a named human who adjudicates disagreements. Published benchmarks and engineering guidance are good sources for the method. They do not replace that loop.
What the three sources say, and what they do not
Our interpretation: We used three public sources. Each comes from a vendor or vendor-run project, so we treat them as attributed claims and design guidance, not independent proof of results. We have not run any of the tools or benchmarks described below.
- Project Arena (Edge Delta project, publication date not supplied; we accessed it on October 19, 2026). The Project Arena repository describes a vendor-neutral starter for deploying a disposable Kubernetes fixture, injecting faults, saving investigation records from any product, and scoring completed investigations with a configurable judge. It offers a 21-scenario full suite and a six-scenario smoke fixture, and it treats cluster type and scenario suite as separate choices. The README also contains the project owner's own comparison of several products against the same incident facts and rubric. We are not reproducing those scores. The comparison comes from a party with a product in it, the excerpt we reviewed is truncated, and a law firm cannot infer its own results from a Kubernetes incident benchmark.
- Anthropic engineering, "Demystifying evals for AI agents" (published January 9, 2026). The Anthropic post on agent evals defines an eval as a test for an AI system: give it an input, then apply grading logic to the output. It argues that without evals teams get stuck in reactive loops, catching issues only in production, where fixing one failure creates others. Its stated limit matters: the post focuses on automated evals that run during development without real users. It is vendor guidance, and we have not assessed the full post’s coverage of production monitoring.
- Anthropic engineering, "Building effective agents" (published December 19, 2024). The earlier Anthropic post separates workflows from agents, and reports that the most successful implementations used simple, composable patterns. Its own note says much of the tooling it describes has changed since December 2024. We use it for vocabulary only, not for current tooling advice.
Why this is a governance question
Our interpretation: We are not asserting a specific bar rule or regulatory duty here. None of the supplied sources address legal obligations, and we will not infer any. The governance point is narrower and practical. If a tool produces a bad outcome, the firm should be able to show what it tested before adoption, what failed, who decided what to do about it, and what changed afterward. Without that record, trust in the tool rests on a demo and a vendor's word.
Our interpretation: There is a second reason to build the loop. A single headline score compresses many different behaviors into one number. We made this argument in an earlier piece on why similar agent scores hide different failures. A failure ledger is the instrument that keeps those differences visible.
Step 1: Define the task, not the tool
Our interpretation: Project Arena's structure suggests a useful discipline: the scenarios, the incident facts and the rubric are fixed, and the product under test is the variable. Do the same. Before you compare or trial anything, write down the task in one paragraph. For example, an illustrative task for an estate planning firm: "Read an inbound web inquiry, collect the minimum facts for a conflict check, flag any stated deadline, and route to a human. Do not give legal advice."
Recommended check: Note what the task includes: inputs, expected outputs, forbidden actions and an escalation path. If you cannot write those, the tool is not ready to be evaluated, and neither is the process around it.
Step 2: Keep a failure ledger
Our interpretation: A failure ledger is a running table of every observed problem, from the first pilot onward. It is the firm's own first-party evidence, and it is the thing a vendor benchmark cannot give you. Suggested fields:
- Recommended check: ID and date of the observation.
- Recommended check: Task and input (a synthetic or redacted copy, not a live client file).
- Recommended check: What happened versus what the rubric expected.
- Recommended check: Category: wrong fact, skipped step, forbidden action taken, missed escalation, confidentiality exposure, tone or format.
- Recommended check: Severity as judged by a named reviewer, not by the tool.
- Recommended check: How it was detected: automated check, reviewer sampling, staff report or client complaint.
- Recommended check: Root-cause hypothesis, labeled as a hypothesis until confirmed.
- Recommended check: Fix and linked regression test, or a documented decision to accept the risk.
Our interpretation: An illustrative entry, not a measured result: "Inquiry mentions a hearing date in the body of a long message; tool routes as routine; expected urgent flag; category: missed escalation; severity: high; regression test added." The value is not any single row. It is that patterns become countable: the same category recurring across tasks tells you something a pass rate cannot.
Step 3: Write a per-task rubric
Project Arena scores every investigation against the same incident facts and the same rubric. That is the part worth borrowing. For each task, the rubric should contain:
- Recommended check: Ground-truth facts for each test case, written by someone who knows the answer before the tool runs.
- Recommended check: Required elements the output must contain.
- Recommended check: Forbidden behaviors, such as giving advice, sending a message without approval, or writing to a field it should not touch.
- Recommended check: Escalation conditions that must trigger a human handoff.
- Recommended check: Grader type for each criterion.
On grader type, Anthropic's definition of an eval, an input plus grading logic applied to the output, leaves room for different kinds of logic. Our working split, which is our own practice suggestion and not a claim from the source: use deterministic checks where the criterion is checkable (did the record get created, was the forbidden field untouched), use a model-based judge for qualitative criteria, and use a human for anything high-severity. Project Arena makes its judge configurable, which is a reminder that the judge is itself a variable. Record which judge and which version produced each score, and calibrate it against human decisions before you rely on it.
Step 4: Turn the ledger into a regression set
Anthropic's warning about reactive loops, where fixing one failure creates others, is the case for a regression set. Every ledger entry that can be reproduced becomes a permanent test case. Run the set whenever anything changes: the model version, the prompt, a tool connection, the CRM integration or the intake form.
Project Arena's two suite sizes offer a workable shape. Keep a small smoke set, in the spirit of its six-scenario fixture, that runs on every change and finishes quickly. Keep a fuller set, in the spirit of its 21-scenario suite, for larger changes such as swapping models or vendors. The counts are the repository's, not a recommended size for a law firm; start with whatever your ledger supports.
Recommended check: Two practices make the set trustworthy:
- Disposable, synthetic fixtures. The Arena fixture is disposable by design. Do likewise: test against synthetic or redacted matters in a sandbox, never live client records.
- Recommended check: Add passing cases too. A set made only of past failures will not tell you when a change breaks something that used to work.
Step 5: Assign human adjudication
Recommended check: Automated graders will disagree with each other and with your staff. Decide in advance who settles it. Name one attorney or senior operator as adjudicator for each task. When the judge and a reviewer disagree, the adjudicator rules, writes a one-line reason, and the ruling is attached to the ledger entry. Over time those rulings become the calibration data for your rubric and your judge.
Recommended check: Also sample passes, not just failures. A reviewer who only sees flagged items cannot learn how often the automation is confidently wrong.
Add a live-use feedback channel
Anthropic states a focus on automated evaluations during development without real users. We have not assessed the full article’s coverage of live-use feedback. Our recommendation is to design a feedback channel for the deployed workflow. A minimal version: give staff a one-click way to report a bad output, route those reports into the ledger weekly, and review the severity-high items within a set window. Whether this catches problems that development-time testing misses is a hypothesis to check in your own data, not a finding from these sources.
Questions to put to any vendor
- Recommended check: Can we run our own task set and rubric against your tool before signing, on synthetic data?
- Recommended check: When your tool, model or prompt changes, how and when do you tell us, so we can rerun our regression set?
- Recommended check: Can we export full run records, not only scores, so reviewers can inspect what the tool actually did?
- Recommended check: Which grader produced any score you show us, and can we see its criteria?
Limits of this method
Our interpretation: A ledger and a regression set show what you have tested, not what you have not. Small samples will not support statistical claims, so report counts and categories rather than percentages until the data justifies more. The illustrative entries above are templates, not measurements from AlphaForge or any firm. And nothing here shows that any particular tool is safe, accurate or suitable for legal work; it describes how a firm can generate the evidence to decide for itself.
Recommended check: The practical starting point is modest: one task, one rubric, ten synthetic cases, one adjudicator, and a ledger that begins on day one of the pilot.