Skip to main content
← Back to Blog
Revenue Leakage & Intake Performance
AI-assisted · source-checked automated editorial review

What a Medical AI Benchmark Can Teach Firms About Testing Intake Agents

|AlphaForge Editorial|7 min read
Intake PerformanceRevenue LeakageAI Agent EvaluationLaw Firm OperationsLegal AI Implementation

As of October 2026, the question for a managing partner is less whether agent features will reach legal work and more how to tell whether one placed near intake helps or quietly drops inquiries. LawNext reported on October 15 that Microsoft launched Copilot for Legal, the fourth major AI company in five months to ship a dedicated legal offering, following Anthropic's launch in May, Google's in August and OpenAI's in September. The news excerpts do not establish intake outcomes for any of these products. They do give us a reason to build a test before an agent arrives, and a preprint published the same week offers an evaluation design worth borrowing.

This article separates what the sources say, what we infer from them, and what only a firm's own data can answer. The intake test in the later sections is our recommendation, not a sourced result.

What the announcements say, and what they leave out

Sourced fact (news reporting, not independent proof): According to LawNext, Microsoft's launch offers more than 20 legal-specific skills in Microsoft Copilot, bringing contract review, drafting, obligations tracking and other legal workflows into Word and Cowork, its agentic workspace, with plugins from more than two dozen legal technology partners. LawNext adds that all four vendors follow essentially the same model: legal skills or agents, connectors into specialist products, and a promise that lawyers can get more done without leaving the platform. LawNext's report lists document workflows and observes that lawyers already live in Word, so Microsoft needs them to use a new capability rather than a new application. That is the publication's observation about Microsoft's offering. It tells us nothing about which tools any particular firm licenses.

One integration example: A second LawNext report describes an integration between Legora and Blue J built on the Model Context Protocol (MCP). Legora customers with a Blue J subscription can query Blue J from inside Legora and receive answers with inline citations linking back to Blue J's sources, and Blue J performs the research on its own infrastructure. This is a tax research case, not intake, and one example does not establish a general pattern. It does suggest two questions for any vendor: where the work runs, and whether answers link back to checkable sources.

What is missing: No retrieved independent source provides measured intake outcomes for these products. A vendor guide from Layer3 Labs asserts intake-time and billing-leakage improvements, but the retrieved excerpt shows no evidence for them, so they should not be treated as independent measurements.

Leakage is a hypothesis until a firm's own records say otherwise

Revenue leakage at intake means inquiries that arrive and then stall, get misrouted, wait too long or never get a defined next step. From public evidence alone, we cannot say any specific firm has this problem. A website, a phone number and a review profile do not show what happens after an inquiry lands. Treat leakage as a hypothesis and test it against first-party records: the inquiry log, timestamps for first response, the assigned owner and the final disposition.

You will also see large, tidy vendor numbers for what administrative drag costs a firm. They are not intake-specific, and we do not rely on them here. Your own inquiry log will say more about your firm than a borrowed average.

A benchmark design from a different field

The most useful retrieved source for evaluation is an arXiv preprint on AI-driven reproducibility in scientific discovery, posted October 16, 2026. It is an unreviewed preprint from a medical research setting, and everything below is the authors' claim as stated in the abstract. It says nothing about law or intake.

According to the abstract, the authors built SEER-Repro, a benchmark of 20 published papers based on the SEER cancer registry, together with the data needed to reproduce their findings. They created an agent that reproduces 77.4% of 2,190 atomic claims. They then built a separate agentic pipeline that identifies the analytical choices behind each finding, proposes defensible alternatives and reruns the analysis. That pipeline flagged potentially consequential choices in 9.5% of fully reproduced findings and routed them to expert review.

Our reading, which is an inference and not something the authors claim for legal work, is that the design has four transferable features:

  • Atomic claims. Output is broken into small units that can each be checked, not graded as one lump.
  • A fixed benchmark. The agent is scored against cases where the right answer is already established.
  • Robustness reruns. Alternatives to the agent's choices are tried to see whether the conclusion changes.
  • Expert review for flagged items. Uncertain results go to a person instead of passing silently.

A headline rate like 77.4% deserves care. It is a rate on a benchmark the authors assembled. It does not tell you the error rate on cases the agent has not seen, and the abstract does not claim it does.

Translating the design into an intake test

This is our recommended approach. It is untested as a package and carries no promised outcome. Before an agent drafts a reply, routes a call or books a consultation on live inquiries, build the equivalent of a small benchmark:

  1. Assemble historical inquiries. Pull a sample across channels (web form, phone notes, email, referral), with identifying details removed or handled under your confidentiality rules. Include messy ones: incomplete forms, after-hours messages, a family member calling for someone else, a matter outside your practice areas.
  2. Record the known-correct handling. Have a senior intake person write down the matter type, urgency, owner, what the first response should contain and when it was due.
  3. Define atomic checks. Examples: matter type captured; names needed for a conflict check captured; urgency flagged when a deadline was mentioned; an owner assigned; a dated next step created; no statement that sounds like legal advice or a promise of representation.
  4. Score the agent claim by claim. Report each check separately. A single overall score hides the failures that matter most.
  5. Separate leak errors from noise. A dropped inquiry, a misroute or a missed deadline flag is a different class of error from an awkward sentence. Weight them differently and set zero tolerance for the worst class before go-live.
  6. Rerun with variations. Rephrase inquiries, change the channel and time of day, and drop a field. Check whether the handling changes when it should not.
  7. Define the flag-for-review rule. Decide in advance what the agent must hand to a person: anything involving a deadline, an adverse party, a minor, a vulnerable client or low confidence in its own classification.

For estate planning, probate, trust and inherited-property intake, the same structure applies. The benchmark simply has to include the situations those practices actually see, such as an urgent call about a recently deceased relative or a question that touches more than one family member's interests. Which of those cases go wrong, and how often, is a first-party question that only your records can answer.

Why the human gate is part of the design

An intake error is asymmetric. A wrong summary is cheap to correct. A promised callback that never happens, or a message that implies representation, is not. The Layer3 guide asserts that any automation touching client data, generating documents or making routing decisions needs attorney oversight built in from the start. That is a vendor's statement, not a rule we can cite, and the applicable obligations depend on your jurisdiction and your own counsel's read. As a design principle, though, we recommend deciding where approval sits before the agent has permission to act. We wrote about the mechanics in an earlier post on gating an agent before it acts.

Artificial Lawyer announced a webinar with Harvey for November 2 on choosing, building and rolling out legal agents, with human review and picking a first use case on the agenda. That is an event announcement from a vendor partnership, not evidence of results.

Questions to put to any vendor, including those in Word

When an agent capability shows up inside a platform your firm already uses, adoption can feel like a settings change. LawNext observes that lawyers already work in Word, which lowers the friction of trying a new capability. Lower friction cuts both ways: a feature that is easy to turn on is easy to turn on without a test. Before enabling anything near intake, ask:

  • Where does the processing happen, and where is client data stored?
  • Does every output link back to something a person can check?
  • Can we export a complete log of inputs, outputs, edits and approvals, and who owns it?
  • What can the agent do without a person, and can we change that setting ourselves?
  • Can we run our own benchmark against it before it sees a live inquiry, and again after each model or configuration update?

What we can and cannot conclude

Sourced facts: Four major AI companies have launched legal offerings in five months, according to LawNext. One MCP-based integration connects tax research to a legal AI platform. A preprint reports a benchmark-and-review design in a medical setting.

Public observation: LawNext's report on Microsoft's launch names document workflows, and the Blue J integration concerns tax research. Those examples do not establish intake performance.

Hypothesis: A firm that cannot show, claim by claim, how an agent handles its own historical inquiries has no basis for believing the agent reduces leakage rather than adding to it. That hypothesis needs your data to confirm or reject.

First-party evidence needed: Time to first response by channel, share of inquiries with an assigned owner within a set window, the disposition of inquiries that went quiet, and the agent's scores on a benchmark built from your own cases. We have not tested any of the products named above, and nothing here predicts what an agent will do for your firm.

A practical next step this month is to pull a sample of your last quarter's inquiries, write down what correct handling would have been, and score your current process against it. That gives you the baseline any agent will have to beat, and it is the same exercise whether or not you ever deploy one.


Revenue Leakage & Intake Performance

Start with the problem in front of you.

The focused review is a conversation starter. Deeper findings move into the Firm Intelligence Audit.

More from the Blog

Market MovesAI Agents

Enterprises Will Spend $201.9B on AI Agents in 2026 — Here's What SMBs Should Steal From the Playbook

Gartner says enterprises will spend $201.9B on AI agents in 2026. Here's the 3-move playbook SMBs can steal — and deploy for $1,200, not $300K.

·4 min read
StrategyPricing

Stop Selling Automation — Sell Outcomes: The New AI Agency Playbook for 2026

Automation is commoditized. Every agency can spin up a chatbot. The agencies winning in 2026 charge for results — qualified leads, closed deals, measurable ROI. Here is the playbook.

·7 min read
MCPTechnical

MCP Hit 97 Million Downloads — Why This Protocol Is the USB-C of AI Agents

Anthropic's Model Context Protocol is now supported by ChatGPT, Gemini, Copilot, and 10,000+ public servers. One universal connector for AI agents. Here is what it means for your business.

·8 min read
Industry NewsStrategy

Mastercard Just Gave Every Small Business a Virtual CFO — What That Means for AI Agents

Mastercard launched Virtual C-Suite — AI agents acting as CFO, CMO, and COO for small businesses. The biggest companies in the world just validated exactly what we build. Here is why custom beats generic.

·8 min read
Voice AIROI

Voice AI Agents Are Killing the Missed Call — Here's the ROI Math

73% of legal leads go to voicemail. 40% of real estate leads come after hours. Voice AI agents report 3.7x ROI per dollar invested. Here is the math and what it means for your business.

·9 min read
ArchitectureMulti-Agent

Multi-Agent Teams: Why One Agent Is Never Enough

Single agents hit a ceiling fast. Specialized teams of 2-5 agents — each owning one job — outperform generalists by 3-5x on complex workflows. Here is how to architect agent teams that actually scale.

·8 min read
IntegrationMCP

MCP Explained: How Your Agents Connect to Everything

Model Context Protocol is doing for AI agents what USB-C did for devices. One standard protocol to connect any agent to any tool — CRMs, email, databases, APIs. Here is what it is and how we use it.

·7 min read
PricingROI

The Real Cost of AI Agents: What SMBs Actually Pay

AI agent pricing ranges from $0 to $50,000 per month depending on who you ask. Here is a transparent breakdown of what things actually cost — LLM APIs, infrastructure, build time, and ongoing management.

·9 min read
DeploymentInfrastructure

VPS vs. On-Prem: Where Should You Host Your AI Agents?

Your AI agents need a home. We break down the trade-offs between cloud VPS hosting and on-premises deployment — cost, security, latency, and control — so you can pick the right setup.

·6 min read
SecurityOpenClaw

How We Secured Our Agents After CVE-2026-25253

When a critical vulnerability hit the OpenClaw framework, we patched every client agent within 4 hours. Here is what happened, what we did, and the security kit we open-sourced.

·8 min read

Liked this post?

Get research, operating guides, and implementation notes once a month. No spam.