As of October 2026, consider an illustrative intake agent that answers calls, texts web-form leads, qualifies inquiries and books consultations. In my judgment, which is an observation and not a sourced finding, an evaluation built on completed transcripts is an easy one to run, because the agent can take as long as it needs. A live prospect does not wait. This article asks one question: does an intake agent that performs well on a paused transcript still perform when the conversation keeps moving? The answer matters for revenue leakage, but it is a hypothesis to test, not a finding about any firm.
What the new evidence says, and what it does not
A preprint posted on arXiv on October 27, 2026, called Gym-Anything-RT, builds 100 visual minigames to test vision-language agents in real-time settings, where the environment can change while the agent is still reasoning. The authors' key design choice is that the same minigame can run in real time or paused. They report large capability gaps. In their example, GPT-6 Astra (xHigh) passes 9.0% of the real-time minigames and 75.0% when the environment is paused. They also report that, when given the ability to write code, the most powerful frontier agents write code that bypasses the need to perceive and act in real time.
Be precise about the limits. What I reviewed is the abstract, and the figures are author-reported. I saw no indication of peer review or independent reproduction, so I treat the results as unreplicated claims. That is my inference from what was retrieved, not a statement in the abstract. The tasks are video-game-style minigames, not phone calls, chats or legal intake. The numbers say nothing about how any intake product performs or how many inquiries a firm loses, and the 75.0% figure is one agent's result on minigames, not a baseline for intake. In my judgment, what may transfer is the method: run the same task twice, once under time pressure and once without, and look at the difference.
Why this matters for intake specifically
Intake is a real-time channel. A caller may hesitate, interrupt, correct a name, mention a deadline and then mention a second matter. A chat visitor may open another tab. A text thread may go quiet for four minutes and then resume mid-thought. Each of these is an example of the environment changing while the agent is still working. This description is illustrative, drawn from how intake conversations commonly unfold in my experience, not from a cited study.
A transcript-based test removes all of that. The agent sees the complete conversation, reasons at leisure and produces a clean summary. That is the paused condition. My inference, not a sourced finding, is that if a vendor's evaluation or your own pilot covers only that condition, it has not tested the live condition, and its results may not tell you how the agent behaves in the channel where prospects actually arrive.
Here is the leakage hypothesis, stated as a hypothesis. If an agent's live behavior is materially worse than its paused behavior, facts could be dropped, interruptions mishandled, or qualifying questions skipped in exactly the moments when a prospect decides whether to stay. Whether that happens at your firm, and how often, is a question for your own recordings and logs. Public evidence cannot answer it.
A paired real-time and paused test for intake
You can copy the preprint's structure without copying its scale. We recommend a paired comparison on the same inquiries, run in this order.
- Build a set of real or realistic inquiries. Use 30 to 50 de-identified past inquiries, or scripted ones that cover your matter types. Include messy ones: interrupted callers, corrections, multi-matter calls, a prospect who gives a deadline late in the conversation. Strip or synthesize any client-identifying detail before it goes into a test harness.
- Write down the expected record for each one. For every inquiry, list the facts a human intake specialist would capture: names, relationship to the matter, jurisdiction, urgency, conflict-check inputs, and the next step promised. This is your answer key.
- Run the paused condition. Feed the agent the complete transcript with no time limit. Score the resulting record against the answer key.
- Run the live condition. Replay the same inquiries through the real channel configuration: actual speech or message timing, actual latency, actual interruptions, with the agent responding turn by turn. Score the record the same way.
- Compare field by field. Do not collapse the result into one pass rate. Report which fields degrade live and which do not.
- Check what the agent does when it is slow. Record the delay between a prospect's turn and the agent's reply, and what the prospect hears or sees during that gap.
Also keep a log of attempted versus completed actions. If the agent tried to book a consultation but the calendar write failed, that is an attempted action, not a completed one, and it should be counted as such. One way leakage could occur is that the conversation looks fine while the downstream record never gets created. Only your own logs can show whether that happens.
Watch for the shortcut behavior
The preprint's observation about code is worth considering by analogy. According to the authors, when able to write code, the most powerful frontier agents wrote code that bypassed the need to perceive and act in real time. By analogy, and this is my hypothesis rather than anything the paper tests, an intake workflow might quietly convert a live conversation into something asynchronous: it promises a callback, drops the prospect into a form, or defers the qualifying questions to a follow-up email. Sometimes that is a sensible design. But it is a different process from the one you thought you were testing, and it changes when and whether the prospect gets a human response. Count it as a result, and label it as a deferral rather than a successful live intake.
The intent gap: your answer key may not exist yet
Step 2 is where firms may stall, and a second preprint offers one possible explanation. In a paper on collaborative agent-driven software engineering, also posted on October 27, 2026, the authors argue that agents enter processes that quietly relied on people to cover gaps: a team's intent was never fully written down because colleagues filled it in, and an agent fills it in by guessing. The paper is about software teams, and the authors describe a methodology that emerged from an industrial need, with plans to evaluate it further. It is not evidence about law firms.
Applying it to intake is a working hypothesis, not a finding. Ask your intake staff what they do that is not in any written procedure. They may know to ask whether the caller is the named fiduciary or merely a relative, to notice that a property address implies a different county, or to treat a certain phrase as urgent. If that knowledge lives only in people's heads, an agent may guess, and a paused transcript test may not reveal the guess because nobody wrote down the right answer. On that reasoning, which is my inference and not a result from the paper, writing the answer key is not paperwork. It is the first piece of the test, and it may expose process gaps that exist regardless of any AI.
Adoption is not the same as revenue effect
Be careful with claims that adoption leads to growth. A legal AI vendor's guide to law firm AI case studies says Clio titled its 2026 solo and small firm report around the finding that small firms lag larger peers on using AI to grow revenue, and that the major 2026 surveys disagree because they measure different things. The guide also attributes to Thomson Reuters a figure of 41 percent of law firms saying their legal teams use generative AI, up from 28 percent the prior year. This is vendor-published material relaying other parties' figures, and the Clio reports are not linked there, so the claims are second-hand. I read it as supporting a narrow point: using a tool and converting that use into business outcomes are separate steps, and the second is harder to evidence.
That is why, in my inference rather than any source's conclusion, a paired test belongs before any rollout metric. A usage count tells you the agent was switched on. It does not tell you whether live inquiries were captured accurately.
How to read the results
Three outcomes are possible, and each suggests a different action. These are my recommendations, not findings from the sources.
- Little gap between paused and live. Encouraging, but only for the inquiry types you tested. We recommend extending the set before widening scope.
- A gap concentrated in a few fields. Fix those fields. If, for example, urgency and conflict-check inputs drop live, have a human confirm those two items before the record is final.
- A broad gap. Keep the agent out of the live channel, or restrict it to after-hours capture with a human review queue, until the cause is understood.
In every case, keep the failures. A running ledger of what broke, with the inquiry that triggered it, becomes your regression set for the next model or configuration change. Our earlier piece on building a failure ledger and regression set before trusting an AI tool covers how to structure that, and the real-time and paused scores are two more columns in it.
What this does not prove
Nothing here shows that a particular firm is losing inquiries, or that fixing live behavior will change conversion or revenue. In my inference, a live-versus-paused gap in a test harness is a signal about measurement validity: it would indicate whether your pilot results can be trusted for the live channel. Whether leakage exists is answered only by first-party data such as call logs, response timestamps, unanswered-inquiry counts and the disposition of each lead, ideally reviewed with the staff who handle them.
The practical recommendation for a managing partner or operator is small and concrete. Before approving an intake agent, ask the vendor or your implementer one question: show me the same inquiry set scored live and scored paused. If they cannot, my judgment is that what you have seen is a transcript demo, not a live-channel evaluation.