Use Risk-Based Acceptance Tests Before an AI Workflow Goes Live

Proposed October 2026 acceptance plan: we recommend evaluating the complete workflow before allowing an AI tool to act on firm records or communications. Start with a specific proposed task, such as preparing an intake record or drafting an acknowledgement. Define what success, refusal and escalation should look like before the test. Treat a passing result as evidence about the approved cases and conditions, and keep the required controls in place afterward.

What an acceptance test can and cannot tell you

Two sources frame the approach. The NIST AI Risk Management Framework Playbook is voluntary guidance, not jurisdiction-specific legal authority. Its Measure section says that measurements which average across test cases can miss pockets of failure where costs are significant, and that current AI systems depend on the context in which they were built, so local evaluation and continuous monitoring are needed. It also says that risks you will not or cannot measure should be documented, not silently dropped.

The second source is the OWASP AI Agent Security Cheat Sheet, also voluntary technical guidance. It says AI agents should undergo structured security testing before production deployment and again after material changes to prompts, tools, memory, retrieval, policies, or model providers.

We recommend having the named risk owner set the acceptance criteria and explain the remaining unknowns. Do not treat a fixed number of clean runs as permission to remove approval or expand access. Record the scope that was tested and the separate decision about what, if anything, may proceed to a supervised pilot.

Step 1: Inventory the actions and tier them by consequence

We recommend listing every action the proposed workflow could take, then assessing the consequences of each. Use the following task families as a starting point for the firm to adapt, rather than assuming an action is safe because it only reads information or changes an internal record.

• Assess reading and retrieval. Test whether the workflow selects the permitted matter and information for the intended person. Include cases with similar names and restricted records. Check whether the result could expose information through a summary or later message.
• Assess internal changes. Check draft records, notes and proposed calendar entries for notifications, synchronization and other downstream effects. Require evidence that a change and its consequences can be undone before calling the action reversible.
• Assess externally visible work. Include proposed messages, portal updates and shared document links. Test the exact destination, content and approval decision before any permitted execution.
• Assess consequential changes. Include scenarios involving deleted or overwritten records, deadline changes, payments, billing and filing. Keep these fictional during the exercise and ask the firm's designated reviewers to define the required restrictions.

The NIST Playbook's Manage section says treatment of documented AI risks is prioritized by impact, likelihood, or available resources, and that risks deemed more serious receive more oversight attention. Your tiers are one way to turn that principle into a working document. Whether the firm may use a particular tool on particular client information is a professional-responsibility and confidentiality question. Take it to your ethics counsel or the relevant bar guidance, not to this article.

Step 2: Build stratified test cases with written authorization

We recommend deliberately selecting cases across the agreed task families and trouble spots, rather than relying only on a convenient recent sample. Document which conditions each case covers. Suggested categories include:

• Action tier: include enough cases in the externally visible and hard-to-reverse tiers that a failure there could actually show up.
• Matter type: include intake, probate administration, trust administration, inherited-property questions, and whatever else you actually handle.
• Messy records: include duplicate contacts, similar names across family members, outdated entries, missing fields, and conflicting dates.
• Ambiguous instructions: include requests that could mean two things, or that name a person who is not the client.
• Hostile or odd inputs: include an inbound email or uploaded document containing text that tries to redirect the agent. The OWASP sheet points readers to a separate LLM Prompt Injection Prevention Cheat Sheet for detailed techniques.
• Out-of-scope requests: include a tool the agent was never meant to have, or an action nobody mapped.

We recommend written authorization from the firm's designated reviewers for the task, test accounts and permitted destinations. Start with approved fictional records in an isolated test environment. Use client material only after the firm has approved its purpose, handling and destination. Obtain permission before interacting with an outside system; a copy of data does not itself authorize access. Record both attempted and completed actions, including deliberate refusals.

Step 3: Define risk-weighted failure criteria before you run anything

We recommend writing the scoring rules first, so results cannot nudge the definition of success. A scheme we would suggest:

1. Classify each failure by consequence, not just by wrongness. For example: cosmetic, correctable internally, externally visible, or harmful to a client or the firm.
2. Name zero-tolerance failures. We suggest examples such as a message addressed to the wrong recipient, confidential matter details surfacing in the wrong context, an irreversible action taken without approval, or an unmapped tool being executed. Under this scheme, one of these in testing fails the workflow regardless of how the rest scored.
3. Report failures by tier and by stratum, not as one overall percentage. We recommend this because a high overall pass rate may coexist with a bad result in exactly the stratum that matters. Our earlier piece on why similar agent scores hide different failures covers the same problem from the scoring side.
4. Record what you did not test. If a channel, matter type, or integration was out of scope, say so in the results.

Step 4: Separate component results from workflow results

We recommend separating retrieval, selection of an action, its parameters, approval and execution when examining results. An evaluation of one component, such as a classifier that screens requests, may describe only that component under the stated conditions. It may not tell you whether the complete intake-to-calendar workflow chooses the permitted record, destination and reviewer. Ask for evidence for each stage and the complete path.

We recommend testing any screening component and also the complete workflow using the approved cases. When reading a vendor or researcher evaluation, check what was measured, the data used and its limits. Keep those reported results separate from the firm's own test record, and record what remains untested.

Step 5: Keep required controls after clean results

OWASP's AI Agent Security Cheat Sheet says the component executing an action must verify that approval belongs to the current actor and exact tool call, remains valid and has not already been consumed.

• Verify the approval scope. Test that the decision applies to the proposed action and authorized person, not just to a general confirmation flag.
• Test changed details. Change the recipient, content or other action parameters in a fictional case and check that the changed action returns for approval.
• Test unavailable checks. Verify that an unknown action or an unavailable required approval check leaves the action unexecuted.
• Keep consequential-action restrictions. Test the controls around deletion, money movement, administrative changes and outward communication. Do not substitute a prompt instruction for the agreed execution checks.

We recommend retaining the required approval and authorization controls after clean results. Treat any proposed expansion of permissions or reduction in review as a separate decision for the firm's designated authority. Document the new scope, risks and testing required before changing that arrangement; do not let a streak of clean runs change it automatically.

Alongside the gates, we recommend a named human owner, a log someone actually reviews, and a way to switch the workflow off quickly.

Step 6: Retest on triggers, with someone independent

We recommend defining retest triggers for changes to models, prompts, tools, memory, retrieval, policies, permissions and connected systems. Ask a reviewer other than the person who configured the workflow to examine failures, expected outcomes and scoring. Keep a record of whether the changed configuration is still within the approved scope, and pause expansion when a required check has not passed.

Build or hire: what to ask for

Whether a vendor, consultant, or internal developer builds the workflow, we suggest asking for the same artifacts:

• Ask for the action inventory with tiers and the authority boundary for each action.
• Ask for the test cases, their strata, and the scoring rules written before the run.
• Ask for results broken out by tier and failure class, including zero-tolerance outcomes and anything untested.
• Ask for a description of where approval is enforced, and how unknown tools are handled.
• Ask for the retest triggers and who owns them.

We recommend requesting the test record and its limitations alongside a demonstration or overall score. Ask the builder to show how the acceptance decision follows from the cases, failures and retained controls.

Recommended pre-launch checklist

Use this proposed checklist to prepare a discussion with the firm's reviewers:

• Check that the test has written authorization and uses permitted accounts and approved material.
• Check that every proposed action has an owner, boundary and consequence assessment.
• Check that the cases cover the agreed task families, ambiguous inputs, sensitive information and stop paths.
• Check that failure categories and required stop conditions were agreed before the run.
• Check that results distinguish attempted, completed, refused and untested actions.
• Check that component scores remain separate from complete-workflow results.
• Check that required human approval and execution authorization remain in place after testing.
• Check that a named owner can pause the workflow and that the pause mechanism was tested.
• Check that retest triggers and a separate reviewer are assigned.
• Request the firm's designated review of confidentiality and professional-responsibility questions before any real use.

Limits of this guide

We recommend adapting this proposed test design to the firm's task and risk decisions. Record the cases, conditions and unknowns beside the result. Confirm separate legal and professional-responsibility questions with the firm's designated advisors. Do not treat this checklist or a clean test run as a certification of a product or workflow.