Proposed October 2026 access-policy review: we recommend writing separate decisions for training collection, search discovery, user-directed retrieval and actions on the firm's website. Start with what a visitor may read and what a requester may do. Keep crawler preferences separate from the checks that permit access to private records or carry out an action.
We recommend using the provider documentation below to identify the purposes each provider describes. Record the source and date beside the firm's decision, then test the actual configuration. Treat those descriptions as provider documentation, not as evidence of measured behavior or business results at your firm.
Four types of automated visit
1. Training collection
OpenAI's crawler overview describes its settings as independent. It says a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training its generative AI foundation models.
2. Search indexing
Record a separate search preference for each provider you decide about. Check its current documentation for the relevant crawler names before changing the configuration. Avoid carrying a training decision into the search row without an explicit review.
3. User-directed retrieval
Anthropic's crawler documentation describes ClaudeBot as collecting content that could contribute to training. It says Claude-User may access websites when individuals ask Claude questions, and describes Claude-SearchBot as navigating the web to improve search result quality.
4. Authenticated or state-changing actions
We recommend assessing form submission, consultation requests, document upload, payment-related steps and portal access as distinct actions. Define the requester, permitted action and required checks for each. Do not use a crawler preference or a user-directed request as permission to disclose client information or perform a transaction.
Record the firm's separate decisions
We recommend recording training and search choices separately, including who approved each choice and which hostnames it applies to. Keep the decision specific to the provider and purpose under review.
Check the provider's current configuration instructions before applying a decision to another hostname or service. Record which settings the firm controls and which require a hosting or software vendor to act. Ask the responsible owner to resolve any unanswered scope question.
We recommend having the firm's designated owner decide its published-content preferences. Record the reason and refer intellectual-property or confidentiality questions to the firm's advisors. Keep that decision separate from permissions for client records and website actions.
Do not treat a crawling preference as a promise of visibility, citation or an inquiry. We recommend measuring the observation you actually need and keeping it separate from the access decision.
Classify what lives on your site
Recommendation: open your site in a private browser window and list every page and action by what it holds.
- List public content. Identify the service pages, attorney biographies, articles and FAQs intended for public reading. Record the crawler preference decisions relevant to those pages.
- List private information. Identify portal records, uploads, matter status and questionnaires that require restricted access. Ask the implementer to show the permission checks before any information is returned.
- List public-form actions. Record what a contact, consultation or scheduling submission may create, change or notify. Review the requested fields and downstream effects with the firm's designated owner.
- List payment-related actions. Record which actors may request or use payment functions and the checks those actions require. Include any vendor-hosted steps in the review.
For each row, write who may read it, who may act on it, and the consequences to consider if the decision is wrong. Label any estimated impact as an estimate and record who reviewed it. Do not diagnose lost inquiries or revenue from a public traffic observation alone.
Put each rule where it can be enforced
Crawl preferences: robots.txt
We recommend keeping separate entries for the named training, search and user-retrieval crawlers the firm has decided about. Follow the current provider instructions for the relevant hostname and record the resulting configuration. Keep private information out of published preference files, and verify its application permissions independently.
Decide each category according to the firm's own goals and permitted content. Do not copy another organization's block as a complete policy. Ask which purpose the rule addresses and which website actions still need separate controls.
Authorization: the server decides every time
The OWASP Authorization Cheat Sheet distinguishes authentication, which verifies identity, from authorization, which checks whether a requested action is approved for an entity. It recommends denying access by default and validating permissions on every request, regardless of how that request was initiated.
We recommend testing the firm's intended portal and document-access boundaries before any data is returned. Use permitted test accounts to check their roles and access to individual fictional records. Do not grant additional access because a request names an AI provider. For the related design principle, see placing refusal controls outside the model. Apply the same separation between a claimed identity and an enforced permission to inbound traffic.
Forms: separate receiving a request from accepting it
Use the following proposed form-review questions with the firm's responsible reviewers:
- Review the fields needed for an initial inquiry and define where additional information should be collected. Ask the firm's designated advisor to decide any confidentiality or professional-responsibility requirements.
- Define which submissions may create an inquiry record and which changes require further review. Test the record type, notifications and other effects instead of assuming that receiving a form authorizes a consequential next step.
- Consider contact-method confirmation as one possible control for the firm to evaluate. Do not use it as proof that a human acted or that the requester has authority over client information. Require the appropriate identity and permission checks for the proposed private action.
- Set abuse controls and submission limits for the actual form and traffic conditions. Test the intended refusal behavior and how permitted requests proceed. Avoid using a fixed request count as a general acceptance threshold.
Client data in forms and uploads
We recommend documenting how submitted information is stored, who may inspect it and which retrieval or sharing actions are permitted. Test protected-file access using fictional material and approved accounts. Have the firm's responsible reviewers resolve questions about collection, access and retention before real use.
Vendor-hosted pieces
Ask each scheduling, intake, payment and portal vendor which controls the firm can configure and which the vendor enforces. Request an explanation of automated submissions, identity, permissions and activity records for the relevant task. Record the answer and any unresolved question.
Acceptance checks, as recommendations
We recommend defining acceptance checks for each action before testing. Use approved fictional records and test accounts in an environment authorized for the exercise. Obtain permission for any outside service involved. Do not treat a fixed 100-request run as a safety certificate or permission to remove required controls.
- Check crawler preferences. Retrieve the configuration for each relevant hostname and compare it with the firm's recorded decisions. Keep observations of crawler traffic separate from tests of application permissions.
- Check public pages. Confirm that the intended public material returns the expected response for a permitted visitor. Keep the private-record checks separate, and do not weaken them to make a public-page test pass.
- Check private records. In the approved test environment, request a protected fictional record without the required session. Then use an authorized test account to request a fictional record assigned to another account. Verify the expected denials and activity record without using real client matters.
- Check inquiry submissions. Submit fictional entries through permitted test paths. Verify the intended record, review step and test-only notifications. Confirm that no real person is contacted and that no unintended matter or consequential action is created.
- Check scheduling. Use an authorized test calendar to verify what a request, confirmation, cancellation and expiry actually do. Compare any slot-holding or release behavior with the owner's agreed expectations.
- Check payment-related steps. Use only an authorized test mode. Verify which identity and action may reach the next step, and confirm that the exercise creates no real charge or unintended transaction.
- Check the activity record. Verify the approved action, result, time and relevant actor or target reference. Keep credentials and unnecessary client content out, and obtain the firm's review of access and retention.
When a check fails, correct the relevant rule and rerun the agreed checks for that action and its connected paths. Record the version, outcome and unresolved limits before deciding whether the proposed scope may proceed.
Common pitfalls
- Review training and search decisions independently rather than using a training crawler as the search policy.
- Record a deliberate decision for each purpose instead of copying a blanket allow or block.
- Check the provider's current opt-out instructions and record the applicable hostname and evidence.
- Ask the implementer to demonstrate protection of private information independently of the crawler-preference configuration.
- Define the allowed inquiry-record workflow separately from changes or disclosures that require additional authority.
- Name a policy owner and a review date. Set review triggers for changes to provider documentation, website actions, permissions or vendors.
A one-page version to draft this week
Write one policy row per content type or action. Record who may read, who may act, where permission is enforced and which approved check tests the intended behavior. Add the owner, source date and unresolved questions. Keep the record current when the configuration or scope changes.