As of October 2026, “allow AI” and “block AI” are not precise website policies. The crawler documentation from OpenAI, Anthropic and Google describes separate controls, though not the same set at each provider. In the excerpts we reviewed, OpenAI and Google each document separate search and training controls, and Anthropic additionally documents a separate control for retrieval at a user's direction. We retrieved those pages on October 20, 2026. None of the excerpts carried a publication date, so treat them as current at retrieval and recheck them before you change anything.
For a law firm, the distinction matters twice. Some information should be easy to find: attorney names, practice areas, jurisdictions, offices and dated articles. Other surfaces should never be reachable by a crawler or an agent: client portals, intake submissions, matter files and internal reports. A crawler policy should start from purpose and surface, not from a user-agent list copied from another site.
Two boundaries apply throughout. First, our reading is that allowing a crawler makes retrieval possible but does not mean any provider will cite, recommend or refer an inquiry to the firm; Google's own AI features guidance says AI Overviews often don't trigger. Second, these are provider technical documents, not legal authority, so nothing here is a statement about what a firm's professional or confidentiality obligations require.
Why one switch does not work
OpenAI's crawler overview states that each setting is independent of the others. Its own example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training its generative AI foundation models. OpenAI also says that if both are allowed, it may use the results from one crawl for both uses to avoid duplicate crawling.
Anthropic's help article describes three robots with three jobs: model development, search, and retrieval at a user's direction. On Google's side, the AI features page names robots.txt directives for Googlebot as the control for how a site is crawled for Search, and points to Google-Extended for limiting AI training and grounding in some other Google systems. The common crawlers page describes Google-Extended as a standalone product token. Because the controls differ by provider, our reading is that a blanket rule written for one provider can do something different, or nothing, at another.
Three access purposes to define
We recommend writing the policy around these three purposes. The documentation we reviewed does not describe all three as separate controls at every provider, so check each provider's current page for the purposes it actually documents.
1. Search and answer indexing
We recommend that most firms keep public pages open to search crawlers unless the firm decides otherwise. OpenAI's example ties OAI-SearchBot to appearing in its search results, and it notes that a robots.txt change can take roughly 24 hours to be reflected in search. Anthropic describes Claude-SearchBot as the robot that improves search result quality, and says that disabling it may reduce a site's visibility and accuracy in user search results.
Google's guidance on AI features and your website is more specific. It says robots.txt directives for Googlebot are the control for how a site is crawled for Search. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet, and Google says there are no additional technical requirements. It also says AI Overviews are shown only when its systems judge them additive to classic Search, and that they often don't trigger. Our inference from those statements is that access is a precondition for appearing, not a predictor of whether it happens.
2. Model training
A firm can choose a different rule for training than for search. OpenAI ties GPTBot to training use in its crawler overview. Anthropic says that when a site restricts ClaudeBot, that signals the site's future materials should be excluded from its model training datasets. Google's common crawlers page describes Google-Extended as a token publishers can use to manage whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google adds that Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal.
Our interpretation: In our view, whether to permit training use is a business and risk decision for the firm, not a technical default. That is a judgment, not a provider statement. What the providers document is that these settings are separate from search, so choosing one answer for training does not dictate the answer for search.
3. User-directed retrieval
Anthropic says that when a person asks Claude a question, it may access websites using Claude-User, and that disabling it prevents retrieval of your content in response to a user query and may reduce visibility for user-directed web search. OpenAI says its robots and user agents act either automatically or when triggered by a user request; the excerpt we reviewed names only OAI-SearchBot and GPTBot, so confirm the current user-agent names for user-triggered requests on OpenAI's page rather than relying on a copied list. We did not find a user-directed control described in the Google common-crawlers excerpt we reviewed, so check Google's current documentation for any user-triggered fetchers before assuming how they behave.
The providers point to robots.txt as the preference signal for their crawlers. OpenAI says it uses robots.txt tags to let webmasters manage how their sites and content work with AI, Anthropic says its robots honor robots.txt directives, and Google says its common crawlers always obey robots.txt rules when crawling automatically. Robots.txt is still not an authorization system. Our operating view, which is an inference and not a provider claim, is that sensitive forms and authenticated systems need application-level access control whatever the robots file says.
Map the public evidence surface first
Before editing robots.txt or a firewall rule, list the facts the firm intends to publish and the one page that owns each. For an estate, probate, trusts or inherited-property practice, that usually means lawyer names, jurisdictions, offices, contact channels, practice descriptions, dated articles and clearly bounded service areas. Each owning page should render its facts in HTML a crawler can read and should agree with any structured data on it.
Do not use structured data to assert things a visitor cannot see, and do not expose matter details, intake submissions, client documents, administrative routes or private reports to make the site more machine-readable. Our earlier note on making firm information readable for retrieval while still testing citations covers the page-level side of this problem. Here the focus is on who may fetch what.
Write the policy in layers
- Robots instructions. State provider-specific preferences for public pages that reflect the firm's chosen search and training stance. Anthropic notes that opting out requires editing robots.txt and that you should do so for every subdomain you want to cover. Recheck user-agent names at implementation time.
- Network enforcement. Use the hosting or security layer for protection, not for preference signaling. Anthropic warns that blocking its IP addresses may not work correctly or guarantee an opt-out, because doing so impedes its ability to read your robots.txt file. Never block the robots.txt file itself. Anthropic's article refers to a list of source IP addresses, and Google's common crawlers page says those crawlers generally crawl from published IP ranges, with a documented reverse DNS mask. Our suggestion is to use that published information to check whether a log entry claiming to be a crawler actually is one.
- Application authorization. Require authentication and role checks on portals, reports, admin tools and confidential workflows. Anthropic says its robots respect anti-circumvention technologies such as CAPTCHAs, but a firm should not build confidentiality on any provider's good behavior.
- Preview and snippet controls. Google's AI features page lists nosnippet, data-nosnippet, max-snippet and noindex for limiting what Search shows from a page. Because Google says a page must be eligible to appear with a snippet to be a supporting link in its AI features, these controls can affect that eligibility. Apply them deliberately and page by page, not site-wide by habit.
- Content evidence. Keep public facts dated, attributable and consistent. Remove obsolete practice, staff, office and contact claims from the site instead of expecting a crawler rule to correct them.
Recommended acceptance tests
The following are recommendations for a policy owner, not provider requirements. They are suggested checks, and we are not reporting results from running them on any firm's site.
- Fetch the robots file on every hostname. Confirm each subdomain serves the intended rules, including any intake or marketing subdomain.
- Test the pages you want found. Confirm the key public pages return a successful status, render their core facts in HTML, and carry no accidental noindex or snippet restriction that would conflict with the firm's goals.
- Test the pages you do not want found. Request protected URLs without credentials and confirm they are refused. Do this separately from the robots check, because they test different controls.
- Read the logs by user agent. Note which named crawlers appear, what status codes they receive, and whether the source addresses match each provider's published information.
- Allow for propagation. OpenAI says search systems can take about 24 hours to adjust after a robots.txt update, so date every change and every observation.
- Re-test conventional services. After any network rule, confirm that standard search crawling, accessibility tools and uptime monitoring still work.
A review checklist for the policy owner
- Which public facts should search and answer systems be able to retrieve, and which page owns each?
- Which provider robots serve search, training or user-directed requests according to current official documentation?
- Which surfaces are excluded because they hold client, prospective-client, employee or operational data?
- Do robots instructions, network rules and application authorization agree with each other?
- Does structured data match visible copy and the canonical page?
- Who reviews provider documentation and access logs, and how often?
- Is every visibility observation dated and described as an observation, not a guaranteed outcome?
Keep the evidence types separate in your own records. What a provider says its crawler does is a sourced claim, to be rechecked against the provider's page. What your logs show is first-party evidence. What the firm decides to allow is a policy choice. Whether open access leads to more or better inquiries is a hypothesis that needs the firm's own intake and attribution data before anyone treats it as fact.