It depends what you want the crawler to do. Some AI crawlers support search or user-requested retrieval; others may collect public content for model development. Blocking them can reduce those forms of access, but the trade-off differs by bot. Do not use one blanket rule until you understand what each crawler is for.
This is where the phrase "AI crawler" becomes too broad.
Two bots may both belong to an AI company.
They may have completely different purposes.
One might help a user find your page.
Another might collect public material that could contribute to future model training.
Those are separate decisions.
Start with the business objective
Ask:
Do I want people to discover this public content through AI search?
Do I want AI assistants to retrieve it when a user asks?
Do I want to permit its potential use in model training?
Those answers do not have to be identical.
Modern platforms increasingly expose separate controls because publishers want that choice.
OpenAI separates Search from training controls
OpenAI's current publisher guidance gives a clear example.
OAI-SearchBot is used for ChatGPT Search discovery.
OpenAI says publishers who want their content included in ChatGPT Search summaries and snippets should make sure that bot is not blocked.
For potential model training, OpenAI points publishers toward GPTBot controls.
So a publisher may decide:
allow search discovery;
restrict potential training.
That is a more precise policy than:
block all AI.
Anthropic also documents separate crawler roles
Anthropic currently documents three relevant robots.
ClaudeBot relates to collecting web content that may contribute to model development.
Claude-SearchBot supports search-result quality.
Claude-User supports retrieval when a Claude user asks for content.
Anthropic says blocking its search/retrieval bots may reduce the site's visibility or ability to be retrieved in those contexts.
Again, purpose matters.
robots.txt is the normal control surface
Major crawlers commonly honour robots.txt.
That file allows a website owner to specify user-agent rules.
A simplified concept looks like:
allow a search crawler;
disallow a training crawler.
The exact syntax should be implemented carefully.
A malformed robots.txt can create unexpected problems.
Do not copy a random blocklist from social media and upload it to a live business site without checking what it does.
Blocking crawlers can have a visibility cost
If an AI search crawler cannot access the page, it may not be able to read the content for search summaries or citations.
OpenAI says blocking OAI-SearchBot affects inclusion in ChatGPT Search summaries/snippets.
Anthropic says blocking Claude-SearchBot or Claude-User can reduce visibility or retrieval in relevant user experiences.
So the decision is not purely philosophical.
It can affect discoverability.
Allowing search crawling does not guarantee traffic
This is equally important.
Allowing the crawler only makes access possible.
It does not guarantee:
indexing;
citation;
ranking;
referral traffic;
recommendation.
The content still needs to be relevant and useful.
Crawler access is eligibility, not success.
Training is a different commercial question
Some businesses are comfortable with public web content potentially contributing to model development.
Others are not.
Publishers, photographers, media organisations and specialist knowledge businesses may have stronger concerns.
A local plumber's public service page presents a different commercial issue from a publisher's paid archive.
There is no one correct policy for every website.
What about content behind login?
If content is genuinely private, sensitive or paid, do not rely only on robots.txt as access control.
robots.txt is a crawler instruction.
It is not authentication.
Private information should be protected by real access controls.
That principle existed long before AI crawlers.
Can I block only parts of the site?
Crawler rules can be path-specific where supported.
A business might allow public Guides but restrict a particular section.
Whether that is useful depends on the site.
For most ordinary small-business sites, simpler rules are easier to maintain.
Complex crawler policies create more opportunities for mistakes.
What about images?
Crawler and model-use controls for images can differ by platform and bot.
If image rights are commercially important, check current documentation carefully rather than assuming a text crawler rule covers every product and use.
This area evolves quickly.
Cloudflare can add another layer
If a site uses Cloudflare, bot-management or AI-crawl-control products can affect what reaches the origin.
That can be useful.
It also means you need to make sure infrastructure and robots.txt express the same intention.
There is little point allowing a crawler in robots.txt while a firewall blocks every request it makes.
This is exactly the kind of technical mismatch that can create invisible discovery problems.
What I'd do
If a small business wants to be discoverable in AI-powered search, my default would be to allow the documented search crawler unless there is a specific reason not to. I would make the training decision separately rather than treating every AI bot as though it serves the same purpose.
Should a small business allow AI search crawlers?
For an ordinary public small-business website, my default view would be:
if you want discovery through AI search, allow the relevant search/retrieval crawlers;
make a separate conscious decision about model-training crawlers;
protect genuinely private content with real access controls;
review the policy periodically.
That is more nuanced than "block AI".
It is also more commercially useful.
Gavin's take
Crawler controls should be a business decision expressed technically, not a mysterious checkbox. The useful question is what you are allowing and why. I would rather document that clearly than install a blanket “block AI” rule that accidentally removes a visibility channel the business wanted.
Gavin’s perspective
Built by Gavin's approach
I want the crawler policy to match the client's goals.
If the whole point of the Guides library is broader discoverability, it would be strange to accidentally block the search crawlers we want reading it.
At the same time, search discovery and model training do not need to be treated as one inseparable choice.
The website owner should know what is being allowed.
The bottom line
Do not decide whether to allow "AI crawlers" as one group.
Identify the crawler.
Identify its purpose.
Then decide.
Search/retrieval bots can support visibility.
Training bots raise a different content-use question.
robots.txt gives publishers a practical control mechanism, but it should be implemented carefully and reviewed as platforms change.
Unsure what your current robots.txt is actually allowing?
I can help you inspect the public crawler setup before changing rules that may affect Google, ChatGPT or other discovery channels.
---