Which AI crawlers should you allow? Allow the crawlers that fetch pages for products you want to be found in. Decide separately whether you want your content used to train models. GPTBot is not the same thing as the bot that fetches a page for a live answer. Blocking every AI user agent because a host turned on a default will also block the surfaces you are trying to measure.
This is a policy choice with a technical file attached. It is not a growth hack.
Split the decision in two
| Question | Examples people lump together | What allowing means |
|---|---|---|
| May this bot use our pages for model training? | GPTBot, Google-Extended, often ClaudeBot in training discussions | You are okay with that use, per their docs |
| May a search or answer product fetch our pages when someone asks? | OAI-SearchBot for ChatGPT search, Googlebot for Google, PerplexityBot for Perplexity | You are willing to be retrieved. You are not guaranteed a citation |
| Are we only trying to cut junk traffic? | Random AI scrapers with no published docs | Handle from logs. Do not copy a blocklist you do not understand |
OpenAI's own notes distinguish training crawlers from search crawlers. Check their current crawler documentation before you paste a robots.txt snippet from a 2024 blog. Names change.
Google's generative AI guide still starts from ordinary SEO: crawlable pages, clear content. It does not ask you to add a special AI file. llms.txt is optional and is not this decision.
A sane default for a SaaS marketing site
If you want a chance to be cited when someone asks ChatGPT, Perplexity, or Google:
- Do not block Googlebot.
- Allow the search-oriented OpenAI crawler if ChatGPT search matters to you. Read OpenAI's current name for it. Do not assume the 2024 name is still the one.
- Allow PerplexityBot if Perplexity citations matter.
- Make an explicit training decision for GPTBot, Google-Extended, and ClaudeBot. Write it down so the next person does not "clean up" robots.txt and undo it.
- Fetch your live robots.txt after any CDN toggle. Some hosts add AI blocks as a default. A toggle in a dashboard can ship a disallow you never typed.
Bloomiro can read a site's robots.txt as part of understanding crawler access. The checklist beside that is crawler readiness before AI visibility. Allowing a bot and publishing an empty page still fails. The page has to answer something.
What this will not fix
If Perplexity cites the same three competitors, robots.txt is probably not why. Those competitors are on pages the model already quotes. Read the source list on each run, then use how to track Perplexity brand mentions and why ChatGPT recommends competitors for the same pattern on other engines.
If you are invisible in Search Console, fix indexing and the page. Crawler policy for Claude will not create Google impressions.
Larger SEO sites already publish long versions of this topic. Rank Math and Ahrefs-shaped courses show up because they teach the file format. Use them to confirm bot names. Use your own robots.txt and a prompt log to decide. The log is how you track mentions.
