A robots.txt decision about AI is really three separate decisions: do you want to appear in AI search, do you want your content used to train models, and what about requests a user triggers. The companies publish different answers for each.
What does each AI bot do?
| Company | Bot | What it is for | Controls |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search results | Allow it to be shown in ChatGPT search answers. Opted-out sites can still appear as navigational links. |
| OpenAI | GPTBot | Crawls content that may be used to train generative AI models | Disallow it to indicate content should not be used for training. |
| OpenAI | ChatGPT-User | Visits a page when a user's question needs it | Because actions are user-initiated, robots.txt rules may not apply. Not used to decide Search inclusion. |
| Anthropic | ClaudeBot | Collects content that could contribute to model training | Restricting it signals materials should be excluded from training. |
| Anthropic | Claude-SearchBot | Indexes content for search quality | Disabling it prevents indexing for search and may reduce visibility. |
| Anthropic | Claude-User | Fetches pages when users ask Claude questions | Disabling it stops retrieval for user queries. |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search; not used for foundation model training | Allow it to appear. Changes can take up to 24 hours. |
| Perplexity | Perplexity-User | User-triggered fetch | Generally ignores robots.txt. |
OpenAI
Anthropic
Perplexity
Each bot has its own robots.txt rule. Google-Extended does not affect Google Search.
Is it true that blocking GPTBot removes you from ChatGPT search?
Some guides say it does. OpenAI's own crawler page says the opposite: each setting is independent, and a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. If you want training opt-out and search visibility, the two bots take different rules.
A starting robots.txt
This example keeps ChatGPT and Perplexity search eligibility and opts out of OpenAI training. Change the training lines to suit your own policy.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlCheck what your CDN or firewall does
A robots.txt that allows a bot does not help if your CDN, firewall or host blocks it. Google's guidance says crawling must be allowed in robots.txt and by any CDN or hosting infrastructure, and Perplexity recommends allowing its published IP ranges as well as the user agent. Check your CDN's bot settings, then look at your server logs for the bot names above.
A real example
For Awesome Mate Removals in Perth, I audited robots.txt and added explicit access for 12 AI crawlers, then confirmed the site was eligible for Google AI Overviews. I did this because explicit rules leave no doubt about the policy, not because a rule guarantees citations. OpenAI says placement in search is not guaranteed, and no vendor publishes a rule that access leads to a citation.
What to do next
Decide your training policy first, then set search bots. Re-check after any CDN or plugin change. A technical SEO audit includes a bot access check.
