HomeSuman Dhami
AI Search

Should You Block GPTBot? How to Set robots.txt for ChatGPT, Claude and Perplexity

8 min read·By Suman Dhami·
Three small robots above a robots.txt file marked with a cross: AI crawlers and robots.txt.

A robots.txt decision about AI is really three separate decisions: do you want to appear in AI search, do you want your content used to train models, and what about requests a user triggers. The companies publish different answers for each.

What does each AI bot do?

CompanyBotWhat it is forControls
OpenAIOAI-SearchBotSurfaces sites in ChatGPT search resultsAllow it to be shown in ChatGPT search answers. Opted-out sites can still appear as navigational links.
OpenAIGPTBotCrawls content that may be used to train generative AI modelsDisallow it to indicate content should not be used for training.
OpenAIChatGPT-UserVisits a page when a user's question needs itBecause actions are user-initiated, robots.txt rules may not apply. Not used to decide Search inclusion.
AnthropicClaudeBotCollects content that could contribute to model trainingRestricting it signals materials should be excluded from training.
AnthropicClaude-SearchBotIndexes content for search qualityDisabling it prevents indexing for search and may reduce visibility.
AnthropicClaude-UserFetches pages when users ask Claude questionsDisabling it stops retrieval for user queries.
PerplexityPerplexityBotSurfaces and links sites in Perplexity search; not used for foundation model trainingAllow it to appear. Changes can take up to 24 hours.
PerplexityPerplexity-UserUser-triggered fetchGenerally ignores robots.txt.

OpenAI

OAI-SearchBotSearch results in ChatGPT
GPTBotTraining
ChatGPT-UserA person's question; robots.txt may not apply

Anthropic

Claude-SearchBotIndexing for search
ClaudeBotTraining
Claude-UserA person's question

Perplexity

PerplexityBotSearch results; not training
Perplexity-UserA person's request; generally ignores robots.txt

Each bot has its own robots.txt rule. Google-Extended does not affect Google Search.

AI crawlers from OpenAI, Anthropic and Perplexity, and what each one does.

Is it true that blocking GPTBot removes you from ChatGPT search?

Some guides say it does. OpenAI's own crawler page says the opposite: each setting is independent, and a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. If you want training opt-out and search visibility, the two bots take different rules.

A starting robots.txt

This example keeps ChatGPT and Perplexity search eligibility and opts out of OpenAI training. Change the training lines to suit your own policy.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Check what your CDN or firewall does

A robots.txt that allows a bot does not help if your CDN, firewall or host blocks it. Google's guidance says crawling must be allowed in robots.txt and by any CDN or hosting infrastructure, and Perplexity recommends allowing its published IP ranges as well as the user agent. Check your CDN's bot settings, then look at your server logs for the bot names above.

A real example

For Awesome Mate Removals in Perth, I audited robots.txt and added explicit access for 12 AI crawlers, then confirmed the site was eligible for Google AI Overviews. I did this because explicit rules leave no doubt about the policy, not because a rule guarantees citations. OpenAI says placement in search is not guaranteed, and no vendor publishes a rule that access leads to a citation.

What to do next

Decide your training policy first, then set search bots. Re-check after any CDN or plugin change. A technical SEO audit includes a bot access check.

Sources

Frequently Asked Questions

That depends on whether you want your content used for model training. OpenAI says disallowing GPTBot indicates content should not be used for training, and that this is independent of OAI-SearchBot.

Allowing it makes you eligible. OpenAI states that placement is not guaranteed.

Not reliably. OpenAI says that because ChatGPT-User actions are initiated by users, robots.txt rules may not apply. Use OAI-SearchBot to manage search opt-outs.

Blocking OpenAI, Anthropic or Perplexity bots does not change Google rankings. Google-Extended, Google's own token, does not affect Search inclusion either.

Related Services

Get a Free SEO Audit