robots.txt keeps AI search engines off the page

ChatGPT search, Claude, Perplexity and DuckDuckGo's AI-assisted answers each build the index they cite from with a crawler of their own, and each obeys robots.txt. A page robots.txt keeps them from can't be quoted or linked in those answers, however well it does in Google. OpenAI says a site that blocks OAI-SearchBot "will not be shown in ChatGPT search answers".

Search and training are separate crawlers, and blocking one does nothing to the other. OpenAI's GPTBot reads for model training and its OAI-SearchBot for ChatGPT search; Anthropic splits ClaudeBot from Claude-SearchBot, and Perplexity says PerplexityBot "is not used to crawl content for AI foundation models". So a site can keep its words out of training and still be found in AI search.

We judge robots.txt for each crawler as for any other: the group that names it, else User-agent: *. A rule for every crawler, written before these existed, is the usual way a site blocks them without meaning to. We say nothing when Google can't read the page either: that is a bigger problem, with a rule of its own.

Sources: OpenAI's crawlers, Anthropic's crawlers, Perplexity's crawlers, DuckAssistBot.

Open /robots.txt and change the group we name so the search crawlers may read the page. To keep model training out while letting AI search in, name them apart:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
Allow: /

If keeping AI search out is deliberate, switch this rule off for the site.