AI crawler rules that do less than they seem

Two rules for AI crawlers are written often and do less than they seem to.

Blocking Google-Extended doesn't keep a page out of Google's AI answers. It controls whether Gemini models are trained on the page and whether Gemini Apps ground their answers in it; Google says it "does not impact a site's inclusion in Google Search". AI Overviews and AI Mode are part of Search: they read what Googlebot reads, and nosnippet and max-snippet are what limit them.

Blocking a fetcher that acts for a person may not keep it out. When someone asks ChatGPT, Perplexity or Google's agents to read a page, the request comes from a fetcher their makers say robots.txt may not govern: OpenAI writes that for ChatGPT-User "robots.txt rules may not apply", Perplexity that Perplexity-User "generally ignores robots.txt rules", and Google that its user-triggered fetchers "generally ignore robots.txt rules".

Neither is wrong to write; each is worth knowing the limit of, which is why this is a notice.

Sources: Google's common crawlers, Google's user-triggered fetchers, OpenAI's crawlers, Perplexity's crawlers, Meta's crawlers, Amazonbot.

Decide what you meant:

• To keep a page out of AI Overviews and AI Mode, use <meta name="robots" content="nosnippet"> (or a small max-snippet) on it. Google-Extended can stay blocked for training either way.
• To keep people's assistants from reading a page, robots.txt isn't the tool: put the page behind a sign-in, or decide it is public.