Rule pack
AI agents rules
Whether AI search engines, AI crawlers and browsing agents can read the page, and what it tells them.
AI crawlers change often and their makers publish little. These rules follow each vendor's own documentation, dated on each rule, and say when something is a measurement rather than a promise.
Indexing
Warning robots.txt keeps AI search engines off the pageagents.ai-search-blockedNotice The page limits what Google's AI answers can useagents.google-ai-snippetNotice Cloudflare challenges automated visitorsagents.bot-challengeNotice AI crawler rules that do less than they seemagents.robots-misreadNotice A noai tag no AI company says it readsagents.noai-metaNotice Bing's AI preferences in a form it is retiringagents.bing-ai-preferencesNotice A broken llms.txtagents.llms-txt