To be cited by AI assistants, allow their search and user-request crawlers — such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot — in robots.txt. Blocking training crawlers such as GPTBot, ClaudeBot, Google-Extended or CCBot is a separate policy choice that does not remove you from AI search answers.
Two kinds of AI crawler
Search and user crawlers fetch pages to answer a question right now and usually cite what they read. Blocking them means the assistant cannot quote you. Training crawlers collect content to train future models. Some businesses block them for licensing or privacy reasons; that is a legitimate choice with a smaller effect on day-to-day visibility. See the full AI crawler directory.
A safe default robots.txt
User-agent: *
Allow: /
Disallow: /wp-admin/
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlBecause AI search bots fall under User-agent: *, they are allowed. Avoid creating a separate group for Googlebot or Bingbot unless you mean to — a named group replaces the * rules for that bot entirely. Or build one with the free robots.txt generator.
Common mistakes
- A leftover
Disallow: /from a staging site. - Security plugins or CDNs that block unfamiliar bots with a firewall rule, even when robots.txt allows them.
- Blocking PerplexityBot or OAI-SearchBot while trying to block “AI” generally.
- Forgetting the Sitemap line.
Check yours
AIRankSurge checks your robots.txt against 18 AI crawlers, separates search bots from training bots, and generates a corrected file that keeps your existing rules.
Sources
- OpenAI, Overview of OpenAI crawlers
- Google Search Central, Google crawlers and fetchers
- Google Search Central, Introduction to robots.txt
- IETF, RFC 9309: Robots Exclusion Protocol