AI crawler directory
The 18 AI crawlers that decide whether assistants can read and cite you — who runs them, what they do, and whether to let them in.
AI crawlers fall into three groups. Search crawlers index pages so an assistant can cite them; user-requested fetchers read a page when someone asks about it; training crawlers gather data for future models. To appear in AI answers, allow the first two groups. Blocking training crawlers is a separate, legitimate choice.
| User agent | Operator | Product | Role | Recommendation |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search | AI search | Allow to be cited |
ChatGPT-User | OpenAI | ChatGPT browsing | User-requested | Allow to be cited |
GPTBot | OpenAI | Model training | Training | Your choice |
ClaudeBot | Anthropic | Model training | Training | Your choice |
Claude-SearchBot | Anthropic | Claude search | AI search | Allow to be cited |
Claude-User | Anthropic | Claude browsing | User-requested | Allow to be cited |
PerplexityBot | Perplexity | Perplexity answers | AI search | Allow to be cited |
Perplexity-User | Perplexity | Perplexity browsing | User-requested | Allow to be cited |
Google-Extended | Gemini grounding/training | Training | Your choice | |
Googlebot | Google Search & AI Overviews | AI search | Allow to be cited | |
Bingbot | Microsoft | Bing & Copilot | AI search | Allow to be cited |
Applebot | Apple | Siri / Spotlight | AI search | Allow to be cited |
Applebot-Extended | Apple | Apple Intelligence training | Training | Your choice |
Meta-ExternalAgent | Meta | Meta AI | Training | Your choice |
Amazonbot | Amazon | Alexa / Rufus | AI search | Allow to be cited |
DuckAssistBot | DuckDuckGo | DuckAssist answers | AI search | Allow to be cited |
MistralAI-User | Mistral | Le Chat browsing | User-requested | Allow to be cited |
CCBot | Common Crawl | Open dataset used by many LLMs | Training | Your choice |
robots.txt that allows AI answers but opts out of training
User-agent: *
Allow: /
# Opt out of model training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlWant different choices per bot? Use the free robots.txt generator, then check the result with a free audit — firewalls and CDNs sometimes block bots that robots.txt allows.
Source: each operator’s published crawler documentation, compiled by AIRankSurge Research. Free to reuse with attribution (CC BY 4.0).
AI crawler questions
Straight answers. If yours isn’t here, email hello@airanksurge.com.
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot collects training data. ChatGPT search results are fetched by OAI-SearchBot and ChatGPT-User, which are separate user agents. Blocking GPTBot while allowing those two keeps you eligible to be cited in ChatGPT answers.
Which AI crawlers should a small business allow?
Allow every search and user-requested crawler — they fetch pages to answer questions and cite sources. Whether to allow training crawlers is a business policy choice; blocking them does not remove you from AI search answers.
Is Google-Extended needed for AI Overviews?
No. Google AI Overviews use Googlebot, the same crawler as Google Search. Google-Extended only controls whether your content is used for Gemini model training and grounding, so blocking it does not remove you from Google Search or AI Overviews.