Free tool
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot and other crawlers can read your site, straight from your robots.txt. Training and search crawlers are told apart, because blocking them means different things.
- No account
- Public pages only
- Nothing changed on your site
Follows the documentation ofOpenAIAnthropicPerplexityGoogleRFC 9309llmstxt.org
Three kinds of crawler
Blocking one is not the same as blocking another.
Most advice treats “AI bots” as one group. They are not. What you block decides whether you are cited, trained on, or both.
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot
They decide whether an engine can show and cite you. OpenAI says opted-out sites are not shown in ChatGPT search answers. Block one and you leave that engine.
GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended
A choice about training. Blocking them does not stop AI search from citing you. Google-Extended and Applebot-Extended are robots.txt tokens, not separate crawlers, and Google says Google-Extended does not affect inclusion in Google Search.
ChatGPT-User, Claude-User, Perplexity-User
They fetch a page when a person asks an assistant about it. OpenAI and Perplexity say robots.txt rules may not apply to these, so a robots.txt line is not a reliable control. Anthropic says its bots honour robots.txt.
Common mistakes
Four ways sites lock themselves out.
A staging rule left in place
User-agent: * with Disallow: / blocks every crawler. It is the most common reason a live site is invisible.
One block for every “AI” bot
Listing search crawlers beside training crawlers in a single Disallow removes you from AI answers when you only meant to stop training.
Expecting robots.txt to enforce
It is a request that well-behaved crawlers honour. It does not remove pages already indexed, does not secure private content, and some user-triggered fetchers do not follow it.
A group that overrides the one you meant
A crawler follows the most specific group that names it. A named group replaces the * rules for that crawler, so a stray Allow or Disallow can change the result.
Background: AI crawlers and robots.txt, with sources to each vendor’s documentation.
Sources
Checked against the people who write the rules.
Where a vendor documents how its crawler behaves, this tool follows that documentation. These are the pages we rely on, so you can verify any claim yourself.
- OpenAI: crawlers and user agents ↗Purpose of OAI-SearchBot, GPTBot and ChatGPT-User, and how robots.txt applies to each.
- Anthropic: how its crawlers work ↗ClaudeBot, Claude-User and Claude-SearchBot, and how they honour robots.txt.
- Perplexity: PerplexityBot and Perplexity-User ↗What each fetcher is for and why Perplexity-User generally ignores robots.txt.
- Google: common crawlers ↗Googlebot and the Google-Extended token, and what Google-Extended does not affect.
- RFC 9309: Robots Exclusion Protocol ↗The rules we apply: most specific group, longest matching rule, Allow wins a tie.
- llms.txt proposal ↗The proposed format. It is a proposal, not a standard any engine has committed to use.
Vendor documentation changes. Last reviewed against these pages on 3 November 2026. See the full methodology.
Questions
Straight answers.
Should I block GPTBot and ClaudeBot?
That depends on whether you want your content used to train models. It is separate from whether you want to be cited. If you want AI search visibility, keep the search crawlers allowed either way.
Does blocking Google-Extended hurt my Google rankings?
Google documents it as a control for Gemini training and grounding and says it does not affect inclusion in Google Search or act as a ranking signal. Googlebot is the crawler that matters for Search.
What is llms.txt, and do I need it?
A proposed plain-text file that points language models to your best pages. No engine has committed to using it, so treat it as optional. The tool drafts one from your real pages if you want to try it.
The result says “Limited”. What does that mean?
The crawler may read your home page but has Disallow rules for some paths. Check those paths are ones you really want hidden.
Why do some fetchers say robots.txt may not apply?
OpenAI and Perplexity document that fetches triggered by a person’s request may not follow robots.txt. We show the rule that applies and flag those fetchers, so you do not rely on it for them.
How do the crawler roles stay accurate?
They follow each company’s published documentation, which changes. The sources are linked below, and we review them regularly.
More free tools
Keep going.
Access is step one.
Next is whether AI answers actually name and link to you. The app records that question by question, with the answer and the date.