Methodology
How every check is measured.
No black box. For each check: what we do, what it is based on, and where our own judgement comes in.
| Check | What we do | Based on |
|---|---|---|
| AI crawler access | We fetch /robots.txt and apply RFC 9309: each crawler follows the most specific group naming it, otherwise the * group; the longest matching rule wins and Allow wins a tie. A missing robots.txt allows everything. We evaluate the home path (/). | RFC 9309; vendor crawler documentation |
| Crawler roles | Each crawler is labelled search, training or visitor-triggered from its vendor’s own documentation. User-triggered fetchers that vendors say may ignore robots.txt are flagged, never presented as controllable. | OpenAI, Anthropic, Perplexity, Google |
| llms.txt | We request /llms.txt and check that it exists, is not an HTML page, begins with a title line and contains links. We state plainly that it is a proposal and no engine promises to use it. | llmstxt.org |
| Text in the raw HTML | We request the page once without running JavaScript and count visible words. Under 50 words is flagged as a failure, under 200 or an empty app shell as a warning. These thresholds are ours and are labelled as such. | Our own thresholds |
| Opening paragraph | We take the first paragraph of 8 or more words under the main heading, and flag it as long above 80 words or as filler when it opens with a greeting or scene-setting phrase. A heuristic; no engine publishes a rule. | Our own heuristic |
| Structured data | We extract every JSON-LD block from the raw HTML, parse it, and check common types for the properties Google and schema.org document, ISO 8601 dates, absolute URLs and placeholder text. It is a structural check, not a promise of a rich result. | Schema.org; Google structured data documentation |
| Technical and trust | Direct requests for robots.txt, sitemap, the http:// and www variants, a made-up address, plus a real TLS connection for the certificate. Response time is timed once from our server and treated as a hint. | Direct measurement |
What we never do
Five promises.
- We do not invent a score. Results are counts of checks passed, with the evidence for each.
- We do not run your JavaScript, log in, or change anything on your site.
- We do not claim that fixing a check will earn a ranking or an AI citation.
- We do not use paid AI services to produce these results.
- We do not store pasted code, and we keep a scan result for at most 24 hours.
Sources
Where the rules come from.
- OpenAI: crawlers and user agents ↗Purpose of OAI-SearchBot, GPTBot and ChatGPT-User, and how robots.txt applies to each.
- Anthropic: how its crawlers work ↗ClaudeBot, Claude-User and Claude-SearchBot, and how they honour robots.txt.
- Perplexity: PerplexityBot and Perplexity-User ↗What each fetcher is for and why Perplexity-User generally ignores robots.txt.
- Google: common crawlers ↗Googlebot and the Google-Extended token, and what Google-Extended does not affect.
- Google: AI features and your website ↗How AI Overviews and AI Mode draw on the Search index.
- RFC 9309: Robots Exclusion Protocol ↗The rules we apply: most specific group, longest matching rule, Allow wins a tie.
- llms.txt proposal ↗The proposed format. It is a proposal, not a standard any engine has committed to use.
- Schema.org ↗The vocabulary: types and the properties each one accepts.
- Google: structured data introduction ↗Which structured data Google can use and the guidelines for it.
- Google Rich Results Test ↗Google’s own check of whether a page is eligible for a rich result.
Last reviewed 3 November 2026. Vendor documentation changes; if you spot something out of date, tell us at [email protected].