Technical SEO
Check which AI crawlers your robots.txt blocks
Enter your domain. We read your robots.txt and show which AI crawlers you allow and which you block, so you can stop accidentally hiding from the engines you want to be cited by.
Which AI crawlers this tool checks, and what each one controls
Blocking an AI crawler is close to the only way to guarantee you are never cited. Most sites that block one did not decide to: it arrives through a plugin default, or a line copied out of another site's robots.txt years ago and never read since.
The check above reads your live robots.txt and evaluates it the way each crawler would, group by group. These are the user agents it tests.
| Crawler | Engine | What it controls |
|---|---|---|
| GPTBot | ChatGPT | Collects pages used to train OpenAI's models. Blocking it keeps you out of what ChatGPT knows without searching. |
| OAI-SearchBot | ChatGPT search | Builds the index ChatGPT search picks from. This is the one that decides whether you can be cited in a live answer. |
| ChatGPT-User | ChatGPT browsing | Fetches a single page in real time when a user's question sends ChatGPT out to the open web. |
| ClaudeBot | Claude | Anthropic's crawler for training Claude. |
| anthropic-ai | Claude | An older Anthropic user agent. Still named in plenty of robots.txt files, so it is worth checking. |
| PerplexityBot | Perplexity | Indexes pages so Perplexity can quote them with a numbered source link back to you. |
| Google-Extended | Google AI Overviews | Governs Gemini and AI Overviews only. It carries no ranking signal, so blocking it costs AI citations and changes nothing in classic search. |
| Applebot-Extended | Apple Intelligence | Decides whether Apple may use your content for Apple Intelligence. Separate from Applebot, which powers Siri and Spotlight. |
| CCBot | Common Crawl (many models) | Feeds the open dataset a long list of models train on, so blocking it reaches further than any single vendor. |
| meta-externalagent | Meta AI | Meta's crawler for the AI features built into its apps. |
Google-Extended is the one worth singling out. It governs Gemini and AI Overviews and nothing else. Blocking it has no effect on how Googlebot crawls you or where you rank in classic results, which makes it the safest to allow if you are cautious about the rest.
How to fix a blocked crawler
A block usually looks like one of these two. A named group with a root Disallow, or a wildcard group that quietly catches every crawler you never listed.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
One detail decides whether the second block works: a crawler obeys the single most specific group that names it and ignores every other group, the wildcard included. Rules are never merged across groups. So a named group with Allow: / overrides User-agent: * Disallow: / for that bot alone, and a named group that exists but contains no rules inherits nothing from the wildcard. Deploy, then re-run the check. Engines cache robots.txt for up to 24 hours, so they take a day to notice.
Crawler access questions
Does unblocking GPTBot get me into ChatGPT right away?
No. Access is the precondition, not the result. GPTBot has to crawl the page, the page has to be indexed, and the answer it supports has to beat whatever the model cites today. Unblocking removes a hard ceiling on your visibility. It does not move you up on its own.
Will allowing Google-Extended change my Google rankings?
No. Google-Extended is a separate control from Googlebot and carries no ranking signal in either direction. It decides one thing: whether your content can be used in Gemini and in AI Overviews. Blocking it removes you from AI Overview citations and leaves your blue-link position untouched.
Every crawler is allowed and I am still not cited. What now?
Access was never your bottleneck, so the page itself is the problem. The usual causes are an answer buried inside long prose with no question-shaped heading, missing schema, and a claim no other site on the web repeats. Run the AI Readiness Grader on the page and work through what it returns.
Should I block AI crawlers to protect my content?
That is a real tradeoff, and the answer depends on how you make money. A publisher selling subscriptions can lose more to summarisation than it gains in referrals. A B2B company whose buyers now open ChatGPT before Google gives up a channel by blocking. If your revenue depends on being found rather than on the page view itself, blocking costs you more than it protects.
Previews the Technical Agent
A blocked crawler means you can never be cited, no matter how good your content is. Rankfastt watches your crawl access continuously and warns you the moment it changes.