Technical SEO
robots.txt Generator
Pick which crawlers to allow or block, add your sitemap, and copy a ready to publish robots.txt. The write side of the AI Crawler Checker.
AI crawlers
Left as allowed by default. Toggle a crawler to block it from your whole site. Blocking a crawler means that engine can never cite you.
- GPTBotChatGPT training
- OAI-SearchBotChatGPT search
- ChatGPT-UserChatGPT browsing
- ClaudeBotClaude
- PerplexityBotPerplexity
- Google-ExtendedGoogle AI
- Applebot-ExtendedApple Intelligence
- CCBotCommon Crawl
- meta-externalagentMeta AI
User-agent: *
Disallow:
Save this as robots.txt at the root of your domain. Check the result with the AI Crawler Checker.
The one rule that decides what your robots.txt does
A crawler obeys the single most specific group that names it and ignores every other group in the file, the wildcard included. Rules are never merged across groups. Almost every robots.txt mistake comes from expecting the opposite, and the file gives you no error and no warning when it happens.
Two consequences follow. A named group with an empty rule set inherits nothing, so it means allowed everywhere, whatever the wildcard says below it. And any path you disallow under the wildcard stops applying to a crawler the moment that crawler gets a group of its own, which means every private path has to be repeated inside it.
The generator writes the wildcard group with your disallowed paths first, one group per crawler you blocked, then the Sitemap line. Sitemap is file-level rather than group-level, so it applies to every crawler regardless of where it sits.
What the generator writes
The first file allows every AI crawler and keeps two paths private. The second blocks one crawler and keeps the rest, and shows the repetition the grouping rule forces.
User-agent: *
Disallow: /admin
Disallow: /cart
Sitemap: https://example.com/sitemap.xml
User-agent: *
Disallow: /admin
Disallow: /cart
User-agent: CCBot
Disallow: /
# GPTBot reads this group and nothing else, so /admin has to be
# repeated here. Without these two lines it would be allowed to
# crawl /admin, because the wildcard group above no longer applies.
User-agent: GPTBot
Disallow: /admin
Disallow: /cart
Sitemap: https://example.com/sitemap.xml
Watch the single character that reverses the file: Disallow: with nothing after it allows everything, and Disallow: / blocks the whole site. Engines also cache robots.txt for up to a day, so give a change 24 hours before you conclude it did not work.
robots.txt questions
Does blocking a crawler remove content an engine already has?
No. A disallow stops future fetches and nothing else. Content already in an index stays until it is refreshed out, and a URL blocked in robots.txt can still be listed from links pointing at it, without its content ever being read. To remove a page, allow the crawl and serve a noindex, or use the engine's own removal process.
Do AI crawlers obey robots.txt?
The named ones publish that they do, and the major crawlers behave. robots.txt is a request rather than an enforcement mechanism, so a scraper with no interest in the convention will ignore it. If you need enforcement, that is a job for authentication, rate limits, or a firewall rule, not for a text file.
Should I block CCBot?
CCBot collects pages for Common Crawl, an open dataset used by many downstream projects including model training. It is not a live citation path, so blocking it costs little visibility today. It also does nothing to stop the retrieval crawlers that decide whether you get cited, so treat the two decisions separately.
Is robots.txt a way to hide a private section?
The opposite. The file is public and is one of the first URLs anyone fetches, so listing a path there advertises it to anyone curious. Anything genuinely private needs authentication. Use robots.txt for pages that are public but not worth crawling, such as faceted search and cart URLs.
Previews the Technical Agent
Publish this once, or let Rankfastt watch your crawl access and warn you the moment a change locks an engine out.