Why you need a robots.txt tester in 2026
Google retired its Search Console robots.txt Tester, and at the same time robots.txt became more important than ever: it is now the primary switch for controlling AI crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), PerplexityBot and CCBot (Common Crawl) all read it. One typo in a Disallow line can deindex a section of your site, or silently leave your content open to crawlers you meant to block. This tester shows you exactly which rule fires for any URL and crawler, before you deploy.
How matching works (RFC 9309, the rules Google follows)
- Group selection: a crawler obeys the group with the most specific matching
User-agentline;*is the fallback. Groups do not combine. - Longest match wins: among the rules in the chosen group, the rule matching the most characters of the URL path decides.
- Ties go to Allow: if an Allow and a Disallow rule match with equal length, the URL is allowed.
- Wildcards:
*matches any characters;$anchors the end of the URL (Disallow: /*.pdf$). - Empty Disallow means "allow everything" — a surprisingly common source of confusion.
Blocking AI crawlers: a quick reference
| Crawler | Company / purpose | Block with |
|---|---|---|
| GPTBot | OpenAI — model training | User-agent: GPTBotDisallow: / |
| OAI-SearchBot | OpenAI — ChatGPT search results | User-agent: OAI-SearchBotDisallow: / |
| ClaudeBot | Anthropic — model training | User-agent: ClaudeBotDisallow: / |
| Google-Extended | Google — Gemini training (does not affect Search) | User-agent: Google-ExtendedDisallow: / |
| PerplexityBot | Perplexity — answer engine | User-agent: PerplexityBotDisallow: / |
| CCBot | Common Crawl — open dataset used for training | User-agent: CCBotDisallow: / |
After editing, paste your draft into the tester above and run your key URLs against All major bots to confirm the policy does what you think it does.
Frequently asked questions
How do I test if a URL is blocked by robots.txt?
Load your robots.txt (fetch it by domain or paste it), enter one or more URLs, pick a crawler, and click Test. Each URL gets an Allowed or Blocked verdict along with the exact rule that matched, following the same longest-match rules Google uses (RFC 9309).
Google removed its robots.txt tester — is this a replacement?
Yes. Google retired its Search Console robots.txt Tester; this tool covers the same workflow: test URLs against a live or edited robots.txt for Googlebot and other crawlers — plus bulk testing and AI crawlers, which Google’s tool never had.
Can I check whether AI bots like GPTBot or ClaudeBot can crawl my site?
Yes — that is a headline feature. Use the “All major bots” mode to see one URL tested against Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot and more at once, so you can verify your AI-crawler policy actually does what you intended.
Which matching rules does this tester follow?
The Robots Exclusion Protocol as standardised in RFC 9309 and implemented by Google: the most specific user-agent group wins, the longest matching path rule wins, Allow beats Disallow on ties, and * and $ wildcards are supported.
Does robots.txt actually stop AI companies from using my content?
It stops compliant crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended all state they respect robots.txt). It is a voluntary protocol though — it does not technically prevent access, and it does not remove content already collected. It remains the standard, machine-readable way to declare your policy.
Can I edit the robots.txt and re-test before deploying?
Yes. The robots.txt box is fully editable — fetch your live file, modify it, and re-run your URL list instantly to preview the effect of a change before you ship it.