How to Block AI Crawlers with robots.txt (GPTBot, ClaudeBot, PerplexityBot & More)

Updated July 25, 2026

🛠️ This guide pairs with our free Robots.txt Tester — Check if Googlebot, GPTBot, ClaudeBot & more can crawl your URLs — bulk testing, no limits.

Every major AI company now runs crawlers that read the public web — some to train models, some to answer user questions in real time. Whether you want to block them all, allow them all, or (most commonly) allow the search-style bots while blocking training bots, the control mechanism is the same 30-year-old file: robots.txt. This guide gives you the exact user agents, ready-to-paste rules, and — the step most people skip — how to verify the rules do what you think.

The AI crawlers that matter in 2026

User agent	Company	What it does
`GPTBot`	OpenAI	Collects training data for GPT models
`OAI-SearchBot`	OpenAI	Indexes pages for ChatGPT’s search answers (with links)
`ChatGPT-User`	OpenAI	Fetches a page live when a ChatGPT user asks about it
`ClaudeBot`	Anthropic	Collects training data for Claude models
`Claude-User`	Anthropic	Live fetches on behalf of Claude users
`Google-Extended`	Google	Opt-out token for Gemini training — does not affect Google Search rankings
`PerplexityBot`	Perplexity	Indexes pages for Perplexity’s answer engine
`CCBot`	Common Crawl	Builds the open web dataset many labs train on
`Applebot-Extended`	Apple	Opt-out token for Apple Intelligence training
`Bytespider`	ByteDance	Training data collection (historically ignores robots.txt more than others)

Two distinctions worth understanding before you block everything:

Training bots vs. answer bots. Blocking GPTBot keeps your content out of future training data. Blocking OAI-SearchBot or PerplexityBot also removes you from AI search answers that link back to you — which for many sites is a growing traffic source. Blocking training while allowing search-style bots is a defensible middle position.
Google-Extended is not Googlebot. Blocking it stops Gemini training use but has zero effect on your search rankings. Never block plain Googlebot unless you want out of Google Search entirely.

Copy-paste recipes

Block all training bots, keep AI search referrals:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

Block every AI bot listed above: add groups for OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot with Disallow: / as well.

Block AI bots from one section only (e.g. your premium content), leave the rest open:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /premium/

The mistakes that silently break robots.txt

Rules in the wrong group. A crawler picks the single most specific User-agent group and ignores all others — groups don’t combine. If you have a User-agent: * group with Disallow: /admin/ and add a User-agent: GPTBot group with Disallow: /, GPTBot only reads its own group (that’s fine here — but if you wanted GPTBot to also respect /admin/, it already ignores your * rules).
Disallow: with an empty value means allow everything. It’s not a typo-safe format.
Case sensitivity in paths. /Premium/ and /premium/ are different paths.
Expecting enforcement. robots.txt is a convention, not a firewall. The major labs state they honour it (OpenAI, Anthropic and Google document this), but it can’t technically stop anyone, and it doesn’t un-train models on content already collected. For hard enforcement, block user agents or ASNs at the CDN level — Cloudflare has one-click AI-bot blocking.

Verify before you ship

A robots.txt typo can deindex half your site or leave the door open to exactly the bot you meant to block — and you won’t get an error message either way. Google retired its Search Console robots.txt tester, so use our free Robots.txt Tester: paste your draft (or fetch your live file), enter a few representative URLs, and run the All major bots matrix. You’ll see Allowed/Blocked per crawler with the exact rule that matched, following the same RFC 9309 longest-match logic Google implements. Edit, re-test, then deploy.

Try the free Robots.txt Tester →