Robots.txt: The Complete Guide (Syntax, Rules & Costly Mistakes)

Updated July 25, 2026

🛠️ This guide pairs with our free Robots.txt Tester — Check if Googlebot, GPTBot, ClaudeBot & more can crawl your URLs — bulk testing, no limits.

Robots.txt is thirty years old, one page long, and still manages to take down organic traffic at companies with entire SEO teams. That’s because its failure modes are silent: nothing errors, nothing looks broken — pages just quietly stop being crawled. This guide covers what the file actually controls, the matching rules crawlers really apply (RFC 9309), and the handful of mistakes behind most incidents.

What robots.txt does — and doesn’t — control

It controls crawling: whether a compliant bot fetches your URLs. It does not control:

• Indexing. A URL blocked by robots.txt can still appear in Google if other sites link to it — as a bare “no information available” listing. To keep a page out of the index, use noindex (meta tag or X-Robots-Tag header) and let it be crawled so the directive is seen. Blocking + noindex is a contradiction: the crawler can’t read the noindex it’s blocked from fetching.
• Access. It’s a convention, not security. Anything sensitive behind a Disallow: line is one URL-guess away — and you’ve helpfully listed the paths.
• Non-compliant bots. Majors comply (Google, Bing, the documented AI crawlers); scrapers ignore it freely.

Syntax that actually matters

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xml

• Groups. One or more User-agent lines followed by rules. A crawler picks the single most specific group matching its name and ignores all others — groups do not combine. If GPTBot has its own group, it never reads your * rules. This is the most misunderstood behavior in the file.
• Paths are prefixes. Disallow: /search blocks /search, /search/, /search-results, /searchable-page — everything starting with that string. Add a trailing slash to scope to the directory.
• Wildcards. * matches any characters, $ anchors the end: Disallow: /*.pdf$ blocks all PDFs; Disallow: /*? blocks every URL with a query string (a common, sometimes catastrophic, rule — it can block faceted pages you wanted indexed).
• Longest match wins; Allow beats Disallow on ties. Allow: /admin/public/ (14 chars) beats Disallow: /admin/ (7 chars) for URLs under /admin/public/.
• Case-sensitive paths, one file per host and protocol (https://www. and https:// are different robots.txt files — worth checking after migrations).
• Disallow: with nothing after it means allow everything. Deleting a path but leaving the line inverts your intent silently.

The mistakes that cause real incidents

1. The staging block that shipped. User-agent: * / Disallow: / belongs on staging; deployed to production it stops all crawling. Traffic doesn’t vanish instantly — it decays over weeks, which makes the cause harder to spot. Put a robots.txt check in your deploy pipeline.
2. Blocking CSS/JS. Google renders pages; blocking /assets/ or /wp-includes/ can make pages render broken to Google and hurt mobile-friendliness assessments. Modern default: don’t block resource paths.
3. Blocking to deindex. As above — blocked pages can’t deliver their noindex. The correct removal sequence: allow crawling, add noindex, wait for recrawl, then optionally block.
4. The overzealous query-string block. Disallow: /*? written to kill duplicate content also kills pagination, filtered categories, or site search pages you monetize. Prefer canonical tags for duplicate control.
5. Editing without testing. Every rule interacts with every other rule via longest-match; eyeballing is unreliable. Google retired its tester, so use our Robots.txt Tester — paste your draft, run your critical URLs against Googlebot and the AI-bot matrix, and see exactly which rule fires per URL before deploying.

A robots.txt worth copying

For most content sites, less is more:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Add AI-crawler groups according to your training-data policy, keep the Sitemap: line current (it’s how crawlers find your sitemap), and resist the urge to enumerate every bot on the internet — every group you add is another place for rules to drift out of sync.

Verification loop

After any edit: fetch the live file (it caches — Google refreshes it roughly daily), run your top pages plus one URL from each Disallowed section through the tester, and confirm both directions: important pages allowed, blocked sections actually blocked. Sixty seconds, and it catches every mistake on this page.

Try the free Robots.txt Tester →