Web Scraping Without Getting Blocked: A Practical, Polite Playbook

Updated July 25, 2026

🛠️ This guide pairs with our free CSS & XPath Selector Tester — Test CSS selectors and XPath live against your HTML, then copy working scraper code.

Most scraper blocks aren’t sophisticated bot detection catching a clever adversary — they’re servers reacting sensibly to clients that behave nothing like the traffic they’re built for: hundreds of requests per second, no headers, no pauses. The fix is mostly to be a better-behaved client, which conveniently is also the ethical way to scrape. This playbook covers the techniques that keep legitimate scrapers running.

First: the checks that make blocking moot

1. Is there an API? Many sites publish one — official APIs are faster, structured, stable, and often permit exactly the use you have in mind. Ten minutes of reading docs regularly saves weeks of scraper maintenance.
2. Is the data in the sitemap or a feed? RSS/Atom feeds and XML sitemaps are published for machine consumption. Requesting exactly the pages listed there, once each, is the politest crawl possible — no discovery crawling, no wasted requests.
3. What does robots.txt say? Compliant scraping starts with reading it (here’s how the rules work). Respecting it isn’t just ethics — Disallowed sections are frequently where the honeypots and aggressive rate limits live.
4. Check the terms of service. Especially for anything commercial, logged-in areas, or personal data — the legal exposure is real and no header trick changes it.

Look like the browser you claim to be

The default user agent of every HTTP library (python-requests/2.32, curl/8.9) is an instant block on many sites. Two legitimate options:

• Identify honestly as a bot — the gold standard for research and monitoring: MyProjectBot/1.0 (+https://example.com/bot-info) with a page explaining what you collect and how to opt out. Many operators whitelist transparent bots.
• Send a realistic browser profile for sites that block all unknown agents. That means the full header set, not just the UA string: Accept, Accept-Language, Accept-Encoding, Referer where natural. A Chrome UA with no Accept-Language is more suspicious than an honest bot — header consistency is what basic fingerprinting checks.

Whichever you choose, keep it consistent per session; rotating identities mid-session is a detection signal in itself.

Rate limiting: the technique that matters more than all others

A human reads a page in 10+ seconds. A scraper hitting 50 pages/second is indistinguishable from a small DoS attack — and gets treated like one.

• One request at a time, 1–5 seconds between requests is the classic polite baseline. Add jitter (random.uniform(1, 4)) — perfectly periodic requests scream automation.
• Respect Crawl-delay if robots.txt declares it, and back off on 429/503: honor Retry-After, and use exponential backoff instead of hammering.
• Scrape off-peak for the target’s timezone when you need volume — same requests, less impact on real users.

Don’t request what you already have

The least blockable request is the one you never send:

• Cache everything and re-request only when content can have changed — the sitemap’s lastmod values tell you exactly which pages updated.
• Use conditional requests: send If-Modified-Since/If-None-Match from the Last-Modified/ETag headers you got last time (header refresher); a 304 costs the server almost nothing.
• Parse from saved HTML while developing. Re-fetching a page 200 times while you debug selectors is how development IPs get banned. Save the page once, then iterate against the file — our CSS & XPath tester exists for exactly this loop: paste the saved HTML, perfect the selector, copy working BeautifulSoup/Scrapy code, all without touching the target site again.

Handle the page you actually received

Blocks aside, most scraper “failures” are really parsing surprises: the server returned a login page, a consent interstitial, or an error page with a 200 status. Check for a sentinel element (the product title, the article body) before parsing, and fail loudly when it’s absent. Selector tips for markup that shifts: prefer stable attributes (id, data-*) over auto-generated class names, and XPath text anchors (//th[text()="Price"]/following-sibling::td) survive redesigns that class-based selectors don’t.

What this playbook deliberately omits

CAPTCHA-solving services, residential proxy rotation to evade IP bans, and header spoofing to defeat active anti-bot systems. Once a site is actively telling you no — CAPTCHAs, bans, legal notices — continuing is trespassing, whatever the technique. At that point the moves are: the API, a licensing conversation, or a different data source. Sustainable scraping is a cooperation problem, not an arms race: identify yourself or blend in honestly, go slow, cache hard, and take the data the site already publishes for machines.

Try the free CSS & XPath Selector Tester →