User Agents for Web Scraping: What to Send and Why It Matters

Updated July 25, 2026

🛠️ This guide pairs with our free User Agent Parser — See and parse any user agent — plus bulk log analysis and current UA strings for scrapers.

The User-Agent header is the first thing a server learns about your scraper, and the default your HTTP library sends — python-requests/2.32, Go-http-client/1.1, curl/8.9 — is an instant, unambiguous “I am a script.” Plenty of sites block on that string alone. This guide covers what to send instead, why the popular advice about rotation is mostly wrong, and how to read the user agents in your own logs.

What a user agent actually says

A modern browser string looks like this:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

Only two pieces carry information: the platform block (Windows NT 10.0; Win64; x64) and the real browser token (Chrome/126.0.0.0). Everything else is compatibility archaeology — Chrome claims to be Safari, which claims to be KHTML, which claims to be Gecko, and every browser on earth still opens with Mozilla/5.0 because 1990s servers sniffed for it. That’s why naive checks like if "Safari" in ua misclassify most traffic, and why parsing needs a specific token order.

The two legitimate strategies

Identify honestly. Send something like MyBot/1.0 (+https://example.com/bot-info) pointing at a page that explains what you collect and how to opt out. This is the right default for research, monitoring and any project you’d be comfortable describing publicly. Many operators whitelist transparent bots, and if you ever do cause a problem, they contact you instead of blocking your whole IP range.

Send a complete browser profile. For sites that block all unknown agents, use a current, real browser string. The important word is complete: a Chrome user agent arriving with no Accept-Language, no Accept-Encoding and no Accept header is more suspicious than an honest bot, because real Chrome never does that. Header consistency is what basic fingerprinting checks, not the UA alone.

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
}

Our User Agent Parser generates current browser strings for exactly this — and because it builds them from your own browser’s version rather than a hard-coded list, they don’t go stale the way copied-from-a-blog-post user agents do. A UA claiming Chrome 103 in 2026 is its own red flag.

Why UA rotation is overrated

The standard advice is “rotate user agents to avoid detection.” In practice this helps less than people think and can actively hurt:

• It’s inconsistent. Real browsers don’t change identity between requests. A session that claims Chrome on Windows, then Safari on macOS, then Firefox on Linux — from one IP, in ninety seconds — is a stronger bot signal than any single UA.
• It doesn’t address what actually triggers blocks. Request rate, concurrency and total volume are what get you rate-limited. Fifty requests per second gets blocked under any user agent.
• Modern detection doesn’t rely on it. TLS fingerprints, HTTP/2 frame ordering, header order and behavioural signals are all checked before anyone reads your UA string.

Pick one appropriate identity, keep it stable for the session, and put your effort into rate limiting and caching instead — the techniques covered in our scraping without getting blocked playbook.

Client Hints: the slow death of the UA string

Chrome is freezing and reducing user agent detail, moving specifics into Sec-CH-UA Client Hint headers that servers must explicitly request. For scrapers this is mildly good news: as UA strings get less unique, they carry less fingerprinting value. The practical takeaways today are that UA-based device detection is getting less reliable, headless browser automation should send Client Hints consistent with the UA it claims, and plain HTTP scrapers can keep sending a standard UA string — it still works everywhere.

Reading the user agents hitting your server

The same skill runs in reverse. Your access logs tell you who’s crawling you, and the answer is often surprising — a large share of traffic on a typical site is automated. Paste your log lines into the bulk analyzer and you’ll get a breakdown by client with a bot percentage and CSV export. What to look for:

• Legitimate search crawlers (Googlebot, Bingbot) — you want these; verify suspicious ones with a reverse DNS lookup, since Googlebot is the most-spoofed UA on the internet.
• AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) — decide your policy deliberately and enforce it in robots.txt; see how to block AI crawlers and verify with the Robots.txt Tester.
• SEO crawlers (AhrefsBot, SemrushBot, MJ12bot) — harmless but bandwidth-hungry; block them if your hosting is metered.
• Default library strings (python-requests, Go-http-client, Scrapy) — someone is scraping you without bothering to hide it, which tells you exactly how much effort they’ll spend if you make it slightly harder.

The short version

Send an honest bot UA if you can, a complete and current browser profile if you must, keep it consistent within a session, and spend your energy on politeness rather than disguise. Scrapers that survive long-term are the ones that behave like considerate clients — the user agent is just the introduction.

Try the free User Agent Parser →