What a user agent tells you
Every HTTP request carries a User-Agent header identifying the client. Servers use
it to choose layouts, gate features, collect analytics — and to decide whether you look like a
bot. Reading these strings matters in two directions: analysing your traffic (who and what is
visiting), and setting your own when you write a scraper.
Anatomy of a modern user agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 └─ legacy token └─ platform └─ engine └─ compat └─ real browser └─ compat
Only two parts carry real information: the platform block and the actual browser token. The
rest is compatibility cruft dating back to the browser wars — Chrome claims to be Safari, which
claims to be KHTML, which claims to be Gecko, and everything claims to be Mozilla. This is why
naive substring matching (if "Safari" in ua) misclassifies most traffic, and why
a real parser checks tokens in a specific order.
User agents for scraping: honest vs. blending in
Sending your HTTP library's default (python-requests/2.32) is the fastest way to
get blocked, since it announces automation to any filter. Two legitimate alternatives:
- Identify honestly:
MyBot/1.0 (+https://example.com/bot)with a page explaining what you collect. Best for research and monitoring — many operators whitelist transparent bots, and it's the ethical default. - Send a real browser profile: for sites that block unknown agents. Consistency
matters more than the string itself — a Chrome UA with no
Accept-Languageheader is more suspicious than an honest bot. Send the whole header set, keep it stable per session.
Whichever you pick, the technique that actually keeps scrapers alive is rate limiting, not disguise — see our polite scraping playbook.
Common bot user agents worth recognising
| Bot | Identifying token | Purpose |
|---|---|---|
| Googlebot | Googlebot/2.1 | Google Search crawling |
| Bingbot | bingbot/2.0 | Bing Search crawling |
| GPTBot | GPTBot | OpenAI model training |
| ClaudeBot | ClaudeBot | Anthropic model training |
| PerplexityBot | PerplexityBot | Perplexity answer engine |
| CCBot | CCBot | Common Crawl dataset |
| AhrefsBot / SemrushBot | AhrefsBot, SemrushBot | SEO backlink indexes |
| facebookexternalhit | facebookexternalhit | Social preview fetching |
Seeing unexpected AI crawlers in your logs? Control them in robots.txt — see how to block AI crawlers, then verify your rules with the Robots.txt Tester.
Frequently asked questions
What is my user agent?
Your browser’s user agent is shown at the top of this page automatically, along with a breakdown of the browser, rendering engine, operating system and device it identifies. It’s the string your browser sends in the User-Agent HTTP header on every request.
What is a user agent string?
A line of text every HTTP client sends to identify itself — browser name and version, rendering engine, operating system, and sometimes device model. Servers use it to decide what to send back: mobile vs desktop layouts, feature support, or whether to block you as a bot.
Can I parse a lot of user agents at once?
Yes. Switch to Bulk mode and paste up to thousands of lines — raw log lines work, since the tool extracts the UA portion automatically. You get a sortable table with browser, OS, device and bot classification, plus CSV export. Most UA tools only handle one string at a time.
Which user agent should I use for web scraping?
Either identify honestly as a bot (MyBot/1.0 (+https://yoursite.com/bot)), which many site operators whitelist, or send a complete, current browser profile. The "For scraping" tab below generates current browser strings derived from a real browser, plus ready-to-paste Python, Scrapy and Playwright snippets.
Why do all browsers say "Mozilla/5.0"?
Historical accident. In the 1990s servers sniffed for "Mozilla" to serve modern HTML, so every browser added it for compatibility — and never removed it. The same reason Chrome claims "Safari" and "KHTML, like Gecko". User agent strings are archaeology, not documentation.
What are Client Hints and are they replacing user agents?
Chrome now freezes and reduces the detail in its UA string, moving specifics to Sec-CH-UA Client Hint headers that servers must explicitly request. This page shows your Client Hints alongside your UA. For scraping, the practical impact is that UA strings are becoming less unique — which is good for blending in.