User Agent Rotation for Web Scraping: What Works, What Doesn't, and What You Actually Need

User-agent rotation is one of the most commonly misunderstood anti-detection techniques. It helps — but it's not magic. Here's what actually works and why. What is a user agent? The user agent string is a header sent with every HTTP request that identifies the client software: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 This tells the server: OS: Windows 10, 64-bit Browser engine: WebKit/Blink Browser: Chrome 122 Websites use this to: Serve device-appropriate content (mobile vs desktop) Detect bots (unusual or outdated user agents) Identify scraper frameworks (Python's requests sends python-requests/2.28.0 by default) The basics: always set a real user agent import requests # BAD - sends "python-requests/2.28.0" — immediately flagged response = requests.get('https://example.com') # GOOD - looks like a real browser headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko