Focus area
Web scraping APIs and proxies
Scraping APIs and proxy networks decide whether a scraper gets the page, gets blocked, or gets a block that looks like a page. Doccraft tests how real sites respond, and writes about it for the companies that build this infrastructure.
Research and guides
- Do you need a browser for web scraping? A 1,000-site test
I tested the top 1,000 websites with a plain HTTP request and a headless browser. Of the 544 working homepages, only 5% needed JavaScript. Blocking was the bigger problem.
- Why a 200 response can still be a block
A status of 200 means the server answered, not that you got the data. How blocked responses arrive with a 200, and a short check that flags them.
- Does copying Chrome's user agent help a scraper? A 472-site test
Copying Chrome's user agent is a common first fix for a blocked scraper. I tested it against 4 other user agents on 472 top sites. It didn't do better than an honest one, and it changed which page some sites sent.
Published for clients
- Scraper API vs. Scraping Browser: When to Use Each One (opens in a new tab)evomi.com
- Most Popular Anti-Scraping Techniques in 2026 (opens in a new tab)brightdata.com
- Bots Block Bots: Why Anti-Bot Systems Flag LLM Browsers (opens in a new tab)scrapingbee.com
- Geographical Pricing Explained (opens in a new tab)brightdata.com
- Cross-Country Ecommerce Price Monitoring: How to Get and Prove Local Prices (opens in a new tab)evomi.com