Focus area
Web scraping APIs and proxies
Doccraft tests how real websites respond to scrapers, from block pages that return a 200 status to robots.txt rules and user agent checks, and writes about it for the companies that build scraping APIs and proxy networks.
Research and guides
- How 16 scraping tools and headless browsers identify themselves, and which check robots.txt by default
I ran 16 popular scraping tools and headless browsers, including browser tools built for AI agents, against a test site whose robots.txt disallowed the page. 6 named themselves in the user agent, 5 sent HeadlessChrome, and 5 sent a regular browser user agent or none at all. Of the 14 I could test locally, only Scrapy in a Scrapy project checked robots.txt by default.
- RSL licensing for AI crawlers: 4 of the top 1,000 sites publish it, and 1 of its 24 featured supporters
I checked the top 1,000 websites and the main domains of the 24 supporters the RSL Collective features on its site. 4 top sites have an RSL License line, and 2 serve a readable RSL file. Of the supporters, only Medium has one, and its license file returned 403 to my crawler.
- GitHub stars barely predicted downloads for Python scraping tools: 39 tools compared
I compared GitHub stars with downloads for 39 open-source scraping, browser and extraction tools. Among the 29 Python tools, stars and downloads had almost no relationship. Archived tools kept millions of installs, and small libraries underneath popular tools got the most.
- Extract Summit 2026: AI writes the scrapers, but the data still needs proof
I watched all 11 talks from Extract Summit 2026 in Austin. AI now writes and repairs scrapers in production. Most speakers focused on how to prove the results are right, with verifiers, independent checks and evidence files.
- What top websites tell AI crawlers: robots.txt rules on 544 sites, bot by bot
I checked the robots.txt of 544 top websites for 15 AI and search user agents. None blocks Googlebot, 17% block GPTBot, and many sites treat AI training, AI search and user requests differently.
- Do you need a browser for web scraping? A 1,000-site test
I tested the top 1,000 websites with a plain HTTP request and a headless browser. 5% needed JavaScript, and on 14% the browser loaded pages that blocked the plain request.
- Why a 200 response can still be a block
A status of 200 means the server answered, not that you got the data. How blocked responses arrive with a 200, and a short check that flags them.
- Block, error or bug: how production teams classify failed scrapes
Status codes lie. On 371 top sites, 12% returned a 200 for a page that does not exist. How production scraping teams classify every response, with a classifier tested on those sites.
- Does copying Chrome's user agent help a scraper? A 472-site test
Copying Chrome's user agent is a common first fix for a blocked scraper. I tested it against 4 other user agents on 472 top sites. It didn't do better than an honest one, and it changed which page some sites sent.
- Skip unchanged pages on a re-crawl: how many top sites return 304
A conditional request lets a site answer "not modified" with no page body. I tested it on 368 top homepages and 224 inner pages. About 40% returned a 304, and half sent nothing to check.
Published for clients
- Benchmarking Datacenter Proxies Under Concurrency (opens in a new tab)evomi.com
- The Ultimate Guide to Proxy IP Types (opens in a new tab)brightdata.com
- How to Verify a Mobile Proxy's Carrier, ASN, and Location (opens in a new tab)evomi.com
- Bots Block Bots: Why Anti-Bot Systems Flag LLM Browsers (opens in a new tab)scrapingbee.com
- Scraper API Retry Handling for Production Pipelines (opens in a new tab)evomi.com
- Python Web Crawler: How to Build One That Scales With Asyncio (opens in a new tab)gologin.com
- Most Popular Anti-Scraping Techniques in 2026 (opens in a new tab)brightdata.com