Blog
Original research, benchmarks and tutorials for developers working with web data and AI.
- How 16 scraping tools and headless browsers identify themselves, and which check robots.txt by default
I ran 16 popular scraping tools and headless browsers, including browser tools built for AI agents, against a test site whose robots.txt disallowed the page. 6 named themselves in the user agent, 5 sent HeadlessChrome, and 5 sent a regular browser user agent or none at all. Of the 14 I could test locally, only Scrapy in a Scrapy project checked robots.txt by default.
· 8 min readBlockingAI - Chrome DevTools MCP's --no-javascript-evaluation flag, tested: 39% more input tokens, 18 of 20 answers right
Chrome DevTools MCP has a --no-javascript-evaluation flag that removes the agent's evaluate_script tool. I ran Claude on 10 real tasks with and without it, twice. With the flag, it got 18 of 20 right, compared with 20 of 20 without it. It also used 39% more input tokens and cost 2.3 times as much.
· 6 min readAIBrowsers - rea vs Playwright MCP for finding a web page's hidden API, tested on 8 pages
I gave Claude 8 real pages and asked it to find the API each page loads its data from, with rea 6.8.0 and with Playwright MCP. Both found the same APIs. rea's 229 tool definitions added 103,531 tokens to every turn, unless the client defers them, as Claude Code does. Network captures of up to 3 million characters ended 4 of 16 runs.
· 8 min readAIBrowsers - Best headless browser for AI agents: Chrome, Lightpanda and Obscura tested on 50 real pages
I ran Chrome 156, Lightpanda 1.0.2 and Obscura 0.2.5 through agent-browser on 50 real pages, twice. Lightpanda matched Chrome's text on 33 of 41 pages with a median of 120 MB against Chrome's 1,636 MB. Obscura matched 26 and returned an empty page on 11. On 10 agent tasks, Claude got all 20 answers right on every engine.
· 9 min readBrowsersAI - RSL licensing for AI crawlers: 4 of the top 1,000 sites publish it, and 1 of its 24 featured supporters
I checked the top 1,000 websites and the main domains of the 24 supporters the RSL Collective features on its site. 4 top sites have an RSL License line, and 2 serve a readable RSL file. Of the supporters, only Medium has one, and its license file returned 403 to my crawler.
· 7 min readAIBlocking - Shopify WebMCP: the cart tool tells AI agents not to ask, and only checkout is marked consequential
28 of 30 Shopify stores I checked register the same 12 WebMCP tools for browser agents. The add_to_cart description says "do NOT ask the user to confirm", and none of the 12 is marked consequential. Only complete_checkout is. In 40 runs, Claude called the tools twice and never went to checkout.
· 8 min readAIBrowsers - Exa Lite found 50 of 53 answers, 1 fewer than Exa Auto, in less than half the time
Exa 3.0 added Exa Lite, a search type at $4 per 1,000 requests. I tested Lite, Auto and Deep on the same 53 questions with known answers. Lite found 50, Auto and Deep found 51, and Lite's median response took 746 ms against 1,732 ms for Auto.
· 5 min readAI - Playwright MCP vs Chrome DevTools MCP vs agent-browser: same accuracy, but tokens per task ranged from 4.1K to 95K
I ran the same Claude agent on 19 real browser tasks, twice, through Playwright MCP, Chrome DevTools MCP and 5 agent-browser setups. All 7 got every answer right. The median task read 31K tokens with Playwright MCP, 46K with Chrome DevTools MCP, and 4.1K to 95K with agent-browser, depending on how it was loaded. In dollars, the typical task cost under 4 cents in every setup.
· 15 min readAIBrowsers - Shopify UCP catalog for AI agents: page prices match, but sold-out sizes are hidden by default
All 34 Shopify stores I checked publish a UCP catalog for AI agents, with no API key. Prices matched the product pages, and stock matched for 50 of 50 sizes. Still, 3 defaults can give a data team missing or wrong data. Sold-out sizes are missing, prices come in the seller's currency, and anonymous calls from one IP address stop at about 100 per store per hour.
· 11 min readAIExtraction - Cloudflare's Clairic got 28 of 34 product prices right, and every wrong one scored below 0.9 confidence
Cloudflare released Clairic, an extraction model that returns typed values with a confidence score. I tested it on 34 product pages with the structured data removed and on 13 block pages. It beat 2 price rules and a local LLM, and its confidence score separated the right answers from the wrong ones.
· 5 min readExtractionAI - Rusteerio matched Cheerio's output on 353 real homepages and was 3.4 times faster
Rusteerio is a new Rust port of Cheerio. I ran the same extraction with both on 353 homepages of top websites. The output was identical on every page, Rusteerio was faster on every page, and the gain grew with page size.
· 4 min readExtraction - Browser Use found the right price on 29 of 34 stores, and duplicate product names caused most misses
I gave the Browser Use agent a store homepage and a product name on 34 real stores, twice. It returned the right product and price for 29 and 28. Most misses were other listings with the same name. Each product took about a minute and about a cent.
· 9 min readAIBrowsersExtraction - GitHub stars barely predicted downloads for Python scraping tools: 39 tools compared
I compared GitHub stars with downloads for 39 open-source scraping, browser and extraction tools. Among the 29 Python tools, stars and downloads had almost no relationship. Archived tools kept millions of installs, and small libraries underneath popular tools got the most.
· 7 min readExtractionBrowsers - Cloudflare's Web Search API, tested on 53 questions: the provider you pick matters far more than the route
I tested Cloudflare Web Search API with Exa, Linkup and Ceramic.ai, and called each provider directly, on 53 questions with known answers. Cloudflare matched direct calls for Exa and Linkup, changed Ceramic's results and dropped untitled results. The providers were very different.
· 11 min readAI - Lightpanda 1.0 built 21 of 28 real JavaScript sites, but only with the right wait setting
I tested Lightpanda 1.0.1 and headless Chromium on 28 real sites that need JavaScript, twice. One wait setting changed the result from 12 sites to 21. Lightpanda was about 2 times faster than Chromium and used about 3 times less memory.
· 8 min readBrowsersAI - Taking the first price from a product page gave a wrong price for 21 of 33 stores
I tested 2 rule-based methods and a local LLM on 33 real product pages, and gave 11 block pages to a text extractor. Count wrong values and empty values separately.
· 8 min readExtractionAI - Extract Summit 2026: AI writes the scrapers, but the data still needs proof
I watched all 11 talks from Extract Summit 2026 in Austin. AI now writes and repairs scrapers in production. Most speakers focused on how to prove the results are right, with verifiers, independent checks and evidence files.
· 14 min readAIExtraction - What top websites tell AI crawlers: robots.txt rules on 544 sites, bot by bot
I checked the robots.txt of 544 top websites for 15 AI and search user agents. None blocks Googlebot, 17% block GPTBot, and many sites treat AI training, AI search and user requests differently.
· 5 min readAIBlocking - MCP's context problem, measured: tool definitions vs tool results
Apify's CEO argued at Prague Crawl 2026 that MCP isn't the problem, the agents are. I measured 6 MCP servers. 95 tool definitions took 15,950 tokens, and one page snapshot took 31,565.
· 5 min readAIBrowsers - Block, error or bug: how production teams classify failed scrapes
Status codes lie. On 371 top sites, 12% returned a 200 for a page that does not exist. How production scraping teams classify every response, with a classifier tested on those sites.
· 6 min readBlockingExtraction - Skip unchanged pages on a re-crawl: how many top sites return 304
A conditional request lets a site answer "not modified" with no page body. I tested it on 368 top homepages and 224 inner pages. About 40% returned a 304, and half sent nothing to check.
· 5 min readExtraction - Common Crawl's WET text vs trafilatura vs Resiliparse, retested
Hugging Face's FineWeb team found Common Crawl's ready-made text too noisy and extracted their own. I retested that choice on 2,000 pages from the September 2026 crawl, with FineWeb's own quality filter.
· 6 min readAIExtraction - JSON-LD for scrapers: what it contains and what it misses
More than half of the top homepages that returned content in my test include JSON-LD, structured data written for search engines. What it contains, how to read it in Python, and what it missed on 8 real product pages.
· 5 min readExtraction - Scraping JavaScript pages without a full browser, retested
ScrapingBee's OxyCon 2026 talk says jsdom runs a page's JavaScript with less than half a browser's memory. The saving held, but on 29 real JavaScript sites, the full browser built far more pages.
· 6 min readBrowsersExtraction - Does copying Chrome's user agent help a scraper? A 472-site test
Copying Chrome's user agent is a common first fix for a blocked scraper. I tested it against 4 other user agents on 472 top sites. It didn't do better than an honest one, and it changed which page some sites sent.
· 6 min readBlocking - Hidden prompt injections that target scrapers: what a 2026 study found
A 2026 study found 15,387 prompt injections hidden in real web pages, aimed mostly at crawlers and scrapers. What they say, where they hide, what the top 1,000 sites do, and how to handle them.
· 6 min readAIExtraction - How to find the JSON data hidden in a page's source
Many pages send their data as JSON inside the HTML. How to find it, the patterns to look for, and working Python code to read it without a browser.
· 6 min readExtraction - What Oxylabs learned from its AI parser agent
At OxyCon 2026, Oxylabs openly shared why its engineers stopped using an AI parser agent, and what its second agent did differently.
· 4 min readAI - Why a 200 response can still be a block
A status of 200 means the server answered, not that you got the data. How blocked responses arrive with a 200, and a short check that flags them.
· 4 min readBlocking - AI in web scraping: what 2026 conferences agree on
I studied more than 60 web scraping talks, podcasts and AMAs and read 10 papers. The clearest finding is that teams use AI to build scrapers, not to run them.
· 10 min readAIBlocking - Do you need a browser for web scraping? A 1,000-site test
I tested the top 1,000 websites with a plain HTTP request and a headless browser. 5% needed JavaScript, and on 14% the browser loaded pages that blocked the plain request.
· 6 min readBrowsersBlockingExtraction