Do you need a browser for web scraping? A 1,000-site test

Only 5% of the top websites I tested needed a browser to show their content. Blocking was a far bigger reason to use one. On 14% of sites, a plain HTTP request was blocked while a headless browser loaded the page.

I ran the test because the question appeared often in the scraping talks I watched for my post on AI in web scraping. Scrape.do’s founder said “more than 99% of our requests work without a browser”. At Prague Crawl, a Browserbase engineer said that without a browser, you have “no chance” of bypassing the TLS fingerprinting that anti-bot systems use. Neither side had shared independent data, so I collected some.

How the test worked

I took the top 1,000 domains from the Tranco list (opens in a new tab) published on September 29, 2026. Tranco is a ranking of popular domains that researchers use because it’s hard to manipulate.

For each domain, I loaded the homepage twice.

  1. A plain HTTP request, with Python’s httpx library and a normal Chrome user agent. This is how many scrapers work.
  2. A headless Chromium browser, through Playwright, with the same user agent. It waited for the page to finish loading.

Then I compared the text each one got. If at least 60% of the words a person sees in the browser were also in the plain HTML, the page didn’t need a browser.

Each site got 1 request to robots.txt, 1 plain request and 1 browser load. I skipped the 32 sites whose robots.txt disallows all crawlers, including Facebook and Twitter. The test didn’t try to bypass any block. It recorded each block as one of the results.

About 40% of the top 1,000 domains aren’t websites people visit. Domains like gstatic.com and akamai.net serve files, ads or network traffic, so 279 had no working homepage and 112 had no real content. After removing those, plus 24 pages the browser couldn’t load and 9 that redirected to another site, 544 homepages remained.

The results

A bar chart of the results for 544 homepages
What 544 of the top 1,000 homepages needed from a scraper.
ResultHomepagesShare
Text in the plain HTML33562%
Data in the page source as JSON183%
Text partly in the plain HTML81%
Needs JavaScript295%
Plain request blocked, browser loaded7714%
Browser blocked, plain request loaded265%
Blocked for both519%

“Text in the plain HTML” means at least 60% of the words shown in the browser were already in the HTML. Changing that threshold to any value between 50% and 80% kept this share between 56% and 63%. The share that needed JavaScript stayed at 5%.

Most pages don’t need JavaScript

On 62% of homepages, the text was already in the HTML the server returned. This included Microsoft, Apple, GitHub, PayPal and Cloudflare.

Another 3% loaded their text with JavaScript, but the same data was already in the page as JSON. Instagram, Uber, weather.com and DuckDuckGo were in this group. A scraper can read that JSON directly, without running any code.

Modern frameworks don’t change this much. 62 homepages were built with Next.js, and 54 of them had their text in the plain HTML. These 54 sites use server-side rendering or static generation.

Structured data was common too. About 55% of the pages with readable content included JSON-LD, the format sites use to describe products, articles and organizations for search engines. It’s often the cleanest place to find prices, dates and names.

Only 29 homepages, about 5%, needed JavaScript. Most were web apps, not content pages, like Spotify, Twitch, Bluesky, Duolingo and iCloud.

Blocking is the more common reason to use a browser

On 77 sites, 14% of the 544 homepages, the plain request was blocked while the browser loaded the page. That’s about 2.7 times the number of sites that needed JavaScript.

The blocks came from a few systems.

  • Cloudflare challenge pages appeared on 41 of the 77 sites, including openai.com and claude.ai.
  • Amazon returned a short challenge page instead of the store on all 12 stores I tested. On 9 of them, including amazon.com, .co.uk, .de and .in, the challenge page had a 202 status.
  • Wikipedia returned a 403 status with a message that begins “Please respect our robot policy”.

The Wikipedia case shows that you don’t always need a browser to fix a block. My HTTP client sent a Chrome user agent and was blocked, while curl with the same user agent loaded the page. Once my HTTP client’s user agent included contact details, as Wikipedia’s bot policy asks, it loaded the page too.

A browser can also get you blocked

The reverse happened on 26 sites, about 5%. The plain request got the full page, and the headless browser was blocked. eBay, Zillow, Salesforce, Lowe’s and Mayo Clinic were in this group.

Headless Chromium has its own detectable signs, and some anti-bot systems look for them. A headless browser looks different to a site, but it doesn’t look more human by default.

Some sites block both

51 sites, about 9%, blocked both methods. They included The New York Times, Stack Overflow, Etsy, Tripadvisor and Perplexity. The Times served a DataDome challenge, and Stack Overflow served a Cloudflare one.

For these sites, neither a plain request nor a default headless browser was enough. Access depends on other factors, like IP reputation and a consistent browser fingerprint. It also depends on what the site’s terms allow.

The cost of using a browser

The median plain request took 0.7 seconds. The median browser load took 9.5 seconds, about 14 times longer.

That browser time includes waiting for the page to finish loading, up to about 9 seconds. A tuned scraper can shorten that wait. It can’t remove the cost of starting a browser and running each page’s JavaScript, which is why speakers at OxyCon and Extract Summit described a browser as the last option, not the first.

What this means for your scraper

  1. Start with a plain request. For most pages, the text is already in the HTML.
  2. Look for data in the page before you render it. Check for JSON-LD, __NEXT_DATA__ and other JSON in <script> tags.
  3. When a request fails, check why. A 403 with a Cloudflare page is a blocking problem, and a page with no text is a JavaScript problem. They need different fixes.
  4. Don’t assume a browser will help. It loaded the page on 14% of homepages where a plain request was blocked, and it was blocked on another 5% where a plain request worked.
  5. Be consistent about who you are. A client that claims to be Chrome but doesn’t behave like Chrome is easy to detect. Some sites, like Wikipedia, publish rules for bots and ask them to identify themselves.

Limits of this test

  • Homepages only. Product, search and article pages often behave differently, and they’re usually what scrapers need.
  • One run, from one place. I ran the test once, from Mumbai. Some sites change content and blocking by country and over time.
  • One HTTP client and one browser setup. A client with a browser-like network fingerprint, or a browser with different settings, would get different results.
  • Words, not meaning. The test compared words only. A page that adds translations with JavaScript, like example.com, counts as needing JavaScript even when its main text is in the HTML.

Data and code

Everything is public, so you can check the results or run the test yourself.

Correction

On October 2, 2026, I found that some block markers also appear on normal pages. For example, Cloudflare adds its challenge-platform script to pages it doesn’t block. 14 sites that loaded normally had been counted as blocked. The analysis now counts a marker only when the page’s text is missing. The share of sites where only the plain request was blocked fell from 17% to 14%, and the share with text in the plain HTML rose from 59% to 62%. The other findings didn’t change.

Doccraft writes research and tutorials like this for web data and AI infrastructure companies.