AI in web scraping: what 2026 conferences agree on

Teams that scrape billions of pages use AI to build their scrapers, not to run them. That’s the clearest pattern in more than 60 talks, podcasts and Reddit AMAs I watched or read from 2025 and 2026. The talks come from OxyCon, Prague Crawl, ZyteCon, Extract Summit, AI Engineer and RAISE. I also read 10 research papers on web agents, bot detection and LLM extraction.

Stan Sadokov, founder of the proxy provider NodeMaven, said it most directly in a Reddit AMA. AI is “excellent at building the scraper and repairing it when a layout changes, and terrible as a component inside the request path”.

This post covers where AI helps, where it still fails, and the risks of using it.

Why AI stays out of the request path

The first reason is cost. At OxyCon 2026, Oxylabs showed the cost calculation for its own traffic. The company scrapes about 7 billion pages a day. Parsing only 10% of them with the cheapest LLM, at about 1 cent a page, would cost “7 million dollars a day”.

Zyte reached the same conclusion in a different way. At ZyteCon 2026, its R&D team said per-page LLM extraction had limited use because of its cost, so they used the LLM to write extraction code instead. Zyte’s CTO said that “generic LLMs right now aren’t yet suited to production scraping at scale”.

Accuracy is the second reason. Scraped data is often used in pricing, finance and research products, where even a small error rate is not acceptable. As Zyte’s R&D speaker said at ZyteCon, an 85% solution isn’t enough, and in many cases you need 99%.

A 2026 study on embeddings shows the same cost gap in a related task. Across 37 tasks, the best LLM performed about the same as the best embedding model overall, at up to 1,431 times the cost (opens in a new tab). The authors’ advice applies to scraping too. Use the cheap model by default, and use the LLM only for the hard cases.

The case for LLM extraction

Not everyone agrees. At Extract Summit 2025, Jerome Choo of Diffbot argued that LLMs had become accurate and cheap enough for per-page extraction. His runs cost “roughly a tenth of a cent to 20th of a cent” per page, and his advice was to “write schemas, not rules”.

His inputs were cleaned text, not raw HTML, and that detail matters. One demo used article text, and the cost figure came from a page converted to Markdown. The NEXT-EVAL paper (opens in a new tab) tested one LLM on 164 real pages with 3 input formats. The F1 score was 0.10 on slimmed HTML and 0.96 on a flat JSON format. The input format changed the result more than anything else tested. So Choo’s results depend on clean input, and raw scraped HTML is not clean.

Where AI helps

AI helps most at build time and in operations. These are the uses that production teams described.

UseExampleResult reported
Writing scrapers from ticketsAn agent pipeline shown at Prague Crawl62 of 70 new GitHub issues needed no human help on the first try
Generating a full dataset pipelineA demo at OxyCon 20261,000 products in about 36 minutes, from an empty project
Diagnosing broken scrapersOxylabs internal agent for scraping engineersMedian time to fix a degraded scraper fell from 5 hours to 2, for problems the team could solve
Reading obfuscated anti-bot codeA speaker at Extract Summit DublinWork that took “three months” now takes about 15 minutes
Maintaining scrapersRafael Levi, Bright Data“I don’t ask the LLM to go do. I always ask, build a script”

All of these numbers come from the companies themselves, and I found no independent test of them.

The agent that engineers stopped using

The most useful story came from the Oxylabs talk, because it showed a project that didn’t work, with the numbers. The team built an agent that made parser changes, and an engineer then checked and deployed each one. It made 251 changes in under a year and cost $847.

The results were worse than the engineers’ own work.

  • 59% of its merge requests reached production, compared with 87% for engineers.
  • 76% passed CI tests, compared with 91%.
  • Engineers made 3.6 extra changes on average to each agent change.

The agent’s share of parser changes fell from 51% to 14%, while the models kept improving. As the speaker described it, “They just quietly stopped calling the agent”.

Oxylabs’ second agent, built for a different problem, worked better. It supported engineers instead of replacing them. It had command-line access, written instructions and guardrails, such as not drawing conclusions from too few jobs.

Where AI still fails

Fully autonomous scraping doesn’t exist yet. Ivan Sanchez of Zyte started his Extract Summit talk by saying “an end to end fully autonomous scraping web scraping agent still does not exist”.

He gave several reasons. A CAPTCHA ends the agent’s run. A single Amazon product page can exceed 1 million tokens of HTML. And sites change over time, so an agent has to keep checking them.

Long web tasks are also slow and expensive. The Odysseys benchmark (opens in a new tab) tested agents on 200 multi-site tasks on the live web. The best model completed 44.5% of them perfectly. Each 100-step run took about 30 minutes and cost about $2.50.

Coding agents reach a limit on the hardest targets. At Prague Crawl, the speaker who showed the agent pipeline said that on difficult sites or at high volume, Claude Code “just cannot do it”. He added that this happens when a team lacks the right skills and tools.

Another OxyCon speaker made the same point about general coding agents. They can scrape some sites, but for most of them “it’s not LLM helping you, but you helping it”.

Too much page content in the context makes agents worse. Hostinger’s team said at OxyCon that its agents received up to 500,000 characters from each web search. The team switched to showing page titles first and fetching full pages on request. Token use fell by 95%, and the agent’s answers improved.

Use the browser last

The scraping teams in these talks treat a full browser as the last option, not the default. The reasons are memory, speed and cost.

At OxyCon 2026, ScrapingBee compared running a page’s JavaScript in a lightweight DOM with loading it in a full browser. The lightweight method used 207 MB of memory, compared with 470 MB for the browser, on one test page. The speaker’s point was cost, because “the memory becomes density… And density becomes cost”. Less memory per page means more scrapers fit on one machine, and that lowers the cost.

ScrapingBee ranks the options from cheapest to most expensive.

  1. Parse the HTML the server returns.
  2. Call the API the page uses.
  3. Run the page’s JavaScript without a browser.
  4. Load the page in a full browser.

The speaker’s advice was to “reach the browser last, not first”.

Other teams report the same thing at a larger scale. Batuhan Özyön, founder of Scrape.do, said “more than 99% of our requests work without a browser”. Geonode’s team said “Half the sites that look like they need a browser are just hitting a JSON endpoint with a signature”.

Anti-bot design explains why some sites still need a browser. As Sarah McKenna of Sequentum explained at Extract Summit, “The only way to effectively block a bot is to make it more expensive and slower. And the way you do that is by forcing the use of browsers”.

AI agents are different. They usually need a browser to act on a page. Paul Klein IV of Browserbase said at AI Engineer that the most reliable browser agents in production “are often writing code alongside using the browser”.

Risks that grow with AI

AI makes some silent failures more likely. 3 were mentioned many times.

Successful requests with the wrong data

A status code of 200 doesn’t prove you got the data. At Prague Crawl, Logan Harless said “vendors with serious mindshare are returning 200 when really what they’re serving you is a CAPTCHA”.

Zyte’s data team described honeypots at ZyteCon, where “you get a 200 but it’s the wrong data”. Scrape.do described partial pages, where “the content is quietly degraded”, for example 5 records instead of 10.

Ian Kerins, CEO of ScrapeOps, warns that this may get worse. In his AMA, he said the next generation of anti-scraping “may make you collect data you cannot trust”. Checking the content of each response, not only its status code, is becoming part of the job.

Stealth tools that make detection easier

A 2026 study from the University of Lille and Inria tested 12 scraping tools and web agents (opens in a new tab) against 9 bot defenses. Its main finding was that “stealth and anti-detection mechanisms often increase detectability rather than decrease it”.

The researchers detected bots with 99.3% accuracy by combining IP, TLS and browser fingerprints. IP or TLS data alone was much less accurate. Saksham Solanki, creator of the httpcloak library, described the same idea in his AMA, saying “the handshake is a gate, the session is a score”. A correct TLS handshake only gives a scraper access. The site still scores everything it does after that.

Prompt injections that target scrapers

LLM-based scrapers read text that site owners control. A 2026 study found 15,387 prompt injections (opens in a new tab) on 11,722 pages, mostly from a scan of about half of a Common Crawl snapshot. Crawlers and scrapers were the main targets.

About 87% of the injections were invisible to human visitors. Many were in HTTP headers and JSON-LD blocks, which are exactly where scrapers look for clean data. The authors’ conclusion was careful. In their words, prompt injection “is not yet a dominant threat, but it is already sufficiently real, structured, and widespread to deserve attention”.

Is the web closing, or getting more expensive?

Speakers disagreed on this question more than on any other. Most positions belong to 4 groups.

PositionWho said itKey quote
Repriced, not closedIan Kerins, ScrapeOps“It is being repriced. Access remains possible, but fewer datasets are profitable to collect at scale”
Splitting in twoStan Sadokov, NodeMaven“open machine paths for the commodity stuff and harder walls around the valuable stuff”
Closing, for some usersKrishna, a lawyer at Black Forest Labs, OxyCon 2026 panelMore data is behind logins, paywalls and platforms in the US and China. For European labs, that creates “an asymmetry in data access”
Opening, for new usersPaul, journalism educator, OxyCon 2026 panel“for the vast majority of people, it was always closed. And actually, it’s opening up now”

The cost numbers support the “repriced” view. As Kerins said, “Proxies themselves are cheaper than they used to be, while the cost of producing one genuinely successful scrape has increased”. At ZyteCon, Zyte shared findings from a cost study of its clients’ scraping projects. People’s time was about 70% of the cost of running scrapers. Anti-ban tools were roughly 15%. And 20% to 30% of what teams counted as bans were malformed URLs.

How to decide where to use AI

Based on these talks, here are 5 practical rules.

  1. Use AI to write and repair scrapers. This is where production teams report the clearest gains.
  2. Keep extraction deterministic where accuracy matters. If you use an LLM, give it clean, structured input, and test the output against known data.
  3. Try the cheapest access method first. Check the HTML, then the page’s own API, and use a browser only when the data needs one.
  4. Check the content, not only the status code. A 200 response can contain a CAPTCHA, a partial page or fake data.
  5. Treat page content as untrusted input. Hidden instructions can reach any LLM that reads a page.

Most of the evidence here comes from vendors describing their own systems, so treat each number as a report, not a benchmark. Even so, companies that compete with each other every day described the same pattern. AI is now part of how scrapers get built. The scrapers themselves still run as regular code.

Sources

Talks and AMAs I used, in the order they appear.

Papers I cited are NEXT-EVAL (opens in a new tab), The Embedder’s Dilemma (opens in a new tab), Odysseys (opens in a new tab), Unmasking Web Agents with Multi-Layer Fingerprinting (opens in a new tab) and Indirect Prompt Injection in the Wild (opens in a new tab).

Doccraft writes research and tutorials like this for web data and AI infrastructure companies.