Blog

Extract Summit 2026: AI writes the scrapers, but the data still needs proof

Satyam Tripathi · · 14 min read

Extract Summit is Zyte’s conference for teams that collect web data at scale. The 2026 event was in Austin on October 7 and 8. I watched all 11 talks, from Zyte’s CEO to a legal panel.

AI now writes scrapers, repairs them and investigates support tickets in production. This matches what I found at other 2026 conferences on AI in web scraping. But most speakers focused on a different question, which was how to prove the results are right. Tests and verifiers check that AI-written code works. Independent checks show whether the data is complete. Buyers ask vendors to prove their data, site owners ask bots to prove who they are, and lawyers ask how the data was collected.

Proof that AI-written code works

Shane Evans, Zyte’s CEO and co-founder, shared Zyte’s numbers on what AI changed. Nearly all of Zyte’s developers use Claude Code. After the team added skills and frameworks around it, development time fell by roughly half from May to September. The support team solved 27 tickets a week in the first half of the year and 58 a week in September, while incoming tickets rose 60%. The change came from a system, deployed in July, that investigates each ticket in detail before an engineer is assigned.

But one large Zyte project went the other way. Its team used more AI and wrote more code, and when Shane audited it, the code looked worse. The project had “lots of AI-generated code, but not enough checks”. Its data listed one desk as 1.7 m long and 10 m wide, an obvious error that no check caught. In another case, an agent was told that a green tea-scented candle isn’t green. It added “green tea” to a list of words that aren’t colors, and kept adding special-case fixes like that. Shane said the agent didn’t learn to parse the website. It “actually just learned to pass the tests”.

After the audit, the team followed his recommendations. The project now looks much better, he said, and the same engineers started their next project by getting the checks right first.

Zyte’s answer is to keep the agent inside a loop of checks. AI writes the extraction code, and AI also extracts data directly from a sample of pages. When the two results differ, Zyte investigates. For example, a store might add a new kind of discount that the code doesn’t capture. The crawl still returns the same fields as before, so nothing looks broken. Only the comparison shows the change.

In production, the loop has 5 steps.

  1. A monitor watches for problems.
  2. An agent finds the cause and writes a fix.
  3. The agent checks the fix against a sample crawl.
  4. The agent adds the case to the test fixtures, so it’s tested every time.
  5. An engineer approves the fix before it’s deployed.

His conclusion was “it’s not necessarily the model, it is the framework here and the loop and the testing around it that makes this work”.

Three other talks showed what that framework looks like.

A spider that repairs itself during the crawl

Konstantin Lopukhin, Zyte’s head of R&D, started with the usual way teams fix broken scrapers. A monitoring service finds a broken scraper and creates a ticket. A coding agent writes the fix, and QA reviews the data. He said this setup copies the company’s team structure, or “we basically shipped our org chart here”. There’s nothing wrong with that, he said, and most teams build it this way. But it has problems. Each step is a separate service. So the details of each problem have to pass from service to service, and the website can change again while they do.

His open-source prototype keeps the repair inside the running crawl, in the same process. He built it on Scrapy and web-poet page objects. The spider uses checks in code to notice a change, like a next-page link or a price it can no longer find. It pauses the crawl. An agent writes a new page object, which is the code for that layout. More checks in code test the fix. Then the spider reloads the new code and continues. If the checks show the agent can’t fix it, the issue goes to a developer.

In one run, a site started showing a second layout, as in an A/B test, and the agent wrote the code in about 40 seconds. He described 2 trade-offs. Problems that show only in the overall crawl statistics, not on a single page, are a little harder to detect this way. And the design puts an agent into production code, which has security risks.

A verifier that can only say yes or no

Joao Drummond founded a Brazilian company that builds scrapers. Its product runs live queries on Brazilian government portals, for example to check a vehicle’s license plate or get a certificate. His design is the opposite of Konstantin’s. Instead of one loop inside the crawl, he uses a team of agents. Each agent plays one role from a software team, such as product owner, tech lead, engineer or QA. One agent coordinates them all to fix broken scrapers. About 90% of the issues still come from people who report that a scraper stopped working.

The engineer agent runs on Claude Code. It writes the tests first, fills them with real data and runs real queries on the live site. An AI code review tool, CodeRabbit, also reviews the code.

The last step is a verifier that “doesn’t trust anyone”. It runs its own tests against the live API, checks the code and reads the PDFs the product generates. It can only answer yes or no, and if it isn’t sure, it says no. In the month before his talk, 65 of 92 issues completed the whole pipeline with no human input. Of the issues that reached the engineer agent, 82% went to production with no human changes. His first lesson was to keep the agent that does the work separate from the agent that checks it, or “split the doer from the judge”.

Both designs depend on checks that the agent doing the work doesn’t control.

Code calculates the numbers, not the model

Pravin Khandke builds a bank reconciliation system, which matches incoming bank payments with his company’s records. The payments arrive in bank files. The remittance details, which say what each payment is for, come by email. They can be in the email body, in an attachment, in a CSV file or in a PDF, so an LLM extracts them. Some details come from websites. His team scrapes those with Playwright scripts that repair themselves, instead of having an LLM read each page, because “each input and output token cost money”.

His warning was that LLMs fail silently. A model can return 4,523 instead of 4,532, and nothing shows an error. His rule was to “avoid trusting the numbers which are returned by LLM”. He advised calculating totals in deterministic code and using the LLM only for unstructured text. His team splits the input into stages, with one prompt for the image of a paper check and another for the remittance text. It also checks amounts against the customer’s payment history. A customer who paid $10,000 last time and now pays $1 million is a mismatch to investigate.

He gave the same warning about LLM-as-a-judge, where one LLM checks another LLM’s output, because “confidence is not a proof”. He didn’t say to avoid LLM judges. He advised against using the same LLM with only a small context.

Proof that the data is complete, and what it costs

Giancarlo Morillo, associate director of AI and data engineering at LexisNexis, leads a team that supplies data about people and organizations to other teams at LexisNexis. His team’s scraping router sends each job to one of 5 routes.

  • A vendor API for broad crawls and the largest jobs
  • Scraping with AI for high-quality structured data
  • Scrapling, an open-source adaptive crawler they run themselves
  • Playwright for sites that need a browser
  • LLM scraping, where an LLM reads the site directly, for the fastest setup, at a high cost

After collection, his team classifies the pages to find the relevant ones, and structures the data. An entity resolution step checks whether each person or organization already exists. A change engine then decides whether to create, update or skip each data point, or send it for human review. Every change is versioned and linked to its source. Classifying pages, along with changes to throughput, reduced processing from about 14 days to a day and a half, roughly 10 times faster.

His cost metric is cost per accepted record, “Total cost divided by usable output”. If spending stays the same but accepted records fall by half, the cost per record doubles. He expected a crawl that targeted only relevant pages to be faster and cheaper. It fetched 91% fewer pages and found 2% more entities, but it took about as long and cost 3 times as much. As he said, scraping is only part of the total cost.

Crawl settings also affected coverage. Scrapling first found fewer than half of his baseline entities, because of a 100-page cap per site. Raising the cap to 500 brought it to 81% of the baseline. The remaining 19% gap is why he uses a router, since “No single route wins every job”.

He also checks coverage against an independent source, Common Crawl, the open web archive I used in my text extraction test. He said its freshness and completeness are inconsistent, but it’s valuable as an independent reference. Comparing their data with it showed that hundreds of records were missing. The vendor had delivered the pages, but a bug in his team’s own extraction code made some records overwrite each other. “Collection metrics showed success. The independent check exposed the loss”.

His team compares routes on the same sites, time window and extractor, so the comparison is fair. It also keeps the collected pages, so it can fix an extraction error without crawling again. “Successful requests won’t tell you when the pipeline is going wrong”.

I found the same problem in my test of 371 top sites. 12% of them returned a 200 OK status for a page that doesn’t exist, so a scraper that checks only the status code records a failure as a success.

Logan Harless, CTO and co-founder of a New York scraping company, looked at cost one request at a time. He treats the choice of fetch method, from a plain HTTP request to a stealth browser, as a multi-armed bandit problem. The system mostly uses the method that has worked best so far, and sometimes tries the others to learn whether they work better now. His team uses Thompson sampling, one algorithm for this problem, in production for some jobs. He compared it with 2 alternatives, upper confidence bound and epsilon-greedy.

His key point was that sites react to scrapers. The more often you send requests the same way, the more often they fail, so old success rates stop being true. The algorithm has to forget old results. When success rates fell in his simulation, Thompson sampling with forgetting performed best. The metric to watch, he said, is cost per success, or “how much do I have to pay to get the data I want”. His advice for new teams was to “Work with the simplest thing until it doesn’t work anymore”.

Proof for buyers

Andrew Harris spent about a decade building ZoomInfo’s web data collection system, and he now runs a data advisory firm for companies that buy data. Buyers used to ask how much data a vendor could get, and how fast. In 2026, he said, the most common question is “what did you miss and how do you know and what are you doing about it?”

He named the 4 things buyers ask about most.

  • Coverage. Show what you cover out of the total. In his example, a vendor covers 3,000 counties and the crawlers for 40 of them break. A random gap is noise, but a gap that repeats in the same place becomes bias. So buyers ask about the pattern of the gaps, not only their size. He said vendors should also share their exclusion lists, the data they deliberately leave out.
  • Freshness. Freshness is a distribution, not a schedule, he said. “Monthly” says how often you crawl, not how old the data is. A median age of 3 days can hide a 99th percentile (P99) of 14 months, so 1% of the data is more than 14 months old.
  • Provenance. Buyers and their lawyers ask what you collected, when, and under what terms. Most teams keep the raw HTML, but they often can’t connect a delivered value back to the page it came from. A storage rule that deletes raw pages after 90 days breaks any promise of provenance. He said more and more deals fail because of this.
  • Reproducibility. Small changes add up. A 3% shift in the data can cost a customer a full sprint looking for a bug in their own code. His fix was to rerun extractors on the stored pages, compare the output of each extractor version, and publish a changelog.

Provenance and reproducibility are where he sees vendors fail most often. He said “late and correct is almost always going to beat fast and a little bit wrong”. He also advised vendors to hold back data they don’t trust, because “we’re not shipping this data” is an easier conversation than “we shipped some bad data”. Customers very rarely look at your fetch architecture, he said. They look at the data you deliver.

Proof of identity for bots

Akshay Philar, Zyte’s VP of engineering, described how bot detection has changed. In 2016, a user agent and a datacenter proxy were enough to scrape most sites, and detection was a yes or no decision. Now detection combines fingerprints with session behavior, like the number of requests, mouse movement and typing speed. The result is a score, “Almost like a credit score”.

That changed how Zyte chooses the way to fetch each page. Around 2017 and 2018, a request tried a datacenter proxy, then a residential proxy, then a mobile proxy. Now there are 10 to 100 times more options, but the choice has to be made within 3 to 20 milliseconds. So Zyte uses simple AI, such as behavior trees and utility AI, to make that choice for each request. Heavier models run separately. They learn from the results and update the rules that the simple AI uses.

He expects agents to multiply traffic, with “one human decision, n agent requests”. An agent that works for one person can visit thousands of sites. In his example, one decision took 5,000 to 10,000 requests, and a decision about a country or an economy could take a million. His answer was standards for 2 things. One is bot identity, like Web Bot Auth at the IETF. The other is authorization, which decides what an identified bot can access. Together, they could create what he called an “agent lane”, where only identified agents get access. His concern was keeping the web open, instead of letting a few large companies decide which bots get access.

Sites already allow or block bots by their user agent name. In my check of robots.txt rules on 544 top sites, 40 sites blocked OpenAI’s training crawler, GPTBot, but allowed its search crawler, OAI-SearchBot. Those rules work only when a bot identifies itself honestly.

Jason Grad, CEO and co-founder of the proxy provider Massive, said the difference between a residential proxy network and a botnet is consent. Each residential IP is someone’s personal device. He said Massive has never resold or combined IP pools from other providers. Every user joins through the company’s own apps and SDK. Its consent screen has no boxes checked in advance, and a user can pause or uninstall at any time. Massive also runs an abuse dashboard with live metrics.

He argued that a clear consent screen shouldn’t change much how many people agree to join. In his words, “There’s very little downside to making it easy for people to opt in, pause, and uninstall”. Until this year, he said, few buyers cared, and now buyers are starting to require consent. He listed questions a buyer can ask a provider, including “Can we see a consent screen?” and “How do you vet your partners?” A provider may keep some answers private, such as which apps carry its SDK.

The legal panel opened with a disclaimer, “We are not your lawyers”. Zyte’s senior counsel Victoria Vlahoyiannis moderated it. Zyte’s chief legal counsel Sally Anne Hinfey gave the EU view, and Scott Lashway, a privacy and security lawyer, gave the US view.

Sally Anne Hinfey’s practical advice was to keep an evidence file, a record that shows how the data was collected. It includes the URLs, the time of each crawl and the collection method. She also recommended recording which sources required a login and which terms you accepted. Record them at the time of the crawl, because the terms can become more restrictive later. On personal data, she said recent EU guidance allows scraping at scale for AI development when there’s a “legitimate interest”, which means you really need the personal data you collect. In practice, she said to collect only the personal data you need, filter out the rest at the source, and avoid sensitive data, such as financial data, health data and children’s data.

What to build and measure

  • A verifier that answers only yes or no. Before AI-written code goes to production, a separate verifier tests it on live data. If it isn’t sure, the answer is no (Joao Drummond).
  • AI extraction on a sample of pages. Compare it with the output of your extraction code to find site changes the code misses (Shane Evans).
  • Checks in code that notice a layout change. A missing next-page link or price can be detected without an LLM, before bad data reaches the customer (Konstantin Lopukhin).
  • Totals calculated in code. Use the LLM to extract values, and use code for every calculation (Pravin Khandke).
  • Cost per accepted record. Divide total cost by the number of usable records, not by pages fetched (Giancarlo Morillo).
  • Coverage against an independent source. Compare your records with a source like Common Crawl to find missing records that your own metrics don’t show (Giancarlo Morillo).
  • Cost per successful request for each fetch method. Choose the method by cost per success, and update the choice as success rates change (Logan Harless).
  • Proof for buyers. Report the pattern of coverage gaps, freshness as a distribution with the P99, a link from each value to its source page, and the changes between extractor versions (Andrew Harris).
  • An evidence file for every crawl. Log the URLs, crawl times, collection method and the terms you accepted at the time (Sally Anne Hinfey).

And keep a person for the final or uncertain decisions. Every agent pipeline in these talks still had one.

The talks

Doccraft writes research and tutorials like this for web data and AI infrastructure companies.