Focus area
Web search and data APIs for AI agents
Doccraft tests how AI agents get live web data, through search APIs, scraping tools, MCP servers and protocols like WebMCP and UCP. It shows how reliable that data is and what production teams build around it. It writes for the companies that build search and web data APIs for AI agents.
Research and guides
- Shopify WebMCP: the cart tool tells AI agents not to ask, and only checkout is marked consequential
28 of 30 Shopify stores I checked register the same 12 WebMCP tools for browser agents. The add_to_cart description says "do NOT ask the user to confirm", and none of the 12 is marked consequential. Only complete_checkout is. In 40 runs, Claude called the tools twice and never went to checkout.
- rea vs Playwright MCP for finding a web page's hidden API, tested on 8 pages
I gave Claude 8 real pages and asked it to find the API each page loads its data from, with rea 6.8.0 and with Playwright MCP. Both found the same APIs. rea's 229 tool definitions added 103,531 tokens to every turn, unless the client defers them, as Claude Code does. Network captures of up to 3 million characters ended 4 of 16 runs.
- Playwright MCP vs Chrome DevTools MCP vs agent-browser: same accuracy, but tokens per task ranged from 4.1K to 95K
I ran the same Claude agent on 19 real browser tasks, twice, through Playwright MCP, Chrome DevTools MCP and 5 agent-browser setups. All 7 got every answer right. The median task read 31K tokens with Playwright MCP, 46K with Chrome DevTools MCP, and 4.1K to 95K with agent-browser, depending on how it was loaded. In dollars, the typical task cost under 4 cents in every setup.
- Exa Lite found 50 of 53 answers, 1 fewer than Exa Auto, in less than half the time
Exa 3.0 added Exa Lite, a search type at $4 per 1,000 requests. I tested Lite, Auto and Deep on the same 53 questions with known answers. Lite found 50, Auto and Deep found 51, and Lite's median response took 746 ms against 1,732 ms for Auto.
- Shopify UCP catalog for AI agents: page prices match, but sold-out sizes are hidden by default
All 34 Shopify stores I checked publish a UCP catalog for AI agents, with no API key. Prices matched the product pages, and stock matched for 50 of 50 sizes. Still, 3 defaults can give a data team missing or wrong data. Sold-out sizes are missing, prices come in the seller's currency, and anonymous calls from one IP address stop at about 100 per store per hour.
- Browser Use found the right price on 29 of 34 stores, and duplicate product names caused most misses
I gave the Browser Use agent a store homepage and a product name on 34 real stores, twice. It returned the right product and price for 29 and 28. Most misses were other listings with the same name. Each product took about a minute and about a cent.
- Cloudflare's Web Search API, tested on 53 questions: the provider you pick matters far more than the route
I tested Cloudflare Web Search API with Exa, Linkup and Ceramic.ai, and called each provider directly, on 53 questions with known answers. Cloudflare matched direct calls for Exa and Linkup, changed Ceramic's results and dropped untitled results. The providers were very different.
- MCP's context problem, measured: tool definitions vs tool results
Apify's CEO argued at Prague Crawl 2026 that MCP isn't the problem, the agents are. I measured 6 MCP servers. 95 tool definitions took 15,950 tokens, and one page snapshot took 31,565.
- What top websites tell AI crawlers: robots.txt rules on 544 sites, bot by bot
I checked the robots.txt of 544 top websites for 15 AI and search user agents. None blocks Googlebot, 17% block GPTBot, and many sites treat AI training, AI search and user requests differently.
- AI in web scraping: what 2026 conferences agree on
I studied more than 60 web scraping talks, podcasts and AMAs and read 10 papers. The clearest finding is that teams use AI to build scrapers, not to run them.
- Hidden prompt injections that target scrapers: what a 2026 study found
A 2026 study found 15,387 prompt injections hidden in real web pages, aimed mostly at crawlers and scrapers. What they say, where they hide, what the top 1,000 sites do, and how to handle them.
- What Oxylabs learned from its AI parser agent
At OxyCon 2026, Oxylabs openly shared why its engineers stopped using an AI parser agent, and what its second agent did differently.
Published for clients
- Microsoft Agent Framework MCP Server: Add Live Web Search and Scraping with Bright Data (opens in a new tab)brightdata.com
- MCP servers for web scraping: carry control, not data (opens in a new tab)scrapingbee.com
- DSPy MCP Server: Build a Web Research Agent with Bright Data (opens in a new tab)brightdata.com
- AI agent vs. MCP server: which one should you build? (opens in a new tab)blog.apify.com
- Is There a Google Search API? Options After Custom Search (opens in a new tab)brightdata.com
- AI agent infrastructure stack: from web data to retrieval (opens in a new tab)blog.apify.com
- What Is WebMCP? How It Works and What It Means for AI Agents (opens in a new tab)brightdata.com