Focus area
RAG and retrieval pipelines
Doccraft covers how to get clean text from web pages into a RAG pipeline, keep the index up to date on every re-crawl, and defend it against prompt injection, instructions hidden in pages. It writes for the companies that turn web data into LLM-ready data for RAG and AI search.
Research and guides
- Common Crawl's WET text vs trafilatura vs Resiliparse, retested
Hugging Face's FineWeb team found Common Crawl's ready-made text too noisy and extracted their own. I retested that choice on 2,000 pages from the September 2026 crawl, with FineWeb's own quality filter.
- Skip unchanged pages on a re-crawl: how many top sites return 304
A conditional request lets a site answer "not modified" with no page body. I tested it on 368 top homepages and 224 inner pages. About 40% returned a 304, and half sent nothing to check.
- Hidden prompt injections that target scrapers: what a 2026 study found
A 2026 study found 15,387 prompt injections hidden in real web pages, aimed mostly at crawlers and scrapers. What they say, where they hide, what the top 1,000 sites do, and how to handle them.
Published for clients
- How to ground a LlamaIndex RAG app in fresh web data (opens in a new tab)blog.apify.com
- How to feed Amazon Bedrock Knowledge Bases with live web data using Bright Data (opens in a new tab)brightdata.com
- AI agents hallucinate prices: LLM grounding with live data (opens in a new tab)blog.apify.com
- How to Build a Data Retriever With Dify AI Agents (opens in a new tab)scrapingbee.com
- How to Build a RAG Pipeline with Bright Data and Weaviate (opens in a new tab)brightdata.com
- Monitor website changes in Python without false alerts (opens in a new tab)evomi.com
- Data Pipeline Architecture for Businesses Explained (opens in a new tab)brightdata.com