What top websites tell AI crawlers: robots.txt rules on 544 sites, bot by bot

AI companies don’t run one crawler each anymore. OpenAI, Anthropic and Perplexity each document separate user agents for training, for search and for fetching a page when a user asks. A site can allow one and block another in robots.txt.

I checked what the top websites actually say to each of them. I used the 544 real websites from my test of the top 1,000 websites, and read each site’s robots.txt for 15 user agents. I also checked the newer signals that sites use for AI, like llms.txt, Content-Signal lines and TDMRep. I ran it from Mumbai on October 12, 2026.

How I tested#

For each site, I downloaded robots.txt and checked whether each user agent may fetch the homepage. I used Protego (opens in a new tab), the robots.txt parser that Scrapy uses, so the rules are read the way a production crawler reads them.

393 of the 544 sites returned a robots.txt I could read. Of the rest, 56 returned a 403 to the request for robots.txt itself, 40 served an HTML page at that address, and 19 returned a 404, which means no rules.

This script checks a site for several user agents at once.

import httpx
from protego import Protego

headers = {"User-Agent": "MyScraperBot/1.0 (https://example.com/bot; you@example.com) httpx/0.28"}
agents = ["Googlebot", "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "CCBot"]

for site in ["https://medium.com/", "https://www.nytimes.com/"]:
    robots = httpx.get(site + "robots.txt", headers=headers, follow_redirects=True).text
    rules = Protego.parse(robots)
    allowed = [a for a in agents if rules.can_fetch(site, a)]
    print(f"{site:26} allows: {', '.join(allowed)}")

https://medium.com/        allows: Googlebot, OAI-SearchBot, ChatGPT-User, CCBot
https://www.nytimes.com/   allows: Googlebot

Medium blocks OpenAI’s training crawler but allows its search crawler. The New York Times allows only Googlebot from this list.

The results#

The purpose of each user agent comes from its company’s own documentation. “Named” means the site has a group for that user agent, not only the general * rules.

Shares are of the 393 sites with a readable robots.txt.

No site blocked Googlebot. All 68 sites that blocked GPTBot still allowed Googlebot. Some sites block AI crawlers, but none of them block search.

Sites treat training differently from search. OpenAI’s training crawler was blocked on 17.3% of sites, its search crawler on 7.4%. Anthropic’s bots show the same pattern, at 17.0% and 8.7%. 40 sites blocked GPTBot but allowed OAI-SearchBot, so they can appear in ChatGPT’s search results while opting out of training.

The most blocked crawlers were CCBot and Bytespider, at 19.8% and 19.1%. CCBot builds Common Crawl, the open dataset that many AI training sets start from.

What the AI companies say they follow#

A robots.txt rule is a request. Each company documents which of its user agents follow it, and the user-triggered ones are different.

• OpenAI says each setting “is independent of the others”. For ChatGPT-User, it says that “because these actions are initiated by a user, robots.txt rules may not apply”.
• Perplexity says that because a user requested the fetch, Perplexity-User “generally ignores robots.txt rules”.
• Meta says its user-triggered fetcher, Meta-ExternalFetcher, “may bypass robots.txt rules”.
• Anthropic documents 3 bots and what blocking each one does. Blocking ClaudeBot signals that a site’s future content should be excluded from training.
• Google says Google-Extended “does not impact a site’s inclusion in Google Search”. It has no crawler of its own. Googlebot does the crawling, and the token only controls how the content is used.

So the 9% to 10% of sites that block the user-triggered agents may not get what they expect. For OpenAI, Perplexity and Meta, the companies themselves say those rules may not be followed. Anthropic describes blocking Claude-User as a way to stop it.

The newer signals#

llms.txt was far more common than I expected. I counted a file only if it looked like the format, Markdown with a heading or links. That check removed responses that weren’t the format at all, like a 1-pixel GIF and a plain IP address.

Content-Signal comes from Cloudflare’s Content Signals Policy (opens in a new tab), published in September 2025. Cloudflare said its managed robots.txt was already turned on for “over 3.8 million domains”, but only 13 of these top sites had the lines. 5 of them, including Cloudflare, NVIDIA and Oxylabs, set all 3 signals to yes. Patreon, Hostinger and Trend Micro allow search and AI input, and say ai-train=no.

What this means for a data team#

• Check the rules for the user agent you run as. A site can allow your crawler under * and block a named AI crawler, or the opposite. Parse robots.txt with a standard parser like Protego, not by searching for “Disallow”.
• Read a block on training crawlers as a statement about training. 20% of these sites block GPTBot or ClaudeBot, and all of them still allow Googlebot. If you collect data to train models, those sites have said no to that use.
• Identify your crawler honestly. The AI companies above publish their user agents, and Perplexity also publishes its IP ranges, so sites can tell the bots apart.

Limits of this test#

• The homepage only. Many sites block or allow specific paths, so rules for deeper pages can differ.
• The rules as served to my client. A site can serve a different robots.txt to different clients, and 56 sites returned a 403 for it.
• What sites say, not what bots do. This test reads the rules. It doesn’t measure which crawlers follow them.

Data and code#

• Results for all 1,000 domains
• The scan and the analysis
• The example script