RSL licensing for AI crawlers: 4 of the top 1,000 sites publish it, and 1 of its 24 featured supporters

RSL, Really Simple Licensing, lets a site tell AI crawlers in robots.txt what they may do with its content and what it costs. Of the top 1,000 websites, 4 had a License: line on October 28, 2026, the same 4 as in my October 12 scan. Only 2 of them, The Guardian and Bild, served a license file a crawler could read.

RSL says more than 1,500 organizations endorse it. The RSL Collective’s site features 24 of them, including Reddit, Yahoo, Ziff Davis and Vox Media. I checked their 33 main domains, and only Medium had a License: line. Its license file returned 403 to my crawler.

The Guardian’s robots.txt also shows a problem. Its License: line is placed inside a User-agent group, so under the grouping rules RSL uses, it applies only to the 37 crawlers in that group, all of them blocked from the site.

What RSL is#

The RSL 1.0 specification (opens in a new tab) became a Recommendation on December 10, 2025, after a launch in September 2025 (opens in a new tab). A site adds a line like License: https://example.com/license.xml to robots.txt. The file it points to is XML, and it can permit or prohibit uses such as ai-train, ai-input and search, and name a payment model such as a subscription, pay per crawl or attribution.

The same file can also be linked from a page’s HTML, an HTTP header or an RSS feed. This test covers robots.txt and each homepage’s HTML and headers.

What the top 1,000 sites publish#

The scan covers the same 1,000 domains as the October 12 scan of AI crawler rules. 547 of them served a readable robots.txt.

• The Guardian permits AI training and AI input under a subscription, with a link to licensing.theguardian.com. It prohibits every other use.
• Bild prohibits AI training and AI input, and offers no payment option. Bild uses RSL only to refuse AI use.
• Medium points to license.xml, but the file returned HTTP 403.
• UNESCO points to its normal terms of use page, which isn’t an RSL file.

None of the 533 homepages that loaded linked to an RSL file in their HTML or headers either. For comparison, my October scan of the same list found an llms.txt file, a guide to a site’s content for AI tools, on 98 of the 544 domains that served a website. RSL is far behind it.

1 of 24 featured supporters publishes it#

The RSL Collective’s site (opens in a new tab) features 24 supporters. They have 33 main domains, such as cnet.com, pcmag.com and ign.com for Ziff Davis.

29 of the 33 served a readable robots.txt, and only Medium’s had a License: line. The other 4, Nolo, PCMag, Stack Overflow and MIT Press, blocked the request, so their rules are unknown.

Support is not the same as deployment. Some supporters, like Raptive, VerticalScope and Evolve Media, run networks of other people’s sites, so their main domain shows little about those sites.

Why the position of the License line matters#

Both The Guardian and Medium put the License: line at the end of robots.txt, after a group that blocks AI crawlers.

The RSL spec says a License: line inside a User-agent group “applies only to clients that select that group”, and one outside any group applies to all. For where a group ends, it uses RFC 9309 (opens in a new tab), the robots.txt standard, which says “the group is terminated by a user-agent line or end of file”. A blank line doesn’t end it.

Under that rule, The Guardian’s license belongs to the last group, 37 crawlers that are blocked from the whole site. GPTBot isn’t in that group, so it sees no license at all. A parser that treats the blank line as the end of the group reads it as a license for everyone. The spec’s own examples avoid this by putting the License: line before the first User-agent line.

Who honors it#

The publishers’ side is only half of RSL. The other half is AI companies reading the file and following it. When RSL 1.0 became a standard in December 2025, none of the major AI model companies had agreed to honor it, as Shelly Palmer noted (opens in a new tab) at the time. As of October 2026, I haven’t found a public commitment since then.

The big CDNs support it. Fastly backed RSL at its launch, and the RSL 1.0 release (opens in a new tab) added Cloudflare and Akamai. Enforcement at the CDN would let a site block crawlers that haven’t agreed to its terms, whether or not they read the file. Cloudflare already runs pay per crawl (opens in a new tab), which responds with HTTP 402 to the AI crawlers a site chooses to charge, and it said it wants to include a standard license in those responses. A research crawler like mine would likely not be charged, so this scan can’t measure how many sites use it.

What to do with it#

If you run a crawler, parse License: lines the way RFC 9309 groups them, and read the terms for the group your user agent selects. Treat a prohibition the way you’d treat a Disallow, and treat an unreadable license file as unknown terms, not as permission. Log the terms you saw and the date, so you can show what applied when you crawled. The checker below does the first part.

If you publish, put the License: line before the first User-agent group, or inside the exact group you mean, so every parser reads it the same way. Make sure the license file itself isn’t behind the bot protection you use for pages, and serve it as application/rsl+xml.

The checker#

This script reads a site’s robots.txt, finds each License: line and the group it belongs to, and prints the terms from the license file. It produced the first image above.

"""Check a site for RSL: License lines in robots.txt (with the User-agent group each one belongs to, per RFC 9309)
and the terms in the license file. Run with: uv run --with httpx python rsl_check.py theguardian.com bild.de"""
import sys
import xml.etree.ElementTree as ET
from urllib.parse import urljoin
import httpx

UA = "DoccraftResearchBot/1.0 (https://doccraft.dev; hello@doccraft.dev) httpx/0.28"
NS = "{https://rslstandard.org/rsl}"


def license_lines(text):
    """(license URL, user agents of its group or None for global). A group ends only at the next User-agent line."""
    out, agents, in_rules = [], None, False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, val = (s.strip() for s in line.split(":", 1))
        if key.lower() == "user-agent":
            agents = [val] if agents is None or in_rules else agents + [val]
            in_rules = False
        elif key.lower() == "license":
            out.append((val, agents))
        elif key.lower() in ("allow", "disallow"):
            in_rules = True
    return out


def terms(body):
    root = ET.fromstring(body)
    for lic in root.iter(f"{NS}license"):
        for tag in ("permits", "prohibits"):
            for e in lic.findall(f"{NS}{tag}"):
                yield f"{tag} {e.get('type')}: {e.text.strip()}"
        for p in lic.findall(f"{NS}payment"):
            yield f"payment: {p.get('type')}"


with httpx.Client(headers={"User-Agent": UA}, follow_redirects=True, timeout=20) as client:
    for domain in sys.argv[1:]:
        robots = client.get(f"https://{domain}/robots.txt")
        print(domain)
        for url, group in license_lines(robots.text):
            scope = "all crawlers" if group is None else (f"User-agent: {group[0]}" if len(group) == 1 else f"a group of {len(group)} user agents")
            r = client.get(urljoin(f"https://{domain}/", url))
            print(f"  License: {url}  (for {scope})")
            if r.status_code != 200 or not r.text.lstrip().startswith("<rsl"):
                print(f"    license file: HTTP {r.status_code}, not readable as RSL")
                continue
            for t in terms(r.text):
                print(f"    {t}")

Test setup and limits#

• Sites. The top 1,000 domains from the Tranco ranking, as in my earlier scans, plus the 33 main domains of the supporters listed on rslcollective.org. 11 of those were already in the top 1,000, so 1,022 domains in all. One request per file, from Mumbai, on a home connection, with httpx 0.28 and a user agent that names Doccraft. October 28, 2026.
• Where I looked. robots.txt, and each homepage’s HTML and Link header. RSL can also be on article pages, in RSS feeds and in media files, and a site could use those without a License: line in robots.txt.
• Blocked requests. 4 supporter domains blocked my crawler, and Medium’s license file returned 403. I didn’t retry with other user agents, so I don’t know Medium’s terms.
• Supporters. The 24 are the supporters the RSL Collective features on its site, out of more than 1,500 that RSL says endorse it. I picked each one’s main domains myself. A network’s own domain doesn’t show what its member sites do.

Data and code#

• Results for all 1,022 domains, the scan, the checker and the supporters summary
• The VHS tapes behind the terminal images