For two years, webmasters and publishers played a losing game of whack-a-mole against artificial intelligence crawlers. Every new foundation model brought another hungry bot scraping articles, artwork, and databases without clear consent. That chaos just hit a historic turning point.
__usage_ai-training token in robots.txt submitted to the IETF by Anthropic's Kevin Kelley.Key Takeaways
- A Rare Big Tech Truce: Anthropic, OpenAI, Google, and Microsoft have jointly committed to support a single, standardized
robots.txttoken:__usage_ai-training. - Tackling Crawler Sprawl: Website operators no longer need to maintain dozens of individual user-agent blocks for GPTBot, ClaudeBot, Google-Extended, and third-party scrapers. One rule covers the major labs.
- Training vs. Search Grounding: The new directive governs model weight training only. Participating engines will still index pages for web search and live conversational citations, protecting referral traffic.
- Adoption Timelines: Anthropic implemented the directive immediately. OpenAI rolls it out in coming months. Microsoft goes live on February 1, 2027. Google activates support in early 2027.
The Chaos of Modern Crawler Sprawl
The open web was never built for large language models. The Robots Exclusion Protocol dates back to 1994. It was designed for a simple trade: search engines index your pages in exchange for sending qualified human traffic back to your site. Generative AI broke that bargain. Foundational scrapers swallowed petabytes of creator work to train models that answer queries directly, cutting out the publisher entirely.
Site owners retaliated by packing their robots.txt files with defensive blocks. But keeping up became impossible. A typical server configuration required separate disallow rules for OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Common Crawl's CCBot, and ByteDance's Bytespider. Miss one bot, and your content was scooped up anyway.
Worse, publishers faced a painful dilemma. Blocking Google's crawler to protect intellectual property risked wiping out visibility on Google Search. Nuance was nonexistent. The web desperately needed a universal standard.
Enter the Unified __usage_ai-training Token
The breakthrough proposal originated inside the Internet Engineering Task Force (IETF). Kevin Kelley, technical lead for standards at Anthropic, authored an Internet-Draft within the IETF's AI Preferences (AIPREF) working group. Rather than inventing an entirely new protocol that server software might reject, the proposal extends standard robots.txt syntax by introducing a reserved pseudo-user-agent token: __usage_ai-training.
When participating AI companies scan a domain, their crawlers check for this designated token. If disallowed, the provider commits to excluding that site's text and media from training foundational AI models.
User-Agent:__usage_ai-training
Disallow:/
# Optional: Allow specific public directories (e.g. press kits or open datasets)
Allow:/press-releases/
Allow:/public-research/
This implementation offers remarkable flexibility. A digital publication can block its investigative reporting archives while leaving its press releases or job boards open for public datasets. Granular control returns to the content creator.
The Crucial Divide: Model Training vs. Search Grounding
The single most important technical detail for creators is the line between model training and retrieval grounding. They are fundamentally different operations:
- Model Training: Ingesting web text into multi-billion parameter neural networks so the model learns concepts, patterns, and grammar. This permanent absorption cannot be undone once training weights are baked.
- Grounding & Retrieval: Fetching a live web page in real time to cite facts, check current news, or provide a link inside ChatGPT Search, Microsoft Copilot, or Google AI Overviews.
Under the unified standard, opting out via __usage_ai-training signals that you do not want your prose turned into model weights. It does not erase you from live web search or conversational answers that link back to your domain. Microsoft specifically confirmed that Copilot will honor the training block while continuing to reference live pages and drive direct traffic. That separation preserves the publisher's business model while protecting original reporting.
| AI Provider | Enforcement Date | Scope of Protection | Live Search Grounding |
|---|---|---|---|
| Anthropic | Immediate (Live Now) | All Claude model training | Separate web fetch rules |
| OpenAI | Coming Months (Late 2026) | GPT training datasets | Preserved in ChatGPT Search |
| Microsoft | February 1, 2027 | Copilot AI training | Bing indexation unaffected |
| Early 2027 | Mapped to Google-Extended | Google Search ranking intact |
What Website Owners Should Do Right Now
Should you immediately rewrite your robots.txt file? The answer is nuanced. The joint announcement represents voluntary corporate commitments rather than a ratified RFC standard. That means smaller independent scrapers, foreign AI labs, and commercial data brokers will not honor it automatically.
For maximum protection, security specialists recommend a hybrid approach. Keep existing explicit blocks for high-volume scrapers (like CCBot and Bytespider) while appending the new __usage_ai-training directive at the top of your configuration. This establishes your clear legal and technical preference ahead of the 2027 enforcement deadlines.
The agreement between Anthropic, OpenAI, Google, and Microsoft will not solve every tension between Silicon Valley and digital creators. Licensing battles and copyright lawsuits will continue in courtrooms. But for the millions of independent writers, photographers, and developers who simply want a clear, dignified way to say "no" to AI training, this standard delivers the first real peace of mind in years.
Frequently Asked Questions
What is the __usage_ai-training token in robots.txt?
The __usage_ai-training directive is a standardized policy token backed by Anthropic, OpenAI, Google, and Microsoft. It allows web publishers to opt out of having their site content used to train generative AI foundation models through a single universal declaration.
Does blocking __usage_ai-training hurt my Google Search ranking?
No. Google explicitly stated that this directive governs generative model training datasets (like Google-Extended). It does not affect standard Googlebot indexing or organic search ranking.
What is the difference between AI training and search grounding?
Training permanently absorbs web content into an AI model's mathematical weights. Grounding retrieves live web pages in real time to cite sources and provide links during user conversations. The unified token blocks training while allowing live citations and referral traffic.
When do Google and Microsoft start honoring the new standard?
Anthropic honors the directive immediately. Microsoft begins enforcement on February 1, 2027. Google announced it will activate support in early 2027.
Reader Discussion