OpenAI, Google, Microsoft and Anthropic back one robots.txt AI-training opt-out token, __usage_ai-training; Anthropic has started, Microsoft honors it from Feb 1, 2027

Background and the industry pain point

For two years, a site that wanted to refuse AI training had to list each crawler in robots.txt by name, such as GPTBot, ClaudeBot and Google-Extended, adding a new group for every model maker and often missing ones it had never heard of. Publishers have sued AI companies over scraping, and the EU Code of Practice for general-purpose AI models treats robots.txt as the main way to express machine-readable preferences. Four leading providers backing the same signal on the same day is the industry's first shared entry point for opting out of training.

How the mechanism works

According to IETF draft draft-kelley-aipref-training-preference-signal-00, there is no new grammar. The policy token __usage_ai-training goes in a User-Agent line and reuses RFC 9309 path matching and longest-match precedence. Disallow expresses a preference against training, Allow expresses one in favor, and a file without the group expresses none; wildcard User-Agent: * groups explicitly do not match. "AI training" takes the IETF aipref vocabulary definition, modifying a generative model's parameters, and excludes purely classification or ranking models. The draft stresses the token is not a crawl directive, existing crawler rules still apply, and private agreements can override it. Google says that if a site does not address Google-Extended directly, an opt-out via the token will be read as disallowing Google-Extended; Microsoft extends the training preference to individual pages and adds a separate AI Grounding preference with a new DisallowExtended value.

Benchmarks and measured results

This is a standards and policy story, so there are no benchmarks. None of the companies has published figures on expected adoption or the effect on training data volume.

Getting started

A site that wants out of training can add User-Agent: __usage_ai-training and Disallow: / to robots.txt, with Allow lines for paths it wants to keep in. The token does not stop crawling, so rate limits or crawler blocks still need per-crawler rules. Google and Microsoft both warn that opting out of training may reduce how often content appears in AI experiences, so avoid switching off Google-Extended or Bing grounding by accident. Sites relying on Bing's NoArchive/NoCache should move to the new preferences before March 1, 2027. With rollouts spread into 2027, the safest setup for now is to keep the new token alongside existing per-crawler rules.