SHOMEDIAOn October 30, Google, OpenAI, Microsoft and Anthropic backed a common way for websites to express whether their content may be used to train generative AI models. The proposal introduces a shared policy token in robots.txt, reducing the need to maintain a different instruction for each AI provider. The mechanism is described in an IETF Internet-Draft by Anthropic engineer Kevin Kelley. It remains a draft, not a ratified internet standard or a legal obligation.
A site operator can create a group headed User-Agent: __usage_ai-training and place Disallow: / beneath it to express an opt-out from training on the entire site, with narrower path exceptions where needed. The distinction is easy to miss: the token concerns the use of content for model training, not permission to crawl pages. It does not replace ordinary crawler rules, and a provider that has not agreed to honor the signal may ignore it.
Different providers, different rollout dates
Google's Search Central announcement targets support in 2027. If a site opts out using the shared token but has no explicit Google-Extended instruction, Google plans to interpret the new signal as a Google-Extended disallow. Company-specific controls will remain available for tailored choices. OpenAI expects to implement the signal in the coming months, while Anthropic has already begun support; there is no single industry launch date.
Microsoft's Bing announcement adds an operational deadline. New training and grounding preferences begin taking effect as Bing crawls from February 1, 2027. By March 1, websites relying on NoArchive or NoCache should move to the new controls: after that date, those older tags will no longer count as training or grounding preferences. Grounding means using source content to inform an AI response and is separate from training a model.
The decision for publishers and SEO teams
Publishers need to decide independently whether to allow search indexing, AI answer grounding and model training. Blocking a search crawler simply to opt out of training may sacrifice discoverability unnecessarily. Before changing robots.txt, teams should review existing GPTBot, Google-Extended and other provider-specific rules, then agree on a policy with editorial and rights holders.
The new token cannot remove content from historical training datasets, force every AI provider to comply or guarantee traffic from AI answers. Google and Microsoft still distinguish between different uses of web content, and the proposed standard is under discussion. Until implementation is confirmed for each provider, existing rules remain relevant.
Комментарии