One token instead of a list

Until now, a publisher who did not want to be trained on had to keep a list. One robots.txt group for GPTBot, another for Google-Extended, another for each new provider that appeared, each with its own name and its own documentation. The October 30 announcements add, in principle, one shared group alongside that list:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read under the draft, that says: a preference against AI training everywhere except the press kit. The mechanism adds no new directive and no new grammar. It reuses the path matching and longest-match precedence that robots.txt already has under RFC 9309, which is the point.

The compatibility argument is straightforward. Under the draft, parsers that do not recognize the new policy token should ignore that group, leaving their existing crawler rules unchanged. The proposal introduces no new robots.txt directives or grammar. That makes the change designed to be backward-compatible, although publishers should still test their complete robots.txt file rather than assume every deployed parser behaves identically.

That portability is the real offer. A small site gets a training preference on record with every participating provider in one edit, once those providers implement it, instead of tracking which lab launched which crawler token this quarter.

Four announcements, four different clocks

The coordination is genuine. The implementation commitments are not identical.

Anthropic is furthest along, at least in its own account. Kevin Kelley, the draft's author and an Anthropic employee, told the AIPREF mailing list that Anthropic has already started implementing support for the draft. He also anticipated complications and edge cases during implementation. That is a progress statement, not confirmation that every Anthropic service is already honoring the signal. OpenAI committed to the signal with no date beyond "the coming months" and a promise of a second update with implementation details.

Google set its target in 2027, and said sites that specifically allow Google-Extended remain eligible to appear, via grounding, in Gemini app responses. Google also said that if a site opts out with the shared policy token and does not explicitly specify Google-Extended, Google plans to interpret that as a Google-Extended disallow, while its existing company-specific controls keep working alongside the shared signal.

That fallback has an important exception: if a publisher already specifies Google-Extended separately, Google's existing provider-specific control continues to apply. Adding the shared token therefore does not automatically override every existing Google setting. Publishers should inspect their current Google-Extended rules before assuming the new group will change Google's treatment of every path.

Microsoft gave the most detailed calendar. It begins honoring the new preferences on February 1, 2027, as it crawls pages. There is a separate transition for publishers already using its older NoArchive or NoCache controls: temporary grounding preferences become available in Bing Webmaster Tools on December 15, 2026, for those affected sites. They must configure replacement preferences by March 1, 2027, to maintain their chosen grounding restrictions. Any temporary preference still in place expires on September 30, 2027, or earlier if Microsoft recrawls the page and discovers the replacement setting.

A publisher can publish the preference now, but should not mistake publication for protection. Anthropic says implementation has begun; OpenAI, Microsoft and Google have later implementation targets. Until each provider confirms that its systems are reading the signal, existing provider-specific controls remain important. The shared token is an additional preference, not a replacement for controls that already operate.

What the signal does not do

Three limits are written into the draft itself, and they matter more than the headline.

First, it does not control crawling. The draft is explicit that content-usage preferences and crawler traffic management are independent. A `Disallow` under the policy token expresses a preference about training; it is not an instruction to stay out. A consumer may choose not to fetch material it cannot use, but that is its choice, not the site's command. Anyone whose goal is to keep bots off the site still needs ordinary crawler rules. Publishers should also distinguish training controls from search controls: OpenAI's GPTBot is associated with potential model-training use, while OAI-SearchBot is used to surface websites in ChatGPT search. Blocking OAI-SearchBot removes a site from ChatGPT search answers, although it may still appear as a navigational link. Blocking GPTBot is a separate decision and does not, by itself, require giving up that search visibility.

Second, it is a preference, not a prohibition with an enforcement mechanism. The vocabulary draft behind it states that it does not ensure preferences are followed. Private agreements may override what a site expresses through the token. And nothing in any of the October 30 posts addresses material already collected: the signal is not retroactive, and adding the line says nothing about datasets or models that already exist.

An IETF draft is not an IETF standard

This is an individual Internet-Draft, version 00, dated the same day as the announcements and expiring on May 3, 2027. The datatracker does not endorse it and it has no formal standing in the IETF process. The definition of AI Training it relies on comes from a working-group vocabulary draft that, by its own note, does not yet reflect working-group consensus. That does not make the effort unserious. It does mean the definition of the thing being opted out of can still move.

Training and grounding are not always separate in practice

Microsoft's announcement makes one distinction particularly clear: training and grounding are different uses. Training modifies a model's learned parameters. Grounding retrieves content at answer time to inform a generated response, potentially giving a publisher a citation and a referral visit.

Microsoft offers separate preferences for the two. Publishers can disallow training while allowing grounding, or choose DisallowExtended, which restricts grounding use to Microsoft's own products rather than its third-party grounding services.

The shared IETF token itself addresses training, not grounding. But Google's planned implementation adds an important complication. Google says that when a publisher disallows training using the shared token and does not explicitly specify Google-Extended, it will interpret the shared preference as a Google-Extended disallow. Google's existing Google-Extended control covers both Gemini training and grounding.

That means the practical effect can differ by provider. A publisher using the shared token could preserve grounding with Microsoft by setting its separate grounding preference, while Google's stated fallback could also remove eligibility for grounding in Gemini Apps unless the publisher explicitly configures Google-Extended.

For publishers, the distinction is commercial as well as technical. Refusing training is a decision about how content contributes to future models. Refusing grounding can affect whether that content informs AI-generated answers and receives citations or referral traffic. A shared training preference does not guarantee that the consequences of opting out will be identical across providers. The existing provider-specific controls still matter.

Sources

  1. [1] Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026 (primary — read first-hand: Google plans to support the shared control in 2027; a shared-token disallow will be interpreted as a Google-Extended disallow unless Google-Extended is explicitly specified; existing company-specific controls continue to function alongside the new signal; Google-Extended has been offered since 2023)Read source
  2. [2] OpenAI, "Making it easier for websites to express AI training preferences", October 30, 2026 (primary — read first-hand: OpenAI will honor a common signal in robots.txt for foundational generative AI training; the effort is voluntary; it aims to implement the signal in the coming months; existing crawler controls continue to apply)Read source
  3. [3] OpenAI, "Overview of OpenAI Crawlers" (primary — read first-hand: GPTBot crawls content that may be used in training generative AI foundation models; OAI-SearchBot is used to surface websites in ChatGPT search; sites opted out of OAI-SearchBot are not shown in ChatGPT search answers but can still appear as navigational links)Read source
  4. [4] Microsoft Bing, "Your content, your choice: clearer preferences for training and grounding", Krishna Madhavan, October 30, 2026 (primary — read first-hand: separate AI Training and AI Grounding preferences for web content used in Bing, Copilot and Microsoft 365; DisallowExtended limits grounding use to Microsoft-operated experiences and excludes third-party grounding services; new preferences honored as pages are crawled from February 1, 2027)Read source
  5. [5] Microsoft, "FAQ - New AI Preferences", Bing Webmaster Tools help (primary — read first-hand: the temporary grounding transition applies only to sites using NoArchive or NoCache as of December 15, 2026; for each page the temporary preference applies until recrawl or September 30, 2027, whichever is earlier; after March 1, 2027, NoArchive and NoCache will not be honored for training)Read source
  6. [6] IETF Datatracker, draft-kelley-aipref-training-preference-signal-00, Kevin Kelley (Anthropic), October 30, 2026, expires May 3, 2027 (primary — read first-hand: defines the __usage_ai-training policy token; the token is not a directive for crawling behavior; wildcard groups must not be read as matching it; private agreements may override preferences expressed through it)Read source
  7. [7] IETF Datatracker, draft-ietf-aipref-vocab-08, AI Preferences working group vocabulary draft (primary — read first-hand: its contents do not reflect working-group consensus; the specification does not ensure that preferences are followed; AI Training is defined as using an asset to modify the learned parameters of a generative AI model)Read source
  8. [8] Kevin Kelley, AIPREF (ai-control) mailing list, October 30, 2026 (primary — read first-hand: "Anthropic has already started implementing support for this draft. This complements our existing crawler controls rather than replacing them"; complications and edge cases expected)Read source
  9. [9] Search Engine Land, "Microsoft Bing gives publishers more control over how AI uses their content", October 30, 2026Read source
  10. [10] Search Engine Land, "Google also gives publishers more control over how AI uses their content", October 30, 2026Read source
  11. [11] MLex, "Google, Microsoft, OpenAI, Anthropic to offer streamlined AI training tool", October 30, 2026Read source
  12. [12] Unite.AI, "OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences", October 30, 2026Read source