The New AI Training Opt-Out Will Not Stop a Single Crawler Request

The whole mechanism is a crawler name that belongs to no crawler. You write a group for a user agent that does not exist:

User-Agent: __usage_ai-training Disallow: /

That is it. No new keywords, no new file, no new header. The draft calls it a policy token, and the double underscore is there so it cannot be mistaken for a real bot name.

It lives in draft-kelley-aipref-training-preference-signal-00, filed on October 30 by Kevin Kelley at Anthropic. It borrows its definition of AI training from the working group's vocabulary draft: using a page to change the learned parameters of a generative model.

Two things about that paperwork matter before you treat this as a settled standard.

First, the datatracker page calls it an Active Internet-Draft (individual) and says in so many words that it is not endorsed by the IETF and has no formal standing in the IETF standards process. It is one author's document that four large companies decided to honour.

Second, the vocabulary draft it borrows from still carries a note above the section that defines the words: This section does not yet have consensus. I pulled both drafts today. The vocabulary is still at version 08 from September 13 and the companion attachment draft is still at 05 from August 18. Neither moved. What moved was a separate individual filing and four announcements.

Section 4.4 of the draft is four short paragraphs, and the first one is the whole answer:

This policy token is not a directive for crawling behavior.

A company may use the preference to decide what to fetch. It is not required to. Microsoft says the same thing in its own words: These settings govern use of content, not crawl access.

So this is a label on a page, not a gate in front of it. You are telling a company what it may do with text it already has. You are not telling its crawler to go home.

That difference is not academic here. In the 28 days from October 3 to October 30 the portfolio took 32,307,474 declared AI crawler requests, and 32,125,636 of them came from four crawlers: GPTBot, Amazonbot, meta-externalagent and ClaudeBot. That is 99.44 percent, and almost all of it is one behaviour. Those crawlers walk the dropdown filters on our city guide directories, one address per combination. On a single host on October 28, GPTBot made 99,540 requests at 99,539 different addresses in a day. Not a repeat in it.

Write the new line on that host and the number stays 99,540.

If you want fewer requests you still need the old tools: a rule naming the crawler itself, a rate limit, or a change to the addresses you publish. I wrote up why a Disallow on filter addresses only half works a few weeks ago, and nothing in this announcement changes it.

Microsoft's post is the most detailed of the four, and the only one that separates the two questions a site owner actually has.

Training is one preference. Grounding, which is Microsoft's word for using your page to help write an AI answer, is a second and completely separate one. You can refuse training and still allow grounding, and Microsoft spells that case out directly.

Grounding is page level only. There is no site-wide version in robots.txt. You add a meta tag or a response header:

<meta name="Bingbot" content="ai-grounding:disallow">

A middle setting, ai-grounding:disallowextended, keeps your content eligible inside Microsoft's own products such as Copilot and AI Mode while shutting out the grounding services Microsoft sells to other companies.

Both preferences default to Allow. Do nothing and nothing changes.

Microsoft is blunt about the cost, which I respect: If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences. It also says twice that none of this changes whether you appear in ordinary Bing results.

One detail will catch people. These preferences replace Microsoft's old NoArchive and NoCache tags, and the old tags stop being read as a training or grounding preference after March 1, 2027. If you set NoArchive years ago and forgot about it, your answer quietly flips to Allow.

Google's post is six paragraphs by John Mueller, and one sentence in it decides what the line costs you:

When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.

Reasonable, on its face. Google already had a training control, so the new line routes into it.

Except Google-Extended was never only a training control. Here is Google's own crawler documentation, unchanged and still live today. Google-Extended manages whether your content may be used for training Gemini models and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.

Training and grounding, one token, no way to separate them. Mueller's post confirms what that buys in the other direction: Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Clicks. His word.

So write the same intent into your robots.txt and here is what each company hears.

Microsoft hears: no training, keep writing answers from my pages.

Google hears: no training, and stop quoting me in Gemini.

Nobody is being sneaky. Both published exactly what they will do, on the same day, in plain English. The two designs simply disagree about whether writing an answer from a page is the same act as training on it. The vocabulary draft treats them as separate categories and Microsoft built to that. Google mapped the new line onto a control it shipped in 2023, and that control was always wider.

This is the part I did not expect, and it changes what a wide-open robots.txt means.

Section 4.2 of the draft: wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token.

And then: If no group contains the __usage_ai-training policy token ... then no preference has been expressed.

RealSEOLife.com's robots.txt opens with User-agent: * and Allow: /. Everything here is deliberately open, including every JSON file. Under the new mechanism that says nothing at all about training. Not yes, not no. Silence.

That reads well for anyone who, like me, has no intention of blocking anything. Silence means no restriction, so doing nothing costs nothing.

But it also means a wide-open file cannot say yes on purpose. To be counted as a yes you have to name the token and allow it, which the draft and Microsoft both show as an option:

User-Agent: __usage_ai-training Allow: /

One property in my portfolio has been in that position for months by accident. DataSetSEO.com has carried an explicit User-agent: Google-Extended with Allow: / since July 20. Six sibling research properties have not. Given what Mueller wrote about explicit allows and Gemini clicks, that is now a test worth running rather than a tidy habit.

Every control in this story can be checked except one.

If I disallow GPTBot I can watch GPTBot stop. The requests carry a name, OpenAI publishes its address ranges, and the row either appears in my logs or it does not. Same for ClaudeBot and Amazonbot. I have checked a crawler's identity against published ranges before and it works.

Google-Extended cannot be checked that way, because of this line in Google's documentation: Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.

There is no fetch to count. The crawling is Googlebot's, which you want. The token only changes what happens to the page later, inside Google, where your server never sees it.

So the one company whose version of this line also costs you citations is the one company whose compliance you cannot measure. That is not an accusation. It is a design fact, and it is the reason I will not write that line on a property I am trying to make visible.

A robots.txt file used to answer one question. Who may fetch what.

After October 30 it answers two. So when I open one now I ask three things in order.

1. Which groups are traffic rules, naming a real crawler that will show up in my logs? 2. Which groups are usage labels, naming a token no request will ever carry? 3. Does any page-level tag quietly override the file?

Mixing those up in either direction is expensive. Treat a usage label as a traffic rule and you wait for a crawl drop that never arrives. Treat a traffic rule as a usage label and you block a crawler you wanted.

The bigger point sits underneath all of it. For three years the argument about AI and publishing has been about training, because training is where the copyright fight is. The thing that actually sends a reader to your page is grounding, and that is the category these announcements handle least consistently.