OpenAI said on Friday, October 30, that it will honor a common signal in robots.txt that tells AI companies whether a site’s pages may be used for “foundational generative AI training.” One line would reach every participating provider. Until now, a publisher had to name each company’s crawler separately.
The signal is not OpenAI’s invention. It comes out of the Internet Engineering Task Force’s AI Preferences working group, known as AIPREF. The text is an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, written by Anthropic engineer Kevin Kelley.
The mechanism is plain. Instead of a crawler name, the User-Agent line carries a “policy token,” __usage_ai-training. Disallow a path and you have stated a preference against training on it. Allow a path and you have stated a preference in favor. The draft adds no new directives, and a wildcard group (User-Agent: *) must not be read as covering the token.
OpenAI had company. Google announced its plans the same day on the Search Central blog. If a site opts out through the token and says nothing about Google-Extended, Google will treat it as a Google-Extended disallow. Microsoft said Bing will honor the token from February 1, 2027, and Kelley told the working group that Anthropic “has already started implementing support,” PPC Land reported.
None of the four has said the signal is live. OpenAI’s timetable is “the coming months,” with another update promised when it ships. Its existing crawler controls stay in force, and the company called the effort voluntary.
Key takeaways
- OpenAI said on Oct 30 it will honor a shared robots.txt signal for AI training.
- The token is __usage_ai-training, from an IETF AIPREF Internet-Draft.
- Microsoft starts Feb 1, 2027; Google targets early 2027; OpenAI gave no date.
How do I block AI training in robots.txt?
Add a robots.txt group with the line User-Agent: __usage_ai-training followed by Disallow: /. OpenAI, Google and Microsoft have said they will honor it, but none had switched it on as of October 31, 2026.
The rule only covers training. It does not stop crawling or affect search rankings, and company-specific rules keep working alongside it. For creators deciding which tools may learn from their work, this sits next to the questions in How AI is Reshaping the Content Creator Stack in 2026.
The timing carries its own message. OpenAI published the announcement on the same day as “Fair use benefits everyone,” a post rounding up authors, librarians, academics and others who back its position on training data.
What to watch next
Two things. First, OpenAI’s implementation note, which should say how its systems will read the token. Second, the draft itself: an individual Internet-Draft is a proposal, and the AIPREF group still has to adopt it before it moves toward a standard.