Google and OpenAI announced on October 30 that they will support a shared signal allowing website operators to express their preferences about content being used to train AI models. The proposal uses the robots.txt file to communicate the same choice to multiple participating providers. Google points to early 2027 for implementation, while OpenAI refers to the coming months.
The change concerns control over the use of material published online. For publishers and businesses managing websites, the proposed mechanism adds a common option alongside individual AI companies' controls. Existing rules remain in operation, and the two announcements describe future implementations with separate schedules for each provider.
A shared identifier in the robots.txt file
The signal is described in a technical draft published at the IETF on October 30 under the name Machine-Readable Training Preference Signal. The document proposes the identifier __usage_ai-training, placed in a robots.txt rule group. Website operators can thereby communicate a training preference using the file's familiar structure.
The proposal retains the Allow and Disallow directives and the path-matching rules of the existing protocol. It introduces no separate language for writing the file. Rather than repeating the same choice for every provider adopting the mechanism, an operator can express it through the shared identifier, alongside rules dedicated to individual companies.
Preferences can be defined for website paths. The draft's example describes a preference against training across the site, with a positive exception for a directory of press materials. This allows different choices to be expressed for distinct sections while retaining the existing robots.txt mechanism for matching paths.
The document nevertheless separates the content-use signal from crawler access rules. A policy identifier can appear in robots.txt without appearing in a robot's HTTP request header. Its role is to communicate the purpose for which content can be used, while rules addressed to a crawler continue to describe that crawler's access.
Google's existing controls remain in place
Google explained how it intends to interpret the new signal. If a website expresses a preference against training through it and does not explicitly specify Google-Extended, the company plans to treat that choice as a disallow rule for Google-Extended. Explicit rules for that control will still be able to express an individual choice concerning Google.
Google-Extended has been available since 2023. According to the current documentation, it controls content use for training future generations of Gemini models and for grounding in Gemini Apps and the Grounding with Google Search service on Vertex AI. Grounding means supplying content to a model at request time to support its response with relevant information.
The company separately specifies that Google-Extended does not affect a website's inclusion in Google Search and is not a Search ranking signal. For search, access rules addressed to Googlebot retain their role. This distinction separates the management of search-engine visibility from preferences concerning the uses covered by Google-Extended.
In its October 30 announcement, Google notes that websites explicitly allowing Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. That statement concerns the Gemini Apps product. It accompanies the explanation that the future shared signal will operate alongside provider-specific controls.
OpenAI retains separate search and training controls
OpenAI described the initiative as a voluntary effort to simplify website operators' choices. The company aims to implement the signal in the coming months and publish details of its implementation. Its current robots.txt controls will continue to apply alongside the shared option.
OpenAI's documentation treats OAI-SearchBot and GPTBot independently. The first is used to discover and surface websites in ChatGPT's search features. GPTBot collects content that may be used to train generative foundation models. An operator can allow OAI-SearchBot while disallowing GPTBot, expressing the two choices separately.
OpenAI also specifies that, when both are allowed, it may use the results of a single crawl for both purposes to avoid duplicate collection. ChatGPT-User has a different role: it is associated with user-initiated actions and does not determine content's eligibility for search. The new announcement preserves these existing controls.
A technical proposal in progress with a defined scope
In the IETF registry, the proposal is an individual Internet-Draft rather than an adopted standard. It draws on discussions in the AI Preferences working group, known as AIPREF. The text defines training as using content to modify the learned parameters of a generative model, distinguishing it from tasks used exclusively for classification, ranking or scoring.
The draft also specifies that a general User-agent group using an asterisk must not automatically be interpreted as a preference for the training identifier. When no group explicitly names that identifier, the document treats the mechanism as expressing no preference. Existing individual controls continue to be interpreted under their own rules.
For software that does not implement the mechanism, the draft seeks to preserve existing behavior. Such software ignores the policy group in the same way it handles an unknown crawler identifier. Likewise, files that do not use the new signal continue to be interpreted without changing their access rules.
According to the draft, the preference applying to a resource is the one expressed in the robots.txt file in effect when that resource was retrieved. The document recommends giving specific choices priority over general ones and allows providers to document how their own controls take precedence over the shared signal.
Google says the proposal reflects requests from publishers, civil society, regulators and other participants for shared controls. OpenAI supports the same direction and encourages other providers to honor the signal. The announced implementation concerns participating companies and preserves individual choices while technical standardization discussions continue within the IETF.
