Running a website now involves a second publishing decision: how should AI providers use its content? Putting that choice into practice can mean maintaining separate rules for different companies. On October 30, OpenAI announced that it will honor a shared robots.txt signal for preferences concerning foundational generative AI training.
The voluntary initiative aims to let one preference reach several participating providers. OpenAI plans to implement the signal in the coming months, with details to follow. Its existing controls will remain in force.
Content use and crawler access are different choices
The technical draft linked by OpenAI uses the familiar robots.txt structure to associate training preferences with website paths. It distinguishes those preferences from rules addressed to individual crawlers.
Consider a publication that wants readers to find its reporting through search while expressing a preference against training on that reporting. Those are separate decisions. OpenAI's current crawler documentation already distinguishes OAI-SearchBot for search from GPTBot for training, with independent settings.
That distinction makes a configuration review more useful than a blanket response. The person editing the file needs to establish whether a rule concerns discovery, collection by a particular bot, or the use of content to change a model's parameters. Otherwise, a broad restriction could affect something the publisher intended to allow.
Implementation will determine the practical value
The proposed mechanism is an individual Internet-Draft, without IETF endorsement. It draws on AI Preferences working group discussions, but it is not a finalized standard. Publishers maintaining infrastructure or publishing plugins should account for the possibility that the proposal evolves.
A shared convention could reduce repetitive configuration. Its operational value will depend on participating providers documenting their interpretations, implementation dates, and treatment of conflicts with more specific rules. A common label is helpful when the organizations reading it apply a sufficiently consistent meaning.
There is another boundary to keep in mind: declaring a preference does not authenticate a visitor or technically stop a client that ignores it. The draft itself treats content-use preferences and crawling behavior independently. That makes the mechanism a way to communicate policy, rather than an access-control system.
For a website operator, the immediate task is to review and document the choices already in place. Configuration changes should follow the actual implementation and its published instructions. Removing provider-specific rules before that point would abandon controls OpenAI says it continues to respect before the new signal's implementation is understood.