AI providers back one training preference in robots.txt
Website operators will be able to convey the same AI-training preference to several providers through their existing robots.txt file. Anthropic has adopted the proposed signal, while OpenAI, Google and Microsoft have different implementation schedules. The voluntary approach preserves existing crawler controls and remains an Internet-Draft, leaving the timing of adoption and each company’s compliance central to its practical use.
Artificial Intelligence··Morning
One file for a shared preference
Website owners are being offered a common way to communicate whether their content should be used to train generative AI. OpenAI, Google, Microsoft and Anthropic support the proposed training-preference signal in robots.txt, the file websites use to give instructions to automated crawlers. It aims to spare operators from maintaining a separate training rule for every participating provider. Google has also described its support for the shared approach. The proposal concerns the use of material collected from a site and keeps that choice separate from the question of whether a crawler may visit it. A training preference serves a different purpose from a general instruction governing access.[1], [2]
Providers have different schedules
Adoption will take place on different schedules. Anthropic has already taken up the approach, while OpenAI plans to implement it over the coming months. Google and Microsoft have put their implementation on next year’s schedule. Meta expressed support for the work but did not make the same commitment to follow the signal, according to The Verge. OpenAI says the robots.txt controls it already provides will remain in force, with more implementation information to come when it adopts the new mechanism. Microsoft has separately described controls distinguishing training from using source content to ground responses. The shared preference is being added alongside existing provider-specific choices.[1]
An Internet-Draft retains the existing file structure
Anthropic’s Kevin Kelley authored the individual Internet-Draft, Machine-Readable Training Preference Signal, dated October 30. Its design preserves robots.txt groups and path-matching rules. A policy token expresses the training preference without introducing new directive names or changing the file’s grammar. Existing parsers that do not recognize the mechanism are intended to keep processing ordinary traffic-management instructions. Discussions about expressing content-use choices continue in the Internet Engineering Task Force’s AI Preferences working group. The draft has not become an approved standard. Participation is voluntary, so a machine-readable request does not itself guarantee compliance by every recipient. Implementation depends on each provider honoring the preference it receives.[1]
