Google, OpenAI, Anthropic, and Microsoft have agreed to adopt a standardized robots.txt directive that allows web publishers to block content harvesting for generative AI training with a single command. The commitment centers on an open standards proposal currently moving through the IETF, authored by Anthropic technical lead Kevin Kelley.
Previously, website operators faced the cumbersome task of identifying individual web crawlers and writing distinct rules for every single AI provider. The new standard introduces a designated user agent specifically for AI training, labeled as __usage_ai-training. Property owners can insert this tag into their robots.txt files to restrict scrapers across participating platforms while retaining granular control over specific directories, such as a designated press kit section.
Importantly, implementing the new signal does not affect traditional search indexing or cause sites to disappear from standard web results. Google clarified that it will interpret the directive as a direct block for its Google-Extended crawler used in training Gemini models. Tech firms also noted that the standard applies exclusively to future data collection and cannot retroactively remove information already gathered.
Anthropic has already rolled out support for the framework. Microsoft plans to adopt the signal on February 1, 2027, with Google and OpenAI following suit in early 2027 and the coming months, respectively.
Comments