OpenAI, Google, Microsoft, Anthropic Back AI Opt-Out — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Kevin Kelley proposed the shared “policy token” as part of the IETF process; it would live in robots.txt and let publishers apply AI-training preferences across multiple crawlers instead of setting them separately for each AI company.
- OpenAI said its AI crawlers will be restricted from accessing a site when they encounter the shared token, with implementation coming in the “coming months”; Google and Microsoft plan support next year, while Anthropic has already adopted it.
- Meta communications head Andy Stone called a common signal an important step and said Meta would work with other AI companies on implementation, but he did not say whether Meta will honor the token.
- Microsoft will offer granular controls letting publishers opt out of AI training or grounding, the process of pointing a model to a verified source to reduce hallucinations; opted-out websites still appear in Bing search results.
- Google already lets sites control whether their content appears in AI search features or is used for AI training without affecting visibility in search, and plans to support the shared token next year.
- Really Simple Licensing gives publishers the ability to charge crawlers that visit their sites, while Cloudflare blocks AI crawlers by default.
Why it matters: For publishers, the token offers one robots.txt setting across four major AI providers instead of separate preferences; Anthropic has already adopted it, and Google and Microsoft plan support next year. Search visibility remains unaffected in Bing and Google, while Meta has not said whether it will honor the opt-out.
Ask SkimNews
