OpenAI, Google, Microsoft Back Shared Scraping Opt-Out — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Anthropic has already adopted a shared “policy token” proposed by technical lead for standards Kevin Kelley through the IETF; the signal will live in robots.txt and let publishers set AI-training preferences across multiple crawlers instead of company by company.
- OpenAI says it will implement the shared token in the “coming months” and restrict its AI crawlers from accessing a site when the signal appears.
- Google plans to support the shared token next year and already offers separate controls for whether content appears in AI search features or is used for AI training, without reducing search visibility.
- Microsoft plans to support the shared token next year, says its granular controls can distinguish AI training from AI grounding, and notes that opted-out websites will still appear in Bing results.
- Meta communications head Andy Stone supports creating a common signal and working with other AI companies, but did not say whether Meta will honor the token.
- Really Simple Licensing gives publishers the ability to charge crawlers that visit their sites, while Cloudflare blocks AI crawlers by default.
Why it matters: For publishers, the practical gain is replacing company-by-company opt-outs with one robots.txt preference. Anthropic already supports it, OpenAI plans implementation in the “coming months,” and Google and Microsoft say next year; their existing controls also leave ordinary search visibility intact.
Ask SkimNews
