Claude chats showed in Google and Bing search results, despite Anthropic's robots.txt saying not to crawl them, likely because the pages lacked a "noindex" tag (Maddy Varner/Wired)
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Anthropic shared Claude chats appeared in Google and Bing search results despite the company's robots.txt directing crawlers not to index those pages
- The likely technical cause was that the shared-chat pages were missing a "noindex" meta tag, allowing crawlers to bypass the crawling restrictions set in robots.txt
- The exposure affected both shared Claude conversations and Artifacts, the shareable outputs users generate through Claude, as confirmed across coverage
- The incident underscores how difficult it is to prevent web crawlers from making ostensibly private AI chatbot conversations indexable once a share-link creates a public URL
Why it matters: Any Anthropic user who clicked "share" on a Claude conversation expecting limited visibility had that chat become searchable on Google and Bing — exposing prompts and responses they may have considered private. The technical gap is instructive: robots.txt merely requests that compliant crawlers stay away, while a noindex tag actively prevents indexing, meaning the two controls are not interchangeable.
