Google Rejects llms.txt as AI Crawlers Ignore It
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- llms.txt adoption sits at 10.13% across 300,000 domains in an SE Ranking study, with only one of the 50 most AI-cited domains publishing the file at all.
- Limy.ai's 90-day analysis of over 500 million AI bot events found only a few hundred requests targeted /llms.txt directly, with GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended overwhelmingly crawling HTML instead.
- Google confirmed it doesn't support llms.txt and isn't planning to, with Gary Illyes stating the position, John Mueller comparing it to the discredited keywords meta tag, and Google's 2026 generative-AI documentation listing it among unnecessary tactics.
- No major AI company has publicly committed to reading or acting on llms.txt in production as of 2026, and the proposal carries no backing from the W3C or IETF.
- HTTP Archive's 2025 Web Almanac found roughly 40% of published llms.txt files were plugin-generated defaults rather than deliberately curated documents.
- The author still recommends shipping one anyway — twenty minutes of work buys asymmetric optionality if agentic routing standardizes, and curating 10-40 priority pages surfaces duplication and orphaned content teams rarely audit otherwise.
Why it matters: llms.txt costs twenty minutes to ship but delivers zero confirmed citation benefit today, and Google's explicit rejection means teams betting SEO strategy on it risk building on nothing. The hidden upside, per the source, is editorial: curating 10-40 priority pages forces teams to identify duplication and orphaned content that usually goes unaudited.
Ask SkimNews
