We still don’t know how people are really using AI
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- AI Observatory, a new public research platform led by Stanford's Anka Reuel and MIT's Shayne Longpre, aggregated 85,633 conversational turns from 24,521 conversations across 7 datasets and 52 models between 2023-2025
- Anthropic's Economic Index filters out non-work conversations — when Observatory researchers applied its methods to their dataset, nearly 48% of conversations would have been excluded
- The filtered-out conversations disproportionately involved health/relationships (44.2% vs 31.2% in Anthropic's analysis), adult/illicit topics (7.9% vs 2.1%), harassment/hate (27.5% vs 5.66%), and sexual content (16.7% vs 2.4%)
- OpenAI's 2025 ChatGPT report similarly found only 30% of consumer use was work-related, yet still reflects a narrower sample than the Observatory's broader methodology captures
- Grok was especially popular for news and politics but also where misinformation concentrated; Gemini saw more social and roleplay use; ChatGPT dominated homework assistance, with users shifting to longer, more iterative exchanges on GPT-4o
- Conversations grew longer and more elaborate over time while sensitive content became less frequent, suggesting improved safeguards alongside rising AI companionship and reduced self-disclosure by chatbots
- The Observatory's dataset is dwarfed by what companies hold internally — Anthropic's latest index analyzes 1 million Claude conversations and OpenAI's report drew on 1.5 million — and the Observatory itself likely underrepresents sensitive uses since conversations were voluntarily shared
Why it matters: With policymakers weighing consequential decisions about AI's risks and benefits largely on companies' selective self-reports, the AI Observatory's finding that Anthropic's methodology alone filters out 48% of conversations reveals a systematic blind spot. Widely-cited metrics like the Anthropic Economic Index describe a sanitized slice of real-world AI behavior, potentially leaving regulators and researchers blind to harassment, sexual content, and emotional dependency patterns the companies choose not to foreground.
Ask SkimNews
