Opinion: AI has created a shadow medical system
Get the Health newsletter
Daily health & science — research, biotech, public health, the studies worth knowing. Free.
- More than 40 million Americans ask ChatGPT a health question every day, with most conversations happening outside clinic hours and not leading to a physician visit.
- A 2025 study found that medical disclaimers have largely disappeared from leading AI models' responses to health questions; today's models will not only respond but ask follow-ups and attempt a diagnosis.
- The authors' JAMA Network Open study tested 21 frontier AI models and found they named the correct diagnosis more than 90% of the time given a complete case but failed to produce a comprehensive differential more than 80% of the time given only initial-triage information.
- Consumer health AI products are expanding rapidly: Oura sells a 50-biomarker blood panel via Quest Diagnostics for $99; Function Health (valued at $2.5 billion in November) lets members order 160 lab tests per year and authorize ChatGPT to read results; Doctronic has run 24 million consultations and writes AI-generated prescription refills in Utah.
- Health care represents close to one-fifth of the U.S. economy — which the authors identify as the financial driver pushing every major AI company into the space and fueling headlines that doctors are becoming optional.
- Anthropic, OpenAI, and Google all operate mechanistic-interpretability research programs, yet the field remains nascent; the authors argue this should give pause to anyone building medical products on these models because engineers cannot fully explain how they arrive at answers.
- The authors argue that "non-inferior to physicians" is the wrong threshold for adopting clinical AI, since physicians carry ongoing accountability for patient harm while AI systems assume none — quoting IBM's 1979 training manual: "A computer can never be held accountable, therefore a computer must never make a management decision."
Why it matters: When AI health tools give wrong advice, responsibility still falls on clinicians — and the authors' JAMA Network Open study found AI models fail at initial-triage differential diagnosis more than 80% of the time. As health care represents roughly one-fifth of the U.S. economy, consumer AI health products are scaling fast with minimal regulatory oversight, putting patients at risk precisely when they are most anxious.
Ask SkimNews
