Study Finds 73% of Users Accept Faulty LLM Answers
Get the Health newsletter
Daily health & science — research, biotech, public health, the studies worth knowing. Free.
- University of Pennsylvania researchers used the Cognitive Reflection Test (CRT) to examine AI‑induced “cognitive surrender.”
- LLM chatbot was deliberately programmed to give wrong answers on about half of the CRT questions and correct answers on the other half.
- Participants who consulted the chatbot accepted its correct reasoning 93% of the time and its faulty reasoning 80% of the time.
- AI‑using participants reported a confidence boost of 11.7% even when the chatbot was inaccurate half the time.
- Incentives (small payments) and immediate feedback raised the rate of correctly overruling faulty AI by 19 points, while a 30‑second time pressure cut it by 12 points.
- Researchers note that cognitive surrender isn’t inherently irrational and could be advantageous when AI systems are statistically superior, such as in probabilistic or data‑heavy tasks.
Why it matters: People who trust AI without verification risk being misled, while those with higher fluid IQ or external incentives are better at correcting errors; the study also notes that surrender can be rational when AI quality is high, highlighting a trade‑off between speed and accuracy in AI‑augmented decisions.
