AI Hallucination Nearly Triggers US Strike on Chinese Vessel — SkimNews
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Military aircraft were already in the air when U.S. officials discovered the intelligence driving an armed operation against a Chinese vessel had been hallucinated by an AI chatbot, per CNN.
- The operation was aborted at the last minute, narrowly averting a potential conflict with China, according to CNN's Friday report.
- A Special Operations Command analyst queried an AI chatbot to synthesize open source data with classified signals intelligence; the chatbot misidentified the ship's cargo manifest, which falsely claimed the vessel carried components for a nuclear weapons program.
- The analyst used the chatbot a second time to format the erroneous findings into an official-looking summary that circulated across command channels.
- The false intelligence report circulated during the war with Iran, according to the source.
- Jake Steckler, research scholar at GovAI and veteran U.S. Army officer, warned that LLM uncertainty is 'especially critical for any decisions that could lead to use of force.'
- The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain against competitors like China, even as this incident shows hallucinations can travel up the chain of command before being questioned.
Why it matters: The Pentagon is racing to fold AI into its kill chain to maintain an edge over China, and this near-miss shows a single chatbot misidentification can escalate from an analyst's query to live military aircraft in the air. The episode hands the military a concrete case for the safeguards Steckler describes — including requiring human verification before targeting decisions — and risks eroding service-member trust in AI tools the Pentagon wants to deploy at scale.
Ask SkimNews
