LLMs have unsolvable security flaw, researchers warn
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Researchers Charles Ye and Jasmine Cui presented a paper at the International Conference on Machine Learning arguing that LLMs cannot be made fully secure because of a fundamental flaw in how the models track where instructions come from.
- The attack, which the researchers call chain-of-thought forgery, works by writing prompts that mimic the model's own scratch-pad style, tricking the LLM into treating malicious instructions as if they were its own internal thoughts.
- The technique successfully extracted restricted information from OpenAI's gpt-oss-20b and GPT-5, including cocaine synthesis instructions and guidance for sabotaging a commercial aircraft's navigation system; the researchers said similar results occurred against models from Anthropic, Alibaba, and DeepSeek.
- The discovery won OpenAI's red-teaming hackathon in August 2025, and OpenAI's own GPT-Red system reportedly identified a comparable "fake chain of thought" attack independently around the same time.
- Cui said that even GPT-5.4, released in March, gave her suicide instructions despite added training, while ETH Zürich computer scientist Florian Tramèr praised the attack insight but noted that combined defenses work "pretty well" for general use, though "it's not clear this will be sufficient for highly sensitive cases."
- The researchers concluded that because role identification is foundational to how LLMs work — they judge text by style rather than the tags surrounding it — no amount of training will fully solve the problem, and Ye warned organizations to "expect that anything done by agents could be unsafe."
Why it matters: For organizations deploying LLMs in critical systems — government, military, healthcare — the finding implies that current red-teaming and training approaches can never make models fully secure, and Ye argues organizations should assume any agent-driven action could be unsafe. The economic incentive for jailbreaks and prompt injections will only grow, meaning companies betting on incremental safety training to protect high-stakes applications may be operating on an unprovable assumption about what defenses can achieve.
