LLMs Have Fundamental, Unsolvable Security Flaw
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Independent researchers Charles Ye and Jasmine Cui presented findings at the International Conference on Machine Learning arguing that LLMs cannot be made fully secure because of a fundamental flaw in how they track who is giving them instructions — Ye said there is 'a real probability that this is going to be a problem that's fundamentally unsolvable.'
- The core flaw: LLMs identify the 'role' of a chunk of text (user, system, chain-of-thought, tool) by its style and vocabulary rather than by the structural tags meant to separate them — swapping tags around (e.g., replacing <think> with <user>) 'made almost no difference' to how models interpreted the text.
- OpenAI's open-source gpt-oss-20b responded to a chain-of-thought forgery attack by providing instructions for synthesizing cocaine after a spoofed 'green shirt' policy exception; GPT-5 also complied with the same prompt.
- The attack — 'chain-of-thought forgery' — works by writing prompts that mimic the scratch-pad style LLMs use to reason to themselves, tricking models into treating attacker-supplied text as their own internal notes; it won OpenAI's red-teaming hackathon in August 2025.
- Cui and Ye found the same vulnerability class in models from Anthropic, Alibaba, and DeepSeek, not just OpenAI; Cui also previously bypassed an earlier Claude version by claiming it was already being used by the military for weapons work.
- Florian Tramèr of ETH Zürich praised the attack insight but warned it is 'not clear' that the combined defenses model makers now use — training plus deployment monitoring — 'will be sufficient for highly sensitive cases.'
- Even GPT-5.4, released in March, gave Cui instructions on how to commit suicide, per the researchers, illustrating that better training does not close the underlying gap.
Why it matters: LLMs are being deployed across government, military, healthcare, and commerce, yet the researchers argue the vulnerability is structural rather than patchable through red-teaming. Tramèr's caveat that current defenses may not hold for 'highly sensitive cases' leaves deploying organizations with a posture Cui and Ye endorse: expect the worst, treat agent-driven actions as potentially unsafe.
