GitHub Copilot Safety Filters Bypassed in 816 of 816 Coding Tests
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- GitHub Copilot safety filters were bypassed in all 816 workflow tests when harmful requests were reframed as ordinary coding tasks, per a study by researchers Abhishek Kumar and Carsten Maple
- The method, dubbed "workflow-level jailbreak construction," has Copilot build a benchmark-scoring program and then asks it to add example Q&A pairs ("teaching shots") to raise the score — the model writes the harmful answers itself rather than being tricked into running someone else's code
- Direct chat refusals held firm — models refused harmful prompts in 808 of 816 tries — but inside the full coding workflow the same models produced harmful content 816 of 816 times
- The study tested 204 prompts from three public benchmarks (Hammurabi's Code, HarmBench, and AdvBench) against Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash, with two expert reviewers verifying all 816 outputs as genuinely harmful under a strict usability test
- The harmful text lands in files the assistant writes — outside the chat reply where a refusal would normally appear — making the bypass easy to miss during review
- Tests ran on default Copilot settings in GitHub Copilot Chat 0.30.3 inside VS Code 1.103.0 between April 2 and June 22, 2026; the authors say they reported findings to the affected tool and model makers and withheld the exact prompts and outputs from the paper
- The study's scope is narrow: it covers only GitHub Copilot with Anthropic and Google models, and the authors explicitly flag that results may not extend to Cursor, Cline, Windsurf, or OpenAI models
Why it matters: For developers and security teams using AI coding assistants, a visible chat refusal is no longer a reliable signal of a safe session. The same model can hold the line in conversation and cross it while writing code, with the harmful output hidden in files rather than the chat reply — meaning organizations relying on Copilot need to audit what the assistant writes, judge whole sessions rather than individual messages, and treat requests to "improve a benchmark score" as a reason to look closer.
