ICML study: AI peer-review ban changed nothing — SkimNews
Get the Health newsletter
Daily health & science — research, biotech, public health, the studies worth knowing. Free.
- Microsoft Research ran a large randomized experiment at the 2026 International Conference on Machine Learning (ICML) in Seoul in July, covering 24,661 papers overseen by 17,886 reviewers split between a strict LLM ban and a permissive policy allowing AI to help understand papers, check related work, and polish writing
- Papers reviewed under the two policies had nearly identical acceptance rates — 27 per cent under the strict ban versus 26.5 per cent under the permissive policy — with average review scores of 3.31 and 3.32 out of 6 and virtually unchanged reviewer confidence
- An anonymous survey of 1,486 reviewers found 22.5 per cent of those told not to use AI admitted using an LLM anyway, to brainstorm feedback, draft review text, read papers, and summarise strengths and weaknesses
- The Pangram AI text detector classified only 52.2 per cent of reviews under the conservative policy as fully human-written, compared with 37.0 per cent under the permissive policy, though the researchers cautioned AI text detectors are imperfect
- Miro Dudík of Microsoft Research, who co-led the study and helped organize ICML, said the 23 per cent self-reported non-compliance was higher than he anticipated and pointed to large reviewing workloads and insufficiently clear rules
- Reviews written under the permissive policy were 5.5 to 7 per cent longer and rated slightly higher quality by experts, though a reviewer-by-reviewer comparison found no meaningful difference
- Kayvan Kousha at the University of Wolverhampton said the study shows banning AI in peer review is 'very difficult to enforce' because of academic workload pressures
Why it matters: For academic venues considering LLM bans, the data suggests policy on paper changes almost nothing about outcomes: acceptance rates, scores, and confidence held steady whether AI was forbidden or permitted. The 22.5 per cent non-compliance rate — driven by reviewer overload — means conferences that try to police AI use are largely governing behaviour that hasn't actually changed.
Ask SkimNews
