Most quantum computing papers fail replication test
Get the Health newsletter
Daily health & science — research, biotech, public health, the studies worth knowing. Free.
- Wolfgang Mauerer and colleagues at TH Regensburg analyzed thousands of quantum computing papers and found only about a quarter provided enough information or code to attempt replication — a result Mauerer called "rather negative" compared to traditional computer science standards.
- The study used a two-part methodology: manual evaluation of 127 papers against five reproducibility criteria (code availability, instructions, hardware documentation, and runnability), then an automated analysis scaled to 4,966 papers.
- Only 24.4% of the manually-analyzed papers provided code the team could attempt to run, and 64.5% of those codes failed to execute successfully on independent quantum computers.
- The automated analysis of 4,966 papers found just 26.8% provided enough information to attempt replication — a rate that had not improved from a smaller study Mauerer's team conducted four or five years earlier.
- Mauerer attributed the poor reproducibility partly to the "unusually variable" nature of still-maturing quantum computing hardware, which unlike conventional machines can shift day-to-day — not just author negligence or missing community standards.
- William Zeng of the Unitary Foundation said he wasn't surprised by the findings but expects agentic AI coding tools will soon make it easier to generate reproduction code directly from published papers.
- Fred Chong at the University of Chicago pushed back, arguing quantum computing and its software are "in their infancy" and that innovation may matter more than reproducible infrastructure at this stage of the field.
- Ralf Ramsauer of TH Regensburg said the response to the study has been positive and that the team's five-criteria framework offers a template researchers can use to build reproducibility packages from the start of each experiment.
Why it matters: The study gives quantum computing researchers a concrete five-criteria template for reproducibility packages, and the field's already-positive response suggests standards may tighten sooner than the decades-long timeline seen in conventional computer science. For a discipline racing toward demonstrated commercial utility, persistent unreplicable results risk undermining the credibility of claimed breakthroughs and the industry's broader scientific standing.
Ask SkimNews
