A Contract-Grade Verifier for LLM-Generated GPU Kernels
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Researchers built a contract-grade verifier of twelve adversarial gates — several tolerance-free — that checks properties a correct GPU kernel must satisfy (NaN/infinity handling, shape robustness, precision), rather than running a kernel on a few random inputs at one fixed shape and checking output closeness.
- The verifier audited 2,638 machine-generated kernels that a public system's own harness had already accepted as correct, finding 39.5% broken beyond any tolerance argument and 62.1% carrying at least one violation.
- The field's standard test accepts 1,487 kernels the new verifier rejects versus only 14 the other way — a roughly 106:1 asymmetry in pass/fail decisions.
- The authors defended their findings four independent ways: a 7/7 positive control, a threshold-calibration sweep, 98.5% agreement with the reference benchmark's own correctness code, and a stratified hand-audit.
- As a parallel contribution, the team delivered the first native Blackwell tcgen05 training backward for the gated-linear-recurrence (GDN) family — including the reverse-state stage the field still runs on a fallback — and validated it against a double-precision oracle before training five GDN members end-to-end.
Why it matters: The paper argues that reported 'high correctness rates' for LLM kernel generation are measured against a single loose test that misses real failure modes like NaN/infinity swaps, shape sensitivity, and fp16 accumulation — with the stricter verifier rejecting 1,487 kernels the standard test passes, weakening the evidence base the field uses to claim progress on automated GPU kernel synthesis.
Ask SkimNews
