A contract-grade verifier finds 39.5 percent of already-accepted LLM-generated GPU kernels broken beyond tolerance

Sources: primary, discussion
Summary: The paper applies twelve adversarial gates, several of them tolerance-free, to 2,638 machine-generated kernels that a public system's own harness had already accepted, and reports 39.5 percent broken beyond any tolerance argument and 62.1 percent carrying at least one violation. It reports the standard test accepting 1,487 kernels the verifier rejects, against 14 the other way. The authors defend the result with a positive control, a threshold sweep, 98.5 percent agreement with the benchmark's own correctness code, and a hand audit.
Why it matters: Reported correctness rates for kernel-generating systems rest on one loose test that runs a kernel on a few random inputs at one fixed shape, so published pass rates overstate what the kernels do.