OpenAI publishes method for separating signal from noise in coding evals

Category: Agentic coding
Status: confirmed
Sources: OpenAI post, HN discussion
Summary: OpenAI published a write-up on measuring coding-agent performance, arguing that run-to-run variance and benchmark contamination make single-number scores unreliable and describing how it separates real capability changes from noise across repeated runs. The post is method and framing rather than a model or product release.
Why it matters: Reproducible evaluation methodology matters as teams pick between the several coding models released this week on vendor-reported benchmarks.