Dan Luu argues AI coding value comes from feedback loops, not model test-writing

Category: Agentic coding
Status: discussion
Sources: Dan Luu post, HN discussion
Summary: Dan Luu argues that agentic coding gains come from wiring LLMs into feedback loops such as fuzzing and verification rather than from models being good at testing on their own. He reports that unguided LLM-generated tests are between worthless and marginally useful, and that fuzzing-benchmark results across models show high within-task variance (p-values spanning 0.04 to 1.0 across tasks), so small-sample model comparisons can support nearly any conclusion. The post references GPT-5.5 and GPT-5.4 and is a practitioner write-up without a single headline benchmark.
Why it matters: It reframes the practical question from which model is best to how to construct the harness and false-positive filtering, and warns that low-sample agent benchmarks are unreliable.