Observational study: reasoning effort beats extra tooling for first-try agentic code generation

Category: Paper
Status: developing
Sources: arXiv 2607.02436
Summary: A single-author observational study (Achint Mehta, 2026-07-05) ran 90 independent agent runs building the same specified application, a real-time retrospective board, scored on a fixed 14-criterion, 42-point functional rubric plus a visual review. It varied model generation, two agent harnesses, two reasoning-effort levels, a testing tool, and two design prompts. It reports that capability tier dominated: frontier models clustered near the ceiling while a low-cost local model scored 24 to 37 points. Container deployment was the most common defect, failing on first try in 44% of runs.
Why it matters: It is evidence that reasoning effort and model tier, not added tools or design prompts, drive first-try reliability in coding agents.
Follow-up: Watch for independent replication and release of the run data and rubric.