• Sources: primary
  • Summary: Neumann applies Goodhart's law to database benchmarks and says AI code generation lets systems tune for known queries regardless of soundness. He cites pgrust grouping strings by hash value alone, which can return wrong results. He proposes ad hoc benchmarks with an update stream and hidden or freshly generated queries, as in Kaggle and LiveBench.
  • Why it matters: Published analytical database benchmark wins now need a check for query-specific tuning before they inform a choice.