Skip to content- Sources: primary
- Summary: Anthropic recommends matching eval design to observational or causal questions, measuring an appropriate response rate where one bad turn fails a whole conversation, and pairing harmful with benign prompts in each domain. It describes a tool where a domain expert writes a scenario spec, a Claude model plays the user, and a grader scores transcripts. It says single-turn harmless response rates sit between 94% and 98% for recent models, so those suites are now mainly regression checks.
- Why it matters: Anyone building model or agent evals gets concrete guidance on construct validity, grader calibration and saturation that applies beyond safety.
- Background: Construct validity asks whether a measure captures the concept it claims to measure. A refusal-rate eval rewards a model that refuses everything, which is not the same as safe behavior. The post borrows reforms from psychology's replication crisis, such as preregistration and better measurement.