Skip to content- Sources: repository, HN discussion
- Summary: The repository publishes an instrument rather than a result, and states what it cannot do: swapping Opus 5 in place of Opus 5.5 was not distinguishable at 99 percent confidence, the design detects roughly a 7.5 point accuracy change per 10-day window, and the first possible call is around 2026-10-24. The method freezes the prompts, pins the CLI version, grades with pure functions rather than a judge model, runs a control model through the same harness, and commits the decision rule before collecting data.
- Why it matters: The pinned CLI is the load-bearing detail for anyone running the same test on a hosted model, because a harness update looks exactly like a model change.
- Follow-up: Read the first window's result around 2026-10-24, and check whether the pre-registered decision rule is applied unchanged.