Skip to content- Sources: primary, discussion
- Summary: The post masks the vocabulary to a fixed option set on Qwen3-1.7B, reaching 59.4 percent on a CommonsenseQA holdout and 62.4 percent after a short finetune. Calibration on the pre-finetune eval was poor. Predictions in the 0.9 to 1.0 confidence bin were correct only 70 percent of the time, and the 0.80 to 0.90 bin 47 percent across 121 predictions.
- Why it matters: Gating an agent step on raw option-token probabilities reads a next-token score as a correctness probability unless a temperature is fitted first.