Skip to content- Sources: primary
- Summary: The authors find that in standard on-policy distillation, style-related token positions show 3.5 times the mean teacher-student log-probability gap of mathematical tokens, so much of the update goes to imitating the teacher's phrasing. Student-Delta OPD instead uses centered differences between the student and its own pretrained base as update coefficients, keeps each only when its sign agrees with the teacher feedback, and recomputes them every round. Across five teacher-student pairs with Qwen3 students from 1.7B to 8B, it improves six-task math average pass@1 over OPD by 7.76 to 10.37 points, with code released.
- Why it matters: Teams distilling reasoning models can expect plain OPD to spend update budget on teacher style and may get larger gains by anchoring updates to the student's own post-training shift.
- Background: On-policy distillation trains a student on its own sampled outputs while a teacher scores each token, which gives dense feedback and avoids the mismatch of training only on teacher-written text. The digest covered two OPD papers on 2026-11-07, one on self-distillation from a GRPO checkpoint and one on a feedback loop behind OPD failures. This paper changes which tokens get the most update weight rather than the training loop.