Skip to content- Sources: primary
- Summary: In arXiv:2611.08016, submitted 2026-11-05, Hong and seven co-authors analyze the learning dynamics of on-policy distillation, which trains a student on its own generated trajectories under dense teacher supervision. They separate the feedback loop into conversion of the distillation signal into parameter updates and broadcast of those updates across prefixes, which reshapes the rollout distribution. The framework accounts for limited student responsiveness despite large teacher-student disagreement, and for degeneration when updates that improve local teacher agreement also raise the probability of unsuccessful continuations. They report that offline distillation can prepare a student for on-policy distillation without improving immediate held-out accuracy, and that on-policy distillation alone may degrade generation but yields substantial accuracy gains after sufficient offline preparation.
- Why it matters: It gives a concrete ordering for post-training pipelines: run offline distillation first, then on-policy, rather than going on-policy from the start.
- Background: Offline distillation trains a student on a fixed dataset of teacher outputs, so the training distribution never moves. On-policy distillation samples from the student instead and scores those samples against the teacher's next-token distributions, which keeps the training data on the student's own distribution but makes the data generator change with every update. That coupling is the same instability reinforcement learning fine-tuning faces, and it is why practitioners mix the two rather than choosing one.