Skip to content- Sources: primary
- Summary: The paper compares reverse-KL on-policy distillation with GRPO across task difficulty, model sizes and domains. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP and often inflates response length. Multi-teacher OPD that merges per-domain GRPO experts costs 5.2 to 8.0 percent of specialist training FLOPs and reaches joint GRPO performance on a 2-domain mix with 42 percent fewer total FLOPs.
- Why it matters: Teams post-training open-weight models get measured guidance that OPD pays off for consolidating domain RL experts, not as a cheaper replacement for RL with a stronger external teacher.