• Sources: primary
  • Summary: The authors show that REINFORCE, the basis of PPO and GRPO, can push the probability of optimal responses near zero while reward rises, even with exact gradients and KL regularization to the base model. Repeatedly resetting the KL reference to the latest policy, formalized as Approximate Policy Mirror Descent, avoids this collapse. They prove global convergence for stylized PPO and GRPO variants with resets and for squared-loss methods like those used to train Kimi models.
  • Why it matters: RL post-training pipelines that keep a fixed base-model KL anchor have a theoretical reason to reset the reference periodically.
  • Background: In RL post-training, a KL penalty keeps the policy close to a reference model, usually the base model, to limit drift. Coverage here means how readily the base model already samples optimal responses, which bounds how much RL can amplify. Policy mirror descent solves a sequence of regularized subproblems, each anchored to the previous policy.