• Sources: primary
  • Summary: Likelihood-Gated Policy Optimization enforces a soft trust region by gating on likelihood, so it never computes old-policy likelihoods. The authors report that removing the importance sampling term does not harm stability while removing the trust region can cause collapse, and that LGPO matches GRPO's stability and peak performance while cutting training time by about 18% on average. In fully on-policy training the GRPO ratio stays at 1, its clipping never activates, and GRPO fails while LGPO stays stable.
  • Why it matters: Teams running GRPO-style RL on open weights can save a full forward pass per step and gain stability in fully on-policy setups.
  • Background: GRPO and PPO compute a ratio of the current policy's token probability to the policy that sampled the data, then clip that ratio to keep each update close to the old policy, a trust region. Computing the ratio needs an extra pass over the rollouts with the old weights. When training is fully on-policy, sampling and updating use the same weights, so the ratio is always 1 and the clip does nothing.