• Sources: primary
  • Summary: Jason Mancuso describes a recipe of GRPO plus masked importance sampling, score centering to remove accumulated drift, and top-p sampling with mask replay, ablated on Qwen 3.6 35B MoE with code environments. Because per-step weight deltas are 100 to 1,000 times sparse, compressed updates let inference workers run outside the training cluster across providers and GPU types, including FP8 pools, with stable reward. He says rollouts cost 15 percent less, and that the code builds on Miles and SGLang with a source release planned.
  • Why it matters: Teams with a production agent harness and evals can treat them as an RL environment and run rollouts on cheaper, scattered GPUs.
  • Background: GRPO, introduced with DeepSeek R1, scores a group of sampled answers against each other instead of training a separate value model. Asynchronous RL keeps samplers and trainer busy at once, but rollouts then come from slightly stale weights, and kernel or precision differences make the sampler's token probabilities differ from the trainer's. Importance sampling reweights tokens to correct for that mismatch.