Skip to content- Sources: paper
- Summary: The paper shows that small error in a quantized second moment does not imply small error in the preconditioner the next step actually uses. ZIP-SR keeps zero in the second-moment codebook and computes stochastic rounding probabilities in preconditioner space instead of state space, closing up to 70 percent of the gap to 32-bit state. The paper's recipes beat TorchAO 4-bit AdamW at every tested size from 130M to 2.7B, and the figures are the authors' own.
- Why it matters: Quantizing optimizer state to 4 bits is routine for fitting a larger model onto the same hardware, and the error that decides the outcome sits in the preconditioner rather than in the stored state.