Skip to content- Sources: primary
- Summary: ALTO combines hybrid 1D and 2D block quantization, randomized Hadamard transforms, stochastic rounding and a simplified weight de-oscillation scheme on the MLPerf Small MoE Training benchmark. De-oscillation cut FP4 convergence overhead to the MLPerf target from 20 percent to 5 percent more steps than BF16. The authors also report negative results for differential gradient estimation, outlier clipping and macro-block scaling, and wall-clock speed was not optimized.
- Why it matters: It gives a concrete, open baseline and a list of techniques that did not help for teams evaluating FP4 training on MoE models.