• Sources: primary
  • Summary: Training transformers from scratch on synthetic biographies with controlled exposure counts, the authors find emulated NVFP4 needs 1.32 and 1.52 times as many exposures as BF16 to match recall while validation loss differs by 0.5% and 2%. Noise injected into the optimizer update carries about two thirds of the cost, and raising the learning rate does not recover it. Gradient statistics of BF16 checkpoints rank the cost of formats with a Spearman correlation of 0.96.
  • Why it matters: Teams evaluating low-precision training recipes should measure rare-fact recall alongside loss, since loss alone hides the gap.
  • Background: NVFP4 is a 4-bit floating point format with block scaling supported in Blackwell tensor cores, used to cut the memory and compute cost of matrix multiplications. Low-precision recipes are usually accepted when their loss curve tracks the BF16 baseline.