• Sources: primary, discussion
  • Summary: Dust reports pretraining transformers with a zeroth-order method at quality the authors describe as competitive with backpropagation. The reported runs reach 243M parameters and 1B tokens. The results are the authors' own, with no independent replication.
  • Why it matters: If credit assignment does not need a backward pass, architectures and hardware that backpropagation rules out come into range.