Dust claims zeroth-order pretraining of transformers competitive with backpropagation
- Sources: primary, discussion
- Summary: Dust reports pretraining transformers with a zeroth-order method at quality the authors describe as competitive with backpropagation. The reported runs reach 243M parameters and 1B tokens. The results are the authors' own, with no independent replication.
- Why it matters: If credit assignment does not need a backward pass, architectures and hardware that backpropagation rules out come into range.