Dust: Q Labs pretrains transformers without backpropagation, matching or beating backprop at small scale and ~10³–10⁴× more efficient than weight-space ES, MIT-licensed code

Q Labs' Dust is a zeroth-order training method that replaces backprop's chain rule with a population estimate. It adds Gaussian noise to every linear layer's output independently at each token, rewards each perturbation by the change in that token's loss (with decayed future-token credit for keys/values), averages reward-weighted noise into an output-error estimate, and takes its outer product with the layer input to form the weight gradient. Because each token is a virtual population member, one forward pass evaluates thousands of members. On GPT-style transformers trained on FineWeb, Dust beats tuned backprop at 100k and 1M tokens, closes the gap with population at 10M and 20M (fitted 20M limit 4.431 vs backprop 4.633, loosely constrained), is about 10^3 to 10^4 times more efficient than the weight-space ES method EGGROLL, and gets more population-efficient with size up to 243M parameters. The authors say it is not yet compute-efficient enough to replace backprop; the MIT-licensed qlabs-eng/dust repo provides a minimal implementation and a backprop baseline.