AI Coding Live News

While Large Language Models rely on deep non-linear activation layers, a groundbreaking mechanistic study titled 'Your Transformer Can Hold Two Thoughts at Once' (arXiv:2609.29845) demonstrates that transformer architectures exhibit fundamental linear superposition. When inputs from two distinct textual contexts are linearly combined in latent space, the model outputs a superposition of the individual next-token probability distributions. The authors reveal this superposition is an intrinsic architectural property that decays during pretraining but can be restored via lightweight fine-tuning. Furthermore, the team introduces a guided decoding procedure that cleanly disentangles superposed representations, enabling the simultaneous generation of two coherent, independent continuations from a single forward pass.

Zoom

A research group focused on mechanistic interpretability has released 'Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs' (arXiv: 2609.29845). Challenging the premise that non-linear activation layers render transformers hopelessly entangling, the authors formulate the Superposition Linearity Hypothesis: linearly combining input representations from distinct text streams yields an output probability distribution that is an exact linear superposition of the individual next-token predictions. Leveraging this intrinsic property, the team introduces a guided decoding method capable of extracting two distinct, coherent narrative continuations simultaneously from a single forward pass.

Zoom