Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs Enables Single-Pass Dual Generation

核心背景与行业痛点

Deep neural networks are universally treated as highly non-linear operators where each forward pass processes a single semantic hypothesis. Generating divergent answers or exploring competing problem-solving hypotheses traditionally requires executing separate forward passes or managing multiple parallel sequences, linearly expanding KV cache memory traffic and computational workloads. Whether an LLM can maintain distinct parallel thoughts simultaneously within a single forward pass has remained an unresolved foundational inquiry.

架构亮点与底层机制

Researchers author 'Your Transformer Can Hold Two Thoughts at Once' (arXiv:2609.29845), proving the Superposition Linearity Hypothesis:

1. Architectural Linear Superposition: Discovers that when latent representations from distinct text inputs are linearly combined, the resulting next-token probability distribution closely matches the linear superposition (average) of their individual output distributions.
2. Inherent Architectural Property: Confirms this linearity is an innate geometric trait of the Transformer architecture that naturally attenuates over extended pretraining as activations saturate.
3. Restoration via Lightweight Adaptation: Demonstrates that targeted, low-cost fine-tuning restores clean linear superposition, closing the divergence between combined outputs and theoretical distributional averages.
4. Guided Disentangled Decoding: Introduces a decoding protocol that separates superposed representation states token-by-token, enabling a single forward pass to simultaneously emit two coherent, disjointed continuations.

权威 Benchmark 与实测跑分对比

Evaluated on synthetic and open-domain text synthesis traces:

1. 85% Reduction in Distributional Divergence: Targeted fine-tuning slashes Jensen-Shannon divergence between superposed predictions and averaged targets by over 85%.
2. Full Semantic Coherence Across Dual Paths: Text generations disentangled via guided decoding achieve perplexity and structural coherence scores statistically indistinguishable from standalone executions.
3. Doubling Test-Time Latent Throughput: Doubles the rate of candidate hypothesis generation per forward pass without requiring duplicated parameter memory access.

开发者实战落地与开箱指南

This study yields compelling implications for speculative decoding, tree-of-thought exploration, and inference systems engineering. By leveraging intrinsic latent superposition, developers can design speculative samplers and multi-agent planners that fold multiple analytical paths into unified forward passes, fundamentally reducing memory bandwidth bottlenecks in extreme low-latency environments.