Transformers Stop Thinking Too Early: A Tiny Single-Layer Rank-8 LoRA Fixes Latent Depth Relay to Skyrocket Accuracy from 15% to 99%

核心背景与行业痛点

Pretrained transformers are conceptually designed to perform hierarchical reasoning across their layered architectures. However, mechanistic interpretability reveals that base transformers terminate computation prematurely when following multi-step references in context. An empirical audit across 13 leading foundation models shows they reliably trace only 1.4 to 3.6 lines of reference dependency before representations plateau, rendering downstream layers functionally dormant regardless of nominal model depth or naive layer looping.

架构亮点与底层机制

Researchers author 'Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It' (arXiv:2609.36585), presenting a transformative diagnosis and low-rank remedy:

1. Single-Layer Rank-8 LoRA Trigger: Training a minimal Rank-8 LoRA adapter at just one early layer—with 100% of all other model weights frozen—sparks an autonomous cross-layer computational relay.
2. Chain Identity Propagation: The localized LoRA aligns early attention patterns, passing reference identities forward into intermediate layers. Dormant frozen attention heads progressively read higher up the dependency tree.
3. Zero-Cost Layer Localization Metric: Introduces an attention-based measurement protocol that locates the optimal intervention layer on frozen models without preliminary fine-tuning.
4. Scalable Reasoning on Looped Models: In weight-tied looped architectures such as Ouro-1.4B, the single LoRA enables linear depth utilization across iterative recurrent passes.

权威 Benchmark 与实测跑分对比

Evaluated on synthetic pointer reference chains and the real-world MuSiQue multi-hop question answering benchmark:

1. Accuracy Catapults from 15.5% to 99%: On 24-line reference chains, Qwen3-8B surges from a baseline exact accuracy of 15.5% to 99.0% with a single-layer Rank-8 adapter.
2. Pushing Limits Beyond 160 Reasoning Steps: Extended training unlocks 50 lines on Qwen3-8B. On Ouro-1.4B, depth scaling reaches 60 lines over four loops and exceeds 160 lines over eight loops.
3. Generalization to Downstream QA: Task-specific single-layer LoRAs yield substantial improvements on the complex multi-hop benchmark MuSiQue, confirming latent depth activation.

开发者实战落地与开箱指南

The authors have released their codebase alongside an interactive visual exploration tool at https://lunamos.github.io/stop-early. For AI practitioners building edge reasoning engines and code agents, targeting early transformer layers with micro-LoRAs provides an ultra-low-compute pathway to unleash massive latent reasoning capacity without post-training heavy parameter footprints.