The Extender: Log-Structured Transformer Architecture Slashes Persistent Attention KV Memory Footprint by 104x

核心背景与行业痛点

Since its inception, the standard Transformer architecture has relied on the residual stream h as a singular superposition channel across layers. Because hidden activations are superposed additively, every subsequent layer must independently project and store dedicated key-value (KV) attention caches. Consequently, the persistent attention memory footprint scales at 2 L d_model per token. In long-context inference workloads (e.g., 128k to 1M tokens), KV caches routinely consume hundreds of gigabytes of VRAM—dwarfing model weights and imposing immense infrastructure expenses.

架构亮点与底层机制

Researcher Jakob Eriksson introduces The Extender (arXiv:2609.32759), a log-structured Transformer variant:

1. Dual-Channel Architectural Decoupling: Augments the residual superposition channel h with an explicit, lightweight concatenation channel x. Each layer l produces both an additive residual update delta_l to h and appends a minute extension vector epsilon_l to x (e.g., |epsilon_l| = 32).
2. Segregated Feed-Forward and Attention Inputs: The FFN and Query projections read the global superposition representation h, whereas Key and Value projections take exclusively the cumulative concatenation channel x as input.
3. Footprint Compression from 2 L d_model to sum |epsilon_l|: Because the extended vector x encapsulates the complete input required for the KV projections of all layers, layer-specific KV projection caching is streamlined into a minimal linear footprint.

权威 Benchmark 与实测跑分对比

Empirically validated on standard short-context suites and rigorous long-context benchmarks:

1. 104x Attention Footprint Reduction: For a 924M-parameter model with hidden dimension 1664 and |epsilon_l| = 32, The Extender's persistent attention memory footprint is 104x smaller than standard Multi-Head Attention (MHA), with savings expanding monotonically as model width increases.
2. Matches Baseline Accuracy on CORE Tasks: Across 199M to 924M parameter checkpoints, The Extender matches standard Transformer accuracy on short-context CORE evaluations without degradation.
3. Surpasses Standard Transformers on RULER Workloads: In rigorous long-context RULER evaluations, The Extender outperforms baseline Multi-Head Attention, overcoming traditional accuracy-memory tradeoffs.

开发者实战落地与开箱指南

The Extender outlines an architectural evolution for foundation model design, particularly for on-device deployment and massive batch concurrency. ML engineers training next-generation LLM backbones can adopt this dual-channel concatenation mechanism to dismantle persistent KV-cache scaling bottlenecks without relying on lossy quantization or restrictive sliding windows.