HKU Introduces ReaDy for Hybrid Linear Attention: Recurrent States as Dynamic Context Accelerates B200 Decoding to 302 TPS

Background and the Problem

Speculative decoding accelerates autoregressive generation by drafting multiple speculative tokens in parallel. However, feature-based drafters (such as EAGLE or DSpark) condition predictions on target hidden features whose contextual emphasis is static and cannot align with the shifting semantic needs of subsequent draft positions. Recently emerging Hybrid Linear Attention (HLA) architectures combine Transformer self-attention with recurrent linear layers (like Mamba or RWKV), continually updating compact recurrent states that accumulate long-term historical context. Researchers from the University of Hong Kong (HKU) introduced ReaDy (arXiv:2610.20503), the first speculative decoding framework specifically designed to harness HLA recurrent states.

Architecture and How It Works

1. Recurrent States as Dynamic Drafting Context:

• ReaDy taps into the compact recurrent memory states maintained by HLA layers, utilizing a lightweight low-rank reader (Rank 8–16) to inject historical context into the drafting process;

1. Orthogonal Projection Decoupling:

• Decomposes the recurrent state via orthogonal projection, separating features that span the current query subspace from complementary information in the null space to avoid redundant drafting signals;

1. Targeted Cross-Layer Injection:

• Systematic layer-ablation reveals that tapping target recurrent states from layers 30 and 62 and injecting them post-attention into the final drafting layer delivers optimal speculative accuracy with virtually zero compute overhead.

Benchmarks and Measured Results

Evaluated on a single NVIDIA B200 GPU running Qwen3.8-27B (Tables 1–4):

• End-to-End Decoding Speed (Tokens Per Second, TPS):
• Standard autoregressive baseline: 126.15 TPS;
• SOTA speculative decoding baseline DSpark: 280.42 TPS (2.22x speedup, acceptance length $\tau = 4.17$);
• ReaDy reaches 302.78 TPS (2.40x speedup over AR, $\tau = 4.53$), gaining an additional +7.97% TPS (+8.50% acceptance length) over DSpark;
• Reader Rank Efficiency (Table 3):
• A reader rank of just 8 achieves 302.78 TPS, fully matching full-rank 128 (302.31 TPS) while adding negligible parameter overhead;
• Generalization: Demonstrates consistent speedups across complex mathematical reasoning (GSM8K, MATH500) and code synthesis (Evol-CodeAlpaca).

Getting Started for Developers

• Research Reference: Full architectural proofs and two-stage training configurations are detailed in arXiv:2610.20503.
• Deployment Advice: Teams serving hybrid linear attention models can plug in ReaDy's low-rank recurrent state reader to break past 300 TPS on NVIDIA B200 accelerators without serving external draft checkpoints.