Cache Engineering for Agents: Tsinghua and CUHK Introduce AKV Paradigm Cutting Inference Costs by 27.7%

Background and the Problem

As autonomous agents engage in extended multi-turn trajectories—spanning complex codebase debugging, web tool use, and long-horizon planning—maintaining context continuity across dozens of interactions is paramount. Today's prevailing practice relies on Context Engineering: when context approaches memory limits, developers truncate text histories or synthesize heuristic summaries. This approach triggers severe serving inefficiencies: modifying or pruning the text prefix irrevocably invalidates upstream Prefix KV Caches in serving engines (e.g. vLLM, SGLang). The serving engine is forced to re-prefill the entire modified context from scratch, generating massive GPU compute overhead, spiking latency, and permanently losing granular variables. To resolve this architectural bottleneck, researchers from Tsinghua University and CUHK introduced Cache Engineering in arXiv:2610.28577.

Architecture and How It Works

1. Direct In-Cache Primitives:

• Replaces destructive prompt rewriting with direct KV-cache manipulation on GPU memory;
• drop Primitive: Empowers agents to exclude obsolete observation tokens (e.g., redundant terminal stack dumps, transient DOM payloads) from future attention maps, freeing VRAM;
• repos Primitive: Real-time recalibration of RoPE position encodings on retained keys, preserving causal relative distances without invalidating shared cache prefixes;

1. AKV Stateful Serving Architecture:

• Solves the history-dependence challenge where identical token sequences yield divergent cache states due to varied cache operations;
• Co-designs the AKV engine (thu-nics/AKV), integrating drop-aware radix trees and dynamic mask managers behind a stateless API wrapper;

1. Adaptive Rolling-Drop Policies:

• Enables agents to self-improve cache retention policies recursively, maintaining high-fidelity reasoning anchors while pruning operational debris.

Benchmarks and Measured Results

Evaluated extensively across complex multi-turn benchmarks including SWE-bench and API agent tasks (Table 1, Figure 3):

• Superior Task Performance Lift:
• Outperformed threshold-triggered text summarization by +1.7 benchmark points, preventing catastrophic context amnesia;
• 27.7% Cost Reduction:
• Eliminating re-prefilling cycles decreased total GPU inference expenditure by 27.7% per trajectory;
• Improved P95 request latency by 1.5x, unlocking substantial throughput gains on high-concurrency clusters.

Getting Started for Developers

• Paper & Source Repository: Available via arXiv:2610.28577 and GitHub at thu-nics/AKV;
• System Design Recommendation: Teams engineering autonomous agent platforms should migrate from superficial string truncation to AKV's cache-native pruning, reclaiming 25%+ of cloud serving budgets while securing deterministic memory persistence.