HuggingPapers highlights Salesforce AI Research’s Random Attention: keep the prompt and evict KV tokens uniformly at random per attention head. It matches learned evictors on reasoning tasks while delivering about 32–43% higher serving throughput in vLLM—a near zero-training cache-compression path for long-context coding and agent inference serving.

Key Takeaways

  • ✓Method: keep the prompt; evict KV tokens uniformly at random per head—no scorer.
  • ✓Result: matches learned evictors on reasoning; ~32–43% higher vLLM serving throughput.
  • ✓Impact: near zero-training cache compression for long-context coding/agent serving.
ADSponsored