HuggingPapers highlights Salesforce AI Research’s Random Attention: keep the prompt and evict KV tokens uniformly at random per attention head. It matches learned evictors on reasoning tasks while delivering about 32–43% higher serving throughput in vLLM—a near zero-training cache-compression path for long-context coding and agent inference serving.

Key Takeaways

  • ✓Method: keep the prompt; evict KV tokens uniformly at random per head—no scorer.
  • ✓Result: matches learned evictors on reasoning; ~32–43% higher vLLM serving throughput.
  • ✓Impact: near zero-training cache compression for long-context coding/agent serving.
⚡

Key Decision Metrics at a Glance

TL;DR Verdict
🏆 Benchmark RankRank
Result: matches learned evictors on reasoning; ~32–43% higher vLLM serving throughput.
Full Leaderboard→
💰 Cost & PlansCompare
API & Subscription
Compare Plans↗
🛠️ Ecosystem & IDERules
Random Attention · KV cache · Salesforce AI Research
Setup IDE Rules↗
🧭

Not enough VRAM? Compare cloud API and self-hosting costs

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

⚔️ Compare Model→💰 Cloud cost calculator↗
Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

⚔️ Launch VS Compare→💳 View Plans↗
Up Nextllm-d 1.0 ships: the Kubernetes-native open-source distributed inference stack led by Red Hat, IBM and Google hits production readiness, with vendors reporting 3,000 self-hosted coding agents at 5-10x lower cost than commercial APIs
Read Now→