Conversational memory for LLM agents traditionally divides into two opposing camps: expensive pipeline extraction that distills dialogues into structured factual memory stores versus raw conversational turn retrieval. Literature has reported contradictory findings. Researchers conducted a pre-registered, double-blind controlled study on held-out LoCoMo and LongMemEval benchmarks (arXiv:2609.34227). The investigation reveals that under constrained context budgets, selecting raw conversational turns via a lightweight typed decision model (Jev) is statistically non-inferior to full LLM fact extraction pipelines (95% one-sided bound -3.0 points vs. -5 margin)—while reducing write-time computational costs by a massive 3,061 times. Furthermore, the authors resolve literature contradictions by proving that reranking gains scale inversely with token budgets, while aggressive reranking degrades an agent's correct abstention reliability. All study protocols, data, and code are open-sourced at github.com/ris3abh/Engram.

Key Takeaways

  • ✓Presents Engram, the first pre-registered double-blind study testing raw-turn selection against LLM fact extraction for agent memory
  • ✓Shows raw turns selected via typed decision model Jev are statistically non-inferior to LLM fact extraction at 3,061x lower write cost
  • ✓Proves reranking gains decay sharply as token budgets expand, while aggressive reranking degrades an agent's correct abstention rate
⚡

Key Decision Metrics at a Glance

TL;DR Verdict
🏆 Benchmark RankRank
See how it ranks on the leaderboard
Full Leaderboard→
💰 Cost & PlansCompare
API & Subscription
Compare Plans↗
🛠️ Ecosystem & IDERules
Agent-Memory · Typed-Decision-Model · Turn-Selector
Setup IDE Rules↗
🧭

Turn your technical choice into a development budget

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

⚔️ Compare Model→💰 Compare plans & limits↗
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🌐
arXiv:2609.34227 - When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model
arxiv.org/abs/2609.34227
↗
📖
arXiv PDF - Engram Paper
arxiv.org/pdf/2609.34227
↗
🐙
GitHub - ris3abh/Engram
github.com/ris3abh/Engram
↗
🔬

In-Depth Technical Analysis

Background and the Problem

Conversational memory systems for autonomous agents face a fundamental design dilemma. Industrial frameworks frequently implement extraction pipelines that parse conversations into structured user facts using frontier LLMs before vector ingestion. This introduces an exorbitant write tax and risks extraction hallucination. The alternative paradigm stores raw conversational turns and relies on retrieval. Historical literature presented conflicting results regarding whether extraction outstrips raw turn selection.

Architecture and How It Works

Engram (arXiv:2609.34227) resolves this dilemma via a pre-registered double-blind empirical trial:

  1. Pre-Registered Double-Blind Methodology: Locks experimental hypotheses on Zenodo, utilizing frozen models, fixed context boundaries, and blind human evaluation to remove evaluator bias.
  2. Typed Decision Model Selection: Evaluates Jev, a lightweight decision model outputting categorical choice probabilities over candidate turns in a single forward pass without autoregressive generation.
  3. Budget-Dependent Dynamic Profiling: Traces how context budget constraints alter the cost-performance Pareto curve between extraction and raw retrieval.

Benchmarks and Measured Results

Evaluated on the multi-session LoCoMo and LongMemEval benchmarks:

  1. Statistical Non-Inferiority: Raw turns retrieved via Jev are non-inferior to full LLM extraction pipelines on LoCoMo under tight budgets (one-sided 95% bound -3.0 vs. -5 margin), confirmed by blind human graders.
  2. 3,061x Write Cost Reduction: Raw turn persistence cuts write-time token and compute overhead by a factor of 3,061 compared to pipeline fact extraction.
  3. Reranking Diminishing Returns and Abstention Cost: Reranking yields steep gains (+17.4 points) under strict 3-of-30 candidate pruning, but flattens (+1.1 to +1.5) at generous budgets, while reducing correct model abstention.

Getting Started for Developers

The Engram benchmark, evaluation suites, and Jev selection harnesses are open at github.com/ris3abh/Engram. Agent architects can replace expensive asynchronous fact extraction pipelines with raw turn logging paired with lightweight classifier retrieval, eliminating thousands of dollars in background LLM write taxes without sacrificing task accuracy.

Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

⚔️ Launch VS Compare→💳 View Plans↗