On October 21 Letta released FrontierMemory 1.0. Instead of quizzing agents on facts such as birthdays or hobbies, it gives an agent a set of past trajectories, has it write memory or skills, and scores that memory by how many coding and knowledge-work tasks a task agent then solves. Version 1.0 has 15 tasks adapted from public benchmarks such as DeepSWE, SkillsBench and APEX-Agents. On the model board (memory written in Letta Dream, tasks run in Letta Code) Opus 5.5 and GLM-5.3 both score 73.3% and Grok 4.7 scores 33.3%; on the harness board Letta's own Letta Dream ranks first. All numbers are Letta's own.
Key Takeaways
- ✓Model board (vendor-run, top effort): Opus 5.5 73.3% (about $2.80 per task), GLM-5.3 73.3% ($3.58), Kimi K3 66.7%, GPT-6.1 Sol 60.0% ($0.77, cheapest), Gemini 4 Argon 60.0%, Muse Wonder 46.7%, Grok 4.7 33.3%
- ✓Harness board: Letta Dream 73.3%, Prime Agent 60.0%, OpenAI Agents SDK 40.0%, Claude Managed Agents 33.3%, Hermes Agent 26.7%
- ✓Version 1.0 has 15 tasks; only 219 of 665 source trajectories are relevant, so the agent has to find the useful experience itself
- ✓Letta says 53% of the source tasks it kept had broken verifiers, which it repaired; memory generation is capped at 6 hours with no web access
- ✓Memory comes in three kinds (task conventions, user preferences, environment facts); outside models and harnesses can be submitted
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and pain points
In the past year nearly every major coding and personal agent gained file-based self-editing memory, and products such as the Claude Agent SDK, Letta Code and Devin can "dream" over past trajectories to update it. Existing memory benchmarks like LoCoMo and LongMemEval mostly test recall of facts, are close to saturated once agents can search files with bash, and say little about whether an agent actually improves from experience.
How it works
FrontierMemory turns tasks that normally carry all their context into tasks that need knowledge from earlier experience. The agent gets 665 past trajectories labelled success or failure (219 of them relevant), has 6 hours to write memory or skill files, and a task agent then uses that memory on the benchmark tasks; the score is the solve rate. Memory falls into three kinds: task conventions (a Word2Vec pipeline must skip files in omit.txt), user preferences (a changelog must be one line, newest first) and environment facts (grid limits live in config/grid/.env.local). At least two researchers reviewed each task, and each has a gold memory to prove it is solvable.
Benchmark results
These are Letta's own numbers over 15 tasks. The model board writes memory with Letta Dream, runs tasks in Letta Code and reports the better of the Opus 5.5 and GPT-6 Sol task models: Opus 5.5 and GLM-5.3 73.3%, Kimi K3 66.7%, GPT-6.1 Sol and Gemini 4 Argon 60.0%, Muse Wonder 46.7%, Grok 4.7 33.3%. GPT-6.1 Sol is cheapest at about $0.77 per task. On the harness board Letta Dream leads at 73.3%, then Prime Agent 60.0% and Claude Managed Agents 33.3%. Letta built the tasks and its own product is on the board, and with 15 tasks one task is worth 6.7 points.
Getting started
If you build agents with memory, the qualitative findings are the useful part: skill descriptions written as instructions that say when to use them get loaded more often, and rules overfit to one past run do not generalize. Task samples and submission details are at frontiermemory.com, so you can run your own harness and compare it against Letta Dream.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.