AI Coding Live News

X Discussion: Vercel CEO Guillermo Rauch Says "Own Your Intelligence," Urging Local Models to Keep Spreading and Developers to Own Their Agent Harness and Data

Vercel CEO Guillermo Rauch posted on X on November 1: "Own your intelligence. Anyone that opposes this has something to sell you (perhaps your own intelligence back to you.)" He argued that local models need to keep proliferating; since the biggest models may be out of reach locally for now, the best option is to keep your options open: own your harness, data, connections and runtime, meaning your agents, and embrace open source. Within about 12 hours the post drew about 2,500 likes, 128K views and over 100 quote posts, with debate centered on how to balance local and cloud AI.

Key Takeaways

  • This is Rauch's personal opinion post, not an official Vercel announcement or product release.
  • The original post (2026-11-01 21:08 UTC) had about 2,533 likes, 255 reposts, 266 replies, 107 quotes, 128K views and 581 bookmarks at fetch time.
  • Core claim: local models should keep spreading; until the biggest models can run locally, keep your options open by owning your harness, data, connections, runtime and agents, and embrace open source.
  • Mark (@marksuman), co-founder and CEO of Maple AI, replied in a quote post that it will be hybrid: cloud will always be more powerful, so local-first AI should be paired with private cloud.
  • Engineer Shantanu Goel (@shantanugoel) quoted it saying "Guillermo says the truth that most others won't!", drawing about 37 likes.
Read full story
Linked Models & Agents:
g
guregodevo (Memdoor author)github.com·
👀14
🔥81

Memdoor reproduces GVS5H: a team of five Qwen3.8-27B agents averages 9.8/10 on the first 10 LiveCodeBench Hard problems, against 8.0 for a single Fable 5 call

On Oct 31 the open-source coding agent Memdoor (GitHub guregodevo/memdoor, Apache 2.0) published its team-solve workflow: using the GVS5H paper authors' exact prompts and loop, five Qwen3.8-27B instances act as planner, two ideators, solver and manager, talk only through files on disk, and a script that runs the tests decides success. Over five passes on the first 10 LiveCodeBench Hard problems from the GVS5H list it averaged 9.8/10, against 9.6 for the authors' own harness and 8.0 for a single Fable 5 call; 49 of 50 runs solved the problem for $35.79 in total, and every request and reply is public. The Hacker News post has about 200 points. This is a community reproduction on only 10 problems, and the author notes that training-data contamination cannot be ruled out.

Key Takeaways

  • Score: a mean of 9.8/10 over five passes (10, 10, 10, 10, 9), against 9.6 for the GVS5H authors' own harness and 8.0 for a single Fable 5 call (team-solve README, project-reported)
  • Cost: 49 of 50 runs solved, $35.79 and 15.3M output tokens in total; a median run took about 40 minutes of model time and $0.39, the longest 4 hours
  • Method: five roles (planner, two ideators, solver, manager) plus a repair worker each start from a fresh context, talk only through files on disk, and a shell script that runs the tests decides success instead of a model
  • Limits: only 10 problems rather than the paper's 100; the contests date from March and April 2025 and may be in training data; Fable 5's column comes from the GVS5H authors' result files and was not rerun in this harness
  • Reach: the HN post has about 200 points as of Nov 1, 16:10 SGT; the original GVS5H paper is arXiv 2608.26480, which reached #1 on HN on Oct 25
Read full story

NVIDIA says AI agents now steer the SETI Allen Telescope Array, calibrating it and analyzing up to 2.4 Tbps of live radio data from plain-language requests

An Oct 30 NVIDIA blog post says Breakthrough Listen and the SETI Institute have brought agents into daily SETI work at the Allen Telescope Array (ATA, 42 dishes of 6 m each) in Northern California: from a single natural-language request, agents can steer the antennas, calibrate and focus, fix data errors, and run AI models on the live stream. Holoscan's DAQIRI library pulls sensor data straight into GPU memory, the SETI Institute's CyberEther handles signal processing, and the web-based orchestrator Nexus acts as the agent harness, handling up to 2.4 Tbps on premises on RTX PRO 6000 Blackwell GPUs. The pipeline has made the first real-time AI detection of a pulsar in the Crab Nebula, and the team is evaluating open models such as Nemotron. The @NVIDIAAI Halloween post on Oct 31 drew about 274 likes.

NVIDIA says AI agents now steer the SETI Allen Telescope Array, calibrating it and analyzing up to 2.4 Tbps of live radio data from plain-language requests

Key Takeaways

  • Throughput: up to 2.4 Tbps of radio data on NVIDIA RTX PRO 6000 Blackwell GPUs, with inference running on premises next to the instrument (NVIDIA blog, vendor-reported)
  • Three layers: DAQIRI moves sensor data straight into GPU memory, CyberEther does the signal math with cuFFT and cuBLAS, and Nexus serves as the agent harness for instrument state, compute, and pipelines
  • Capability: one plain-language request can steer the ATA's 42 six-meter dishes, calibrate and focus, fix data errors, and plot results; NVIDIA says this compresses weeks or years of human work into hours
  • Milestone: the first real-time AI detection of a pulsar in the Crab Nebula; no false-positive rates or other metrics were published, and no extraterrestrial signal has been found
  • Next: evaluating open models such as Nemotron for observatories where data locality matters, possibly one day running offline on the far side of the Moon
Read full story

DeepSeek Harness team open-sources dsh-superagent, a proactive coding agent prototype for a single GitHub repo, alongside its Super-agent architecture article

On Oct 31, deepseek-ai open-sourced dsh-superagent-prototype on GitHub under the MIT license: a proactive coding agent that runs as a DeepSeek Harness bundle, lives on one GitHub repository and can optionally listen to Feishu. When an allowed person asks for work, it opens a work item, works in its own session and directory, and answers in the original conversation, for example with a pull request. The README states it is a research prototype, not an official DeepSeek product. The companion article, Exploring Super-agent Design, proposes a two-layer Router-Worker architecture; the team test, as reproduced by media, reports a p95 completion time of 15.7 minutes versus 29.5 minutes for a single-session setup.

Key Takeaways

  • Open source: deepseek-ai/dsh-superagent-prototype went public on Oct 31 under MIT, written in TypeScript, with about 1,800 stars at the time of writing
  • Scope: each deployment serves 1 GitHub repository, with optional Feishu; it acts only when an allowed person addresses it or a team-written rule allows it
  • Delivery: by default it only allows auto-merge for an approved pull request opened for a ticket with fewer than 200 changed lines, and never enables auto-merge as the agent account
  • Team test (as reproduced by media, vendor-reported): in a simulation with about 300 messages and 20 requests, p50 / p95 completion was 12.4 / 15.7 minutes versus 20.4 / 29.5 for a single session with a subagent tool
  • Risk: worker sessions run commands without approval as the operator's OS user, and the README asks users to review and adapt the code first
Read full story
ADSponsored

YC Labs pits AI agent swarms in Minecraft and Mindustry: GPT swarms win 12 of 16 head-to-head games against Claude, while a 50-agent colony slides into bureaucracy

YC Labs, a research project at Y Combinator, and multi-agent environment startup Olam Labs published "Agents in Games" on Oct 29. Fifty GPT-6.1 Sol agents told to "build a thriving civilization" in survival Minecraft spent three hours huddled in shelters rationing food, sending about 5,900 messages and inventing ration ledgers, permits and job titles. In nine Bed Wars games, 10 GPT-6.1 Sol agents beat 10 Claude Opus 5.5 agents 6–3; in seven 50-vs-50 Mindustry matches lasting about 9.5 hours, GPT won 6–1. The leader was almost always whichever agent was listed first in the directory, and stuck GPT agents fetched the game's public source code on their own. The Minecraft harness is open source. These are one lab's experimental observations, not a standardized benchmark.

YC Labs pits AI agent swarms in Minecraft and Mindustry: GPT swarms win 12 of 16 head-to-head games against Claude, while a 50-agent colony slides into bureaucracy

Key Takeaways

  • Head-to-head: GPT-6.1 Sol won Bed Wars 6–3 over nine games and Mindustry 6–1 over seven, 12 of 16 overall (see the YC Labs report)
  • Civ50: 50 GPT agents sent about 5,900 messages in three hours; 22 of 34 deaths came on the first night, after which the colony turned cautious
  • No election: 46 of the other 49 agents first messaged Ada, first on the roster, who received 616 DMs and became leader
  • Communication: in one Bed Wars game GPT players sent 1,147 DMs versus Claude's 82; GPT drifted into compressed shorthand while Claude voted with "+1"
  • Open source: the long-running Minecraft agent harness is public at yc-software/minecraft-agent-harness
Read full story
T
Tsinghua University & Z.AI (HarnessPro, arXiv 2610.26770)arxiv.org·
👀10
🔥80

Tsinghua and Z.AI open-source HarnessPro: agents rewrite their own harness, reaching 80.8% on Terminal-Bench 2 at 34% lower cost per task

HarnessPro (arXiv 2610.26770, submitted Oct 19 and revised Oct 20), from Tsinghua University, Z.AI (Zhipu) and collaborators, is a label-free way to improve an agent harness: specialized agents diagnose execution traces, rewrite instructions and runtime policies, score candidates with LLM-generated tests, and keep Lite, Medium and Heavy harnesses along the performance-cost frontier. Per the paper, with GPT-5.5 and Codex the Heavy mode reaches 80.8% on Terminal-Bench 2, 4.5 points above the prior RHO method, while cost per task drops from $1.09 to $0.72. The code is MIT-licensed and drew wider media coverage this week, including from Pandaily.

Tsinghua and Z.AI open-source HarnessPro: agents rewrite their own harness, reaching 80.8% on Terminal-Bench 2 at 34% lower cost per task

Key Takeaways

  • Terminal-Bench 2 (GPT-5.5 with Codex, mean of 3 runs): Heavy reaches 80.8% at $0.72 per task, versus RHO at 76.3% and $1.09, a 33.9% cost cut (paper figures, arXiv)
  • Averaged over three benchmarks: +2.0 points pass rate and -26.1% cost per task versus RHO, across Terminal-Bench 2, SWE-Bench Pro and ProgramBench
  • Holds across models: with GLM-5.2 and Claude Code on Terminal-Bench 2, Heavy hits 81.4% at $0.62, versus vanilla at 74.6% and $0.92 and RHO at 78.0% and $1.00
  • Budget modes: Medium matches RHO at 76.3% on Terminal-Bench 2 for only $0.56 per task; Lite gets 72.9% at $0.43
  • Cheap to train: only 10 training tasks per benchmark and 3 to 4 evolution epochs, tested on 59, 100 and 100 held-out tasks; the three harness tiers are MIT-licensed on GitHub
Read full story

Cartesia launches Managed Agents voice agent platform: #1 of 30 on Speech Agent Arena, $0.03 per minute at launch

Voice model company Cartesia launched Managed Agents on Oct 29, an end-to-end voice agent platform for enterprise phone calls that tunes its own speech-to-text and text-to-speech models together with the agent harness, and bundles telephony, knowledge bases, tool calls and outbound batches. Cartesia says it ranks #1 on Artificial Analysis's Speech Agent Arena with a 1115 Elo and scores 87.4% accuracy and 90.7% experience on ServiceNow's EVA-Bench (vendor-reported, with GPT-5.5 as the reasoning model). Launch pricing is 50% off at $0.03 per minute.

Cartesia launches Managed Agents voice agent platform: #1 of 30 on Speech Agent Arena, $0.03 per minute at launch

Key Takeaways

  • On Artificial Analysis's Speech Agent Arena blind preference test, Managed Agents ranks #1 of 30 agents with a 1115 Elo, ahead of Gemini 3.1 Flash Live Minimal (1099), Grok Voice 3.0 (1064) and GPT-Live-1 (1041) (Cartesia blog, as cited by the vendor)
  • First-attempt results on ServiceNow EVA-Bench across 200+ customer service, IT service and HR scenarios: 87.4% accuracy and 90.7% experience, versus 73.7%/71.2% for Grok Voice 3.0 and 59.0%/61.6% for GPT Realtime 2.1 (EVA-Bench)
  • All these benchmark runs use GPT-5.5 as the reasoning model; Cartesia's internal evaluation says 85% of calls had no turn-taking issues, versus 67% for other agents
  • Launch pricing is 50% off at $0.03 per minute; Cartesia says the platform has already handled tens of millions of conversations in finance, retail, hospitality and healthcare
  • It supports 44 languages; English, Spanish, Hindi, French and Japanese can switch mid-call today, with the rest coming soon
Read full story

Factory launches Droid 1.0: programmatic tool calling by default, hundreds of tool calls per model request, leads native harnesses on its own Terminal-Bench 4 runs

AI coding agent company Factory announced Droid 1.0 on X and its blog at 22:31 UTC on Oct 29, calling it the largest update yet to its agent harness. The key change makes programmatic tool calling (PTC) the default: the model writes a JavaScript program that runs up to hundreds of tool calls in a sandbox in one model request. In Factorys own Terminal-Bench 4 runs, Droid scored 57.6% with GPT-6.1 Sol ($0.79 per passed task) and 65.2% with Claude Opus 5.5 ($4.73), ahead of Codex, Claude Code, Cursor, OpenCode and Devin on the same models.

Factory launches Droid 1.0: programmatic tool calling by default, hundreds of tool calls per model request, leads native harnesses on its own Terminal-Bench 4 runs

Key Takeaways

  • Programmatic tool calling is now the default: the model writes JavaScript that runs in a sandbox with no file, network or system access of its own, making up to hundreds of tool calls per request, with independent calls run concurrently
  • Factory example: moving and verifying 151 Linear issues took 312 tool calls; direct calls needed about 36 requests and 3.1M input tokens, a Script needed 2 requests and 61K tokens
  • Terminal-Bench 4 (Factory runs, Oct 23-28, three trials per task): with GPT-6.1 Sol, Droid 57.6% vs Codex 54.0%, OpenCode 51.0%, Devin 45.0%; with Opus 5.5, Droid 65.2% vs Cursor and OpenCode 62.1%, Devin 59.6%, Claude Code 57.6%
  • The first request shrank from 20,887 to 13,055 tokens (-37%) and up-front tool definitions went from 11 to 2; newly loaded MCP tools are appended to keep the prompt cache intact, and a typical production request reads 98% of input from cache
  • Available now as a macOS and Windows desktop app, a terminal UI and the Droid SDK (@factory/droid-sdk); the benchmarks are vendor-run with no third-party reproduction yet
Read full story

Microsoft opens the GitHub-built Copilot harness in Foundry Agent Service: public preview, 20+ frontier models plus BYOK, billing from Dec 1

During GitHub Universe on Oct 28, Microsoft put the Microsoft Copilot harness, the shared agent execution layer behind Copilot Code, Copilot Studio, the GitHub Copilot app, the CLI and VS Code, into Foundry Agent Service as a public preview. Microsoft runs the agent loop and sandbox while developers supply instructions, skills, tools and model choice; Microsoft says it works across 20+ frontier models plus bring-your-own-key. Long-running resilience for hosted agents also reached GA. Harness runtime usage is free until Dec 1.

Microsoft opens the GitHub-built Copilot harness in Foundry Agent Service: public preview, 20+ frontier models plus BYOK, billing from Dec 1

Key Takeaways

  • Microsoft says the harness spans 20+ frontier models plus bring-your-own-key, so you can swap models without rewriting orchestration (official blog)
  • Preview building blocks: reusable skills, Foundry and MCP tools with explicit auth and approvals, file and code execution with built-in subagent delegation, and source-controlled agent definitions
  • Pricing: harness runtime is free until Dec 1; after that sandbox compute is USD 0.0994 per vCPU-hour and 0.0118 per GiB-hour, with model inference billed separately (Foundry blog)
  • Long-running resilience for hosted agents went GA the same week: interrupted work restarts with its original input, and your app saves checkpoints
  • Limits: you pick the harness only when creating a new agent; existing agents are unchanged and switching later is not supported yet
Read full story

Cursor launches Agent SDK: agents as code with Markdown prompts, TypeScript tools, built-in evals, and hosted or self-run deployment

On Oct 26 Cursor announced that its Agent SDK is generally available. An agent's system prompt is written in Markdown and its tools in TypeScript; agents can be deployed on Cursor's hosted infrastructure or run on your own machines, and built-in evals measure every change. Cursor's docs say it uses the same Cursor SDK harness that powers Bugbot, Security Review, Approval Bot and Rollouts, can be triggered from source-control events, issue trackers, Slack or schedules, and is managed under Deployed Agents after deployment. Cursor has not published pricing or benchmarks.

Cursor launches Agent SDK: agents as code with Markdown prompts, TypeScript tools, built-in evals, and hosted or self-run deployment

Key Takeaways

  • Same harness: Cursor's docs say Agent SDK runs on the harness behind four of its own agents: Bugbot, Security Review, Approval Bot and Rollouts
  • Project layout: bot/instructions.md is the required always-on system prompt, each bot/tools/*.ts file is one typed tool, with folders for skills, MCP connections, subagents, channels, hooks and schedules, and evals live in evals/*.eval.ts
  • Triggers and deployment: runs can be triggered from source control, issue trackers, Slack, webhooks or schedules; deploy on Cursor-managed hosting or self-host, then manage under Deployed Agents
  • CLI: agent-sdk validate, run and eval check the project, run a real turn and run evals; Cursor says only run and eval need a Cursor credential
  • Cursor has not published pricing or benchmarks
Read full story
Linked Models & Agents:
ADSponsored

FrontierCS 2 benchmark: agents must rebuild a paper's deleted contribution in its own repo and beat the authors' code; Claude Opus 5.5 leads at 58%, 16 tasks unsolved

Teams from UC Berkeley, Princeton and others released FrontierCS 2 (arXiv 2610.29932): 48 tasks across 13 CS domains, each built by deleting a paper's contribution from the authors' released code. An agent has 5 hours to write it back and passes only if it beats the original on every metric and every hidden workload. Of 12 agents, Claude Opus 5.5 in Claude Code leads with a 58% pass rate, every other agent passes fewer than half, and 16 tasks have no passing run. On Oct 26 maintainer Wenhao Chai said a trace audit showed the Pi harness ends runs on refusals; rerun in Antigravity, Gemini 4 Argon now beats GPT-6 Astra and sits right behind Opus 5.5.

Key Takeaways

  • 48 tasks, 13 domains, 12 agents, one run per agent per task (paper)
  • Strict pass rule: the final submission must beat the authors' code on every metric and every hidden workload; a higher total score can still fail
  • Claude Opus 5.5 in Claude Code leads with a 58% pass rate; 16 tasks have no passing run
  • Under the Pi harness, 10 of Gemini 4 Argon's 48 runs ended on provider refusals; rerun in Antigravity it beats GPT-6 Astra (Wenhao Chai's post)
  • Tasks and evaluators are open source on GitHub; the authors estimate a full 48-task run with Opus 5.5 at about $1,000 in API fees, $300 GPU and $100 CPU
Read full story

BOSS Zhipin's Nanbeige lab open-sources Nanbeige4.3-33B, a 33B dense agent model with 512K context under Apache 2.0, reporting 80.9 on Terminal-Bench 2.1 vs 73.0 for Qwen3.8-27B

At 12:11 UTC on 2026-10-25, the Nanbeige LLM Lab of BOSS Zhipin (Kanzhun Limited) announced Nanbeige4.3-33B from its account @nanbeige (post), with the model and its Base version released as open weights on Hugging Face under Apache 2.0 (model card). It is a 33B dense model on a Llama-style architecture with a 512K context window, positioned as the teacher model of the Nanbeige4.3 series that will be distilled into smaller models such as Nanbeige4.3-3B. The official model card compares it with Qwen3.8-27B, Muse-Glimmer-30B and Gemma4-31B using the same harness at highest thinking effort: 80.9 on Terminal-Bench 2.1 (Qwen3.8-27B 73.0) and 65.5 on DeepSWE (42.5), but 60.3 on SWE-Bench Pro, slightly below Qwen's 61.7, and 84.8 on GDPval vs 86.4. All figures are self-reported and not yet independently reproduced.

Key Takeaways

  • Official sources: @nanbeige post on X and the Hugging Face model card; the post had about 616 likes, 79 reposts and 38K views when we pulled it
  • Specs: 33B dense, Llama-style architecture, 512K context, Apache 2.0, with a Base checkpoint released too
  • Self-reported coding scores: Terminal-Bench 2.1 80.9, Terminal-Bench 4.0 23.8, DeepSWE 65.5, SWE-Bench Pro 60.3 (Qwen3.8-27B: 73.0, 6.3, 42.5, 61.7)
  • Self-reported cowork scores: Automation Bench 54.2, JobBench 44.8, GDPval 84.8 (Qwen3.8-27B: 38.3, 34.5, 86.4)
  • Deployment: Transformers, SGLang and vLLM, with official examples using 4-way tensor parallelism; Nanbeige4.3-3B distillation is next
Read full story
Linked Models & Agents:
⚡ Nanbeige4.3-33B⚡ Qwen3.8-27B⚡ Gemma4-31B⚡ Muse-Glimmer-30B🌐 Nanbeige🌐 BOSS Zhipin🌐 Kanzhun

Anthropic guide: fold most subagents back into one main agent loop with on-demand skills and tools; one customer ran about 2x faster

On 2026-10-22, Anthropic's Applied AI team published "Building agents: When to fold subagents into your main agent loop" on claude.com, along with an open-source reference implementation, anthropics/single-agent-loop-kit. The article says that across several enterprise deployments, a single agent with skills frequently beat both the one-big-prompt design and the subagent design on quality, often at lower cost and latency per task; one customer that swapped its multi-agent harness for a single loop saw roughly 2x faster runtime. Anthropic also says it cut Claude Code's system prompt for more advanced models by over 80%. Subagents still make sense in three cases: measurable context bloat, parallel tasks with disposable context, or trouble selecting tools and conflicting instructions. The piece has drawn wide discussion on X over the past two days.

Anthropic guide: fold most subagents back into one main agent loop with on-demand skills and tools; one customer ran about 2x faster

Key Takeaways

  • Original article: Building agents: When to fold subagents into your main agent loop, dated 2026-10-22 on the page
  • Customer result (vendor-reported): one customer comparing its multi-agent harness with a single agent loop saw roughly 2x faster runtime with the single loop
  • Anthropic says it cut Claude Code's system prompt for more advanced models by over 80%, partly by progressively disclosing skills and tools
  • Keep subagents in only 3 cases: measurable degradation from context bloat, parallel tasks with disposable fresh context, or difficulty choosing tools and conflicting system-prompt instructions
  • Companion open-source single-agent-loop-kit migrates in three steps (measure, combine, fold); Anthropic suggests starting with an end-to-end eval of as few as 10 golden examples
Read full story

Artificial Analysis launches Coding Agent Index v2.0: FrontierCode from Cognition replaces DeepSWE, and Claude Code with Opus 5.5 leads at 61.7

On 2026-10-23 Artificial Analysis launched Coding Agent Index v2.0, which scores model-plus-harness combinations across 290 tasks with three equally weighted benchmarks: FrontierCode v1.1 from Cognition (100 tasks on real open-source repos, graded on whether a maintainer would merge the change) replaces DeepSWE v1.1, SWE-Atlas-QnA becomes the stricter SWE-Atlas-QnA-AA, and Terminal-Bench 4.0 is unchanged. Claude Code with Claude Opus 5.5 (max) leads at 61.7, ahead of Sonnet 5.5 at 59.3 and Muse Wonder Full at 58.9. Scores fall across the board: the 21 configurations measured on both versions drop from a mean of 54.8 to 47.4.

Artificial Analysis launches Coding Agent Index v2.0: FrontierCode from Cognition replaces DeepSWE, and Claude Code with Opus 5.5 leads at 61.7

Key Takeaways

  • Top of the index: Claude Code with Claude Opus 5.5 (max) 61.7, Claude Sonnet 5.5 (max) 59.3, Muse Code with Muse Wonder Full (max) 58.9, Claude Fable 5.1 (max with fallback) 57.6 (Artificial Analysis figures)
  • On FrontierCode v1.1, Opus 5.5 scores 56.0%, ahead of Codex with DeepSeek V4.1 Flash at 52.6%; 15 of 24 configurations sit between 43% and 49%
  • Version shift: 21 configurations average 54.8 on v1.5 and 47.4 on v2.0, with 20 of 21 moving down; the same set averages 45.1% on FrontierCode versus 65.6% on DeepSWE
  • Four points on the cost frontier: DeepSeek V4.1 Flash 47.3 at $0.39 per task, GPT-6.1 Sol 54.1 at $1.50, GPT-6 Astra 54.7 at $7.04, Opus 5.5 61.7 at $11.42 (pay-per-token API costs, not subscriptions)
  • SWE-Atlas-QnA-AA rewords 127 of 1,306 rubric criteria and uses GPT-6 Luna (xhigh) as judge with five votes per criterion; dataset and judge config are on Hugging Face
Read full story

Letta releases FrontierMemory 1.0, a benchmark that scores agent memory by downstream task success; Opus 5.5 and GLM-5.3 tie for first at 73.3%

On October 21 Letta released FrontierMemory 1.0. Instead of quizzing agents on facts such as birthdays or hobbies, it gives an agent a set of past trajectories, has it write memory or skills, and scores that memory by how many coding and knowledge-work tasks a task agent then solves. Version 1.0 has 15 tasks adapted from public benchmarks such as DeepSWE, SkillsBench and APEX-Agents. On the model board (memory written in Letta Dream, tasks run in Letta Code) Opus 5.5 and GLM-5.3 both score 73.3% and Grok 4.7 scores 33.3%; on the harness board Letta's own Letta Dream ranks first. All numbers are Letta's own.

Letta releases FrontierMemory 1.0, a benchmark that scores agent memory by downstream task success; Opus 5.5 and GLM-5.3 tie for first at 73.3%

Key Takeaways

  • Model board (vendor-run, top effort): Opus 5.5 73.3% (about $2.80 per task), GLM-5.3 73.3% ($3.58), Kimi K3 66.7%, GPT-6.1 Sol 60.0% ($0.77, cheapest), Gemini 4 Argon 60.0%, Muse Wonder 46.7%, Grok 4.7 33.3%
  • Harness board: Letta Dream 73.3%, Prime Agent 60.0%, OpenAI Agents SDK 40.0%, Claude Managed Agents 33.3%, Hermes Agent 26.7%
  • Version 1.0 has 15 tasks; only 219 of 665 source trajectories are relevant, so the agent has to find the useful experience itself
  • Letta says 53% of the source tasks it kept had broken verifiers, which it repaired; memory generation is capped at 6 hours with no web access
  • Memory comes in three kinds (task conventions, user preferences, environment facts); outside models and harnesses can be submitted
Read full story