On November 11 Daniel Lenton, founder of YC-backed Unify, released Continual-ARC, a benchmark for whether agents accumulate skill over a working lifetime. It turns the three ARC-AGI editions into a lifelong stream: tasks return after dormant gaps with fresh inputs, new tasks keep arriving, and demonstrations and wrong answers both cost budget. After the stream the learner is frozen and tested on seen tasks (memorization) and unseen tasks (generalization). The finding: harnesses that learn through notes and skill files, such as OpenClaw, Hermes Agent and Prime Agent, mostly failed to keep anything useful and did not improve on new tasks, while training Qwen3.5-9B's weights to imitate a stronger teacher raised its held-out solve rate from 18.4% to 62.3%. The authors caution it covers one domain and a few models, none the strongest.
Key Takeaways
- ✓Stream: the ARC-AGI-1 track draws 25 of 400 training tasks over 200 instances, then tests fresh instances of seen tasks and 100 unseen tasks
- ✓Hermes Agent (GPT-6 Luna) opened a skill on 524 of 525 returns, but mostly stored general tips; its stream budget use U was 28.1, about the same as a demo-cache floor at 27.4
- ✓A program-library learner that stores one program per task cut U to 11.5 and solved 94.5%, showing the floors can be beaten
- ✓Weight learning (Qwen3.5-9B imitating GLM-5.3 Flash, 2 users): seen tasks 22.2% to 79.8%, unseen tasks 18.4% to 62.3%
- ✓Teacher-written notes actually fell on unseen tasks, 24.8% to 20.5%; code is open on GitHub under Apache 2.0
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the pain point
A popular idea is that agent harnesses can learn on the job through growing piles of Markdown notes and skill files. Existing benchmarks mostly cover one domain such as coding, serve each instance once, and rarely compare memory-based and weight-based learners on one scale, so memorizing answers cannot be told apart from learning a skill.
Design and mechanism
Continual-ARC uses ARC grid puzzles that need no domain knowledge. On each instance the learner sees only the task id and input; a demonstration costs 1 (up to 8), a wrong answer costs 1, and a failure costs the full 16. Returning tasks bring fresh inputs, so answers cannot be cached. After the stream the learner is frozen and tested for memorization and generalization. A collective mode lets several users share one set of notes or weights.
Results (author figures)
Persistence in the harness generally did not improve memorization without expert guidance, and even with it most harness methods learned less than a floor that simply caches demonstrations. No harness method gained on unseen tasks, while weight learning did (unseen 18.4% to 62.3%). Prime Agent never stored anything; OpenClaw kept every conversation but rarely looked back.
Getting started
The code is at github.com/unifyai/continual-arc; install with uv to build streams, run your own learner and submit it for verification. The site has step-by-step replays of every run and a leaderboard. The authors note it covers one domain and a few non-frontier models, and stronger models may retain more, so treat it as design input rather than a verdict.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.