Microsoft Research, Microsoft Security and UC Berkeley open-sourced TokenEconomizer on GitHub (Apache 2.0, v0.1.0 released October 26), a plugin for Claude Code, Codex, GitHub Copilot CLI and Pi, with experimental opencode support. It learns from earlier sessions in the same repository: micro-skills remember repo facts, tool pruning drops tool definitions that go unused, and a loop guard nudges an agent that is stuck. The paper reports 18.4–30.9% lower mean cost across five agent and model combinations on 277 SWE-Bench tasks (paper figures).
Key Takeaways
- ✓Cost: on 277 SWE-Bench tasks across 6 repositories and 5 agent and model combinations, mean cost fell 18.4–30.9%, and all 30 repository-by-combination cells got cheaper (paper figures, including the plugin's own model calls)
- ✓Success rate: Pass@1 rose in 4 of 5 combinations, e.g. Copilot CLI with Sonnet 5 cost 18.4% less with Pass@1 +2.6, while Claude Code with Opus 4.8 cost 30.9% less with Pass@1 -1.0
- ✓Three mechanisms: micro-skills (repo facts stated only after three sessions confirm them), tool pruning (by default a tool unused over 10 prompts across 2 sessions is left out), and a failure-loop guard (one redirect when a judge is at least 0.7 confident, never stopping a turn itself)
- ✓Generalization: 16–21% lower cost on Web-Bench and 6–20% lower on two security-auditing benchmarks where every task is a new repository
- ✓Deployment: Apache 2.0, Python 3.10+, no API key of its own, and learned experience stays local in ~/.tokeneconomizer
Key Decision Metrics at a Glance
Heavy Claude Code use: compare subscription limits and API bills
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the problem
Coding-agent bills behave like cloud bills: every turn re-sends the system instructions, every loaded tool definition and the conversation so far. Developers reuse the same agent on the same repository, so the same facts get rediscovered (for example, that only python -m pytest works here), the same unused tools get sent, and the same failing loops burn money. TokenEconomizer targets that repeated waste without changing the model or the agent's code.
How it works
It is an add-on plugin with three mechanisms. Micro-skills learn repository facts from failures that repeat across sessions and from a model review every 3 sessions; a fact is stated at session start only after three sessions confirm it, and is dropped if later sessions contradict it. Tool pruning removes, through the agent's own settings, a tool unused over 10 prompts across at least 2 sessions or over 5 sessions, and one command restores it. The failure-loop guard asks a judge model whether a turn is stuck when the same failures repeat; at 0.7 confidence or higher it injects one redirect, and it never stops a turn itself. Cost is computed with cache pricing and includes the plugin's own calls.
Benchmarks
All figures are from the paper (arXiv 2610.22732): on 277 SWE-Bench tasks from 6 repositories, mean cost fell 18.4–30.9%. By combination: Claude Code with Sonnet 5 down 23.6% (Pass@1 +1.3) and with Opus 4.8 down 30.9% (-1.0); Codex with GPT-5.6 Terra down 24.7% (+0.9) and with GPT-5.6 Sol down 26.1% (+1.6); Copilot CLI with Sonnet 5 down 18.4% (+2.6). Web-Bench cost fell 16–21% and security-auditing benchmarks 6–20%. There is no independent reproduction yet.
What developers can do
You need Python 3.10+ and a signed-in agent; the plugin needs no API key of its own. Claude Code users clone the repo, run claude plugin marketplace add and claude plugin install tokeneconomizer@tokeneconomizer, then /tokeneconomizer:status; Codex users run codex plugin marketplace add and codex plugin add. Gains are largest for teams that live in one repository, and smaller for one-off repos, so compare a week of bills before turning it on by default.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.