Terminal-RSI: Recursive Self-Improving Terminal Agents via Trajectory Self-Rewrite and Critic-Inheriting Policy Refinement Hit 86.52% on Terminal-Bench 2.1

Background and the Problem

In real-world command-line administration, container debugging, and automated build engineering, terminal agents must execute dozens of sequential Bash commands while parsing verbose stdout/stderr streams. Existing reinforcement learning pipelines for terminal agents face two fundamental bottlenecks: first, sparse binary verifier outcomes fail to convert partial troubleshooting progress into actionable policy updates; second, standard Generalized Advantage Estimation (GAE) decays value signals across thousands of non-policy environment observation tokens (such as compiler logs and stack traces), severely distorting long-horizon credit assignment.

Architecture and How It Works

To bridge partial local repairs and strict end-to-end task resolution, researchers from Tencent AI Lab and NUS introduced Terminal-RSI, a five-stage recursive self-improvement paradigm:

1. Alternating Recursive Self-Rewrite (RSR) and Policy-Refine (RPR): Starting from a Qwen3.8-27B backbone, the pipeline alternates trajectory consolidation and online RL (RSR-1 -> RPR-1 -> RSR-2 -> RPR-2). RSR rewrites verified trajectories into compact, noise-free execution paths, while RPR refines the policy using a Warm-up Critic Model, recursive critic handoff across stages, and stage-specific learning rate schedules;
2. Turn-Boundary Skip-Observation GAE (TB-GAE): Preserves fine-grained token-level baselines on agent action tokens while bypassing environment observation spans when applying temporal discount decay, restricting decay strictly to interaction turn boundaries;
3. Family Dense Reward (FDR) & Success-Prioritized Reward (SPR): FDR calibrates semantic progress across diverse task families, while SPR strictly separates full task completion from bounded partial credit to prevent reward hacking.

Benchmarks and Measured Results

Evaluated across standard and ultra-long-horizon terminal benchmarks, Terminal-RSI-27B establishes a new frontier for open-weight CLI agents:

• Dominant Terminal-Bench Results: Achieves 86.52% on Terminal-Bench 2.1, 79.00% on Terminal-Bench Hard, 16.95% on Terminal-Bench 3.0, and 19.84% on Terminal-Bench 4.0;
• Long-Horizon Breakthrough on LHTB: Elevates process reward on LHTB from 0.21 to 0.37 while lifting strict end-to-end completion from 0/46 to 5/46;
• Doubled Resolution on SWR100: Doubles strict resolution accuracy on SWR100 from 3.00% to 6.00%, confirming that alternating trajectory rewriting and critic-inheriting RL consolidates multi-step debugging capabilities.

Getting Started for Developers

Developers building autonomous DevOps and CLI coding agents can directly integrate Terminal-RSI's core mechanisms:

• Adopt TB-GAE in Multi-Turn Agent RL: Modify RLVR training frameworks (such as veRL or TRL) so that GAE discount factors ($\gamma \lambda$) skip over masked tool-output tokens and decay only across turn boundaries;
• Offline Trajectory Compaction: Implement an RSR stage between RL iterations to prune redundant exploratory shell commands from successful rollouts before policy distillation;
• Continuous Critic Handoff: Retain and transfer the value critic checkpoint across consecutive curriculum stages rather than re-initializing from scratch.