AutoEnvScaling: Microsoft and Stanford Automate RL Environment Design via Terminal Agents, Unlocking 25x Data Flywheel

Background and the Problem

Terminal agents resolve complex challenges across software engineering, DevOps, and computational science by interacting directly with command-line shells. However, training robust terminal agents via reinforcement learning (RL) is fundamentally bottlenecked by the scarcity of diverse, verified interactive environments. Historically, RL environments have been meticulously hand-crafted by human developers, capping environmental supply to human engineering bandwidth. Pure prompt-based generation fails catastrophically due to missing dependencies, unexecutable configurations, and unverified validation suites.

Architecture and How It Works

To overcome this environmental bottleneck, Microsoft Research and Stanford University formulated AutoEnvScaling, framing interactive environment creation itself as a terminal execution task:

1. Proposer-Solver Flywheel: A proposer agent analyzes the rollout trajectories of the active solver policy to synthesize tailored, targeted RL environments, while new rollouts from the solver condition the subsequent round of environment generation;
2. Executable Terminal Construction: By commanding terminal tools to install binaries, populate filesystems, and verify exit codes, AutoEnvScaling boosts valid-environment generation rates by up to 25x over prompt-only generation, while slashing cost per valid environment by 75%;
3. Recursive Self-Improvement (RSI): Because both designing and solving environments are framed as terminal tasks, a single foundation model can simultaneously assume both roles, co-evolving without human intervention.

Benchmarks and Measured Results

Empirical evaluations across standard Terminal-Bench benchmarks demonstrate profound self-improving trajectories:

• +12.2 Points on Terminal-Bench 2.1: A single Qwen3.6-35B-A3B model operating in dual proposer-solver roles gained 12.2 percentage points on Terminal-Bench 2.1;
• Doubled Pass Rates on Terminal-Bench 4.0: On the notoriously challenging Terminal-Bench 4.0 suite, the co-evolved agent doubled its task resolution score from 1.6 to 3.2;
• Broad Cross-Domain Generalization: Autonomously synthesized verifiable execution sandboxes spanning software engineering, computer use, office workloads, and low-level GPU kernel programming.

Getting Started for Developers

AutoEnvScaling outlines a direct blueprint for developing recursive self-improving coding and DevOps agents:

• Paper Architecture: Review arXiv:2610.14163 for exact environment verifier schemas and proposer-solver prompting protocols;
• Automate RL Sandbox Factories: Encapsulate environment creation as Bash-executable sandbox tasks, allowing agents to configure containers with real package managers and build tools;
• Accelerate Low-Level Systems Tuning: Apply AutoEnvScaling to Triton/CUDA kernel synthesis, letting agents build isolated profiling harnesses to autonomously optimize hardware-aware kernels.