AWS & UIUC Introduce Lean4AgenticRL: Verifiable Multi-Level Process Rewards in Lean4, Reaching 77.3% on TerminalBench

Background and the Problem

Reinforcement Learning (RL) has become the cornerstone for elevating Large Language Models to solve complex software engineering (SWE) and terminal-execution tasks. However, current agentic RL paradigms rely overwhelmingly on high-level, sparse outcome rewards (e.g., whether a test suite passes). This binary signal is severely deficient: when an agent executes a 30-step trajectory that derails at step 15, outcome-only rewards provide zero credit-assignment visibility. Conversely, naive heuristic process rewards frequently induce reward hacking. Researchers from Amazon Web Services (AWS) and UIUC unveiled Lean4AgenticRL (arXiv:2610.22448), grounding agent execution in formal mathematical proofs.

Architecture and How It Works

1. Multi-Level Process Verification via Lean4 Dependent Types:

• The first open-ended agentic RL framework that formalizes low-level tool calls, mid-level workflow stages, and terminal-state guarantees in the unified Lean4 interactive theorem proving environment;
• Establishes verifiable process bounds that eliminate intermediate hallucinated actions without relying on subjective reward models;

1. Workflow-Stage-Normalized GRPO:

• Upgrades Group Relative Policy Optimization (GRPO) by normalizing advantages per formal workflow stage, containing exploration variance and preventing multi-turn trajectory collapses;

1. The LAR-9B and LAR-27B Model Family:

• Applied to 9B and 27B parameter open backbones, producing agents with unmatched command-line precision and deterministic error-recovery loops.

Benchmarks and Measured Results

Evaluated comprehensively on TerminalBench-2.1 and SWE-bench (Table 1, Figure 1, Appendix D):

• TerminalBench-2.1 Evaluation:
• LAR-27B achieves a top score of 77.3%, leading all models in its size category;
• Outperforms base models by an average of +9.89%;
• Decisively surpasses the closest comparable open-weights baseline by +5.07%;
• SWE-bench Issue Resolution:
• Demonstrates sharp reductions in tool hallucination rates, endless terminal retry loops, and command parameter errors across multi-repository debugging tasks;
• Ablation Studies:
• Removing mid-level workflow stage rewards triggers a 42% collapse in convergence speed, confirming that verifiable intermediate constraints are crucial for credit assignment in long-horizon agentic RL.

Getting Started for Developers

• Research Reference: Full mathematical formalizations and harness specifications are provided in arXiv:2610.22448;
• Engineering Blueprint for Agent Post-Training: This work demonstrates that future coding agents should integrate formal verification checkers (such as Lean4 or structured bash state auditors) directly into the RL credit assignment loop. This prevents process drift and dramatically accelerates policy convergence without costly human feedback.