Eliminating Off-Policy Bias in Asynchronous RLVR: Proximal KL Enables Token-Wise Importance Sampling, Beating GRPO on Qwen3 with Zero Training Collapse

Background and the Problem

In reinforcement learning with verifiable rewards (RLVR) driving frontier mathematical and logical reasoning agents, massive cluster throughput is achieved by physically decoupling asynchronous rollout worker pools from central training pipelines. This asynchronous paradigm inherently generates pervasive off-policy trajectories: gradient steps advance the active policy while incoming rollouts originate from stale behavior checkpoints, exacerbated by cost-saving response reuse mechanisms. Historically, resolving off-policy discrepancies forced an intractable dilemma: sequence-level importance sampling multiplies autoregressive likelihood ratios across thousands of reasoning tokens, triggering catastrophic variance explosion; conversely, per-token likelihood weighting breaks trajectory-level credit assignment, introducing systemic policy gradient bias that precipitates severe degradation over iterative optimization loops.

Architecture and How It Works

To break this fundamental mathematical impasse, researchers from UIUC and HKUST introduced Proximal KL, a rigorous algorithmic framework that theoretically and empirically reconciles token-wise importance sampling with stationary proximal policy updates:

1. Uniqueness of Stationary Policy under Future KL Credit: The authors mathematically prove that under standard coverage and local expressiveness conditions, allocating full future KL credit renders the exact proximal optimizer the unique stationary policy of the update iteration. This holds true even over arbitrary mixtures of historical policies, completely dispensing with exploding full-response product weights or separate fitted critics;
2. Decoupling Proximal Reference from Behavior Policy: Unlike standard PPO, which clamps probability ratios around the sampling distribution via heuristic clipping, Proximal KL formally separates the proximal anchor from the behavior policy, replacing arbitrary epsilon-clipping with bounded future KL divergence constraints;
3. Truncated Future Credit Horizons: To guarantee finite-sample reward bounds and suppress multi-step variance drift across long thinking chains, Proximal KL implements an optimal horizon truncation mechanism, maintaining tight mathematical stability across time-varying behavioral distributions.

Benchmarks and Measured Results

Comprehensive empirical evaluations conducted on Qwen3-1.7B and Qwen3-4B backbones across eight rigorous mathematics benchmarks reveal decisive advantages:

• Consistent Superiority over GRPO: Across seven strictly matched off-policy configurations spanning varying learning rates, response reuse ratios, and severe rollout staleness, Proximal KL systematically outperforms GRPO, delivering micro-averaged mathematical accuracy gains of up to +2.58%;
• Breakthrough Gains on AIME24: On the demanding AIME 2024 benchmark, applying finite future horizons significantly boosts both pass@16 and mean@16 relative to infinite-tail formulations, demonstrating pinpoint gradient guidance on complex multi-step reasoning chains;
• Absolute Immunity to Training Collapse: In periodic behavior-refresh stress testing where standard Policy Gradient (PG) and CISPO suffered catastrophic policy collapse and numerical explosion, Proximal KL maintained unbroken monotonic improvement, achieving higher peak validation scores in significantly fewer policy updates than GRPO.

Getting Started for Developers

The official PyTorch implementation and core math formulations are available on GitHub at https://github.com/BiEchi/proximal-kl-core:

• Drop-in Loss Replacement: Teams running distributed RL training on Ray, vLLM, or DeepSpeed can replace traditional PPO/GRPO loss operators with ProximalKLLoss, maintaining compatibility with asynchronous streaming rollout architectures;
• Long-Horizon Reasoning Configuration: When training reasoning agents on extended chain-of-thought problems, configure the future KL truncation horizon to match typical sub-goal verification spans (e.g., 64–128 tokens) to minimize gradient variance while preserving causal correction;
• Resource Optimization: By eliminating the need for a dedicated Critic network while maintaining stationarity on off-policy rollouts, engineering teams can reallocate substantial GPU VRAM toward larger generation batch sizes, drastically accelerating RLVR training throughput.