Framework20d ago
Eliminating Off-Policy Bias in Asynchronous RLVR: Proximal KL Enables Token-Wise Importance Sampling, Beating GRPO on Qwen3 with Zero Training Collapse
Off-policy data in reinforcement learning with verifiable rewards (RLVR) causes severe policy gradient bias during asynchronous generation and response reuse. Researchers from UIUC and HKUST introduce Proximal KL, theoretically proving that full future KL credit renders the exact proximal optimizer the unique stationary policy without requiring full-response weights or fitted critics. Evaluated across Qwen3-1.7B and 4B models, Proximal KL outperforms GRPO on 8 math benchmarks by up to 2.58% while completely averting training collapse.
Models22d ago
Salesforce Releases SRD: Self-Retrospection Distillation Turns Hindsight Experience into Prior Foresight for RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) powers modern reasoning LLMs by translating post-interaction scalar outcomes into optimization signals. However, in group-relative objectives (e.g., GRPO), reward signals vanish completely when all rollouts within a group share the same score (such as widespread mutual failures), discarding rich diagnostic feedback regarding task requirements and failure modes. In smaller models (such as 2B parameters), up to 98% of rollout groups result in universal failure, leaving standard RLVR stranded at 0% task success. Salesforce AI Research introduces Self-Retrospection Distillation (SRD, arXiv:2610.08077, repo: github.com/SalesforceAIResearch/SRD). SRD instantiates prospective learning, using privileged post-hoc trajectories to supervise pre-interaction foresight representations within the same policy without adding inference latency. Across 10 tool-integrated reasoning and long-horizon agentic benchmarks, SRD delivers gains of up to 24.2 percentage points over baseline RLVR and self-distillation. Critically, in 2B model regimes where 98% of groups are all-failure and standard RLVR flatlines at 0.0%, adding SRD reaches a 60.6% success rate under identical rollout budgets.
Framework22d ago
NVIDIA and UC Berkeley Unveil Post-Training Performance Predictors for Coding Agents from Base Checkpoints
Agentic post-training (scaling multi-turn RL and SFT) requires massive GPU investments, but AI labs lack reliable heuristics to predict which base model checkpoint possesses the highest potential before committing to multi-million-dollar post-training runs. Cold-start zero-shot evaluations on SWE-bench fail because unaligned base checkpoints cannot emit formatted tool invocations, while single-turn patch benchmarks ignore multi-step state management. Researchers from NVIDIA and UC Berkeley (including Bryan Catanzaro and Ashish Vaswani) present a lookahead evaluation framework (arXiv:2610.10478). By replaying successful post-trained trajectories and rerunning unit tests after every edit, they isolate the 'decisive step' that flips a repository from failing to passing. They introduce three lightweight evaluation screens at this decisive step that require zero cold-start tool calling: Decisive-Action Bits Per Byte (BPB), Patch MCQ, and prefix-conditioned pass@K. Across ten pairs of public base and post-trained models, these screens rank candidate checkpoints in tight agreement with final post-trained SWE-bench Verified pass@1 scores, establishing a low-cost lookahead methodology for model selection.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.