AI Coding Live News

Off-policy data in reinforcement learning with verifiable rewards (RLVR) causes severe policy gradient bias during asynchronous generation and response reuse. Researchers from UIUC and HKUST introduce Proximal KL, theoretically proving that full future KL credit renders the exact proximal optimizer the unique stationary policy without requiring full-response weights or fitted critics. Evaluated across Qwen3-1.7B and 4B models, Proximal KL outperforms GRPO on 8 math benchmarks by up to 2.58% while completely averting training collapse.

Zoom

To improve credit assignment in long-horizon agentic tasks, combining reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD) has emerged as a leading paradigm. However, existing pipelines neglect the dynamic evolution of agentic skills and the reliability of privileged teacher signals. Researchers from CUHK and Tencent introduced RSIPO, enabling skills and policies to co-evolve recursively. RSIPO extracts skills directly from trajectory rollouts, optimizes skill generation via replay, and employs Empirical Bayes shrinkage to calibrate student trust in teacher logits. Operating on Qwen2.5-7B, RSIPO achieves 98.6% success on ALFWorld, outperforming standard GRPO by 17.4 percentage points.

Zoom

In reinforcement learning with verifiable rewards (RLVR) for reasoning models, standard PPO critics fail to track token-level value evolution along trajectories under sparse terminal feedback. LINs-lab introduced TPPO (Tracking-Value PPO), demonstrating that successive prefix values adhere to a Bayesian decomposition with additive updates in log-odds space. By parameterizing cumulative value outputs summed into value logits with terminal-only supervision, TPPO boosts aggregate mathematical reasoning by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, extending robustly to scientific coding.

Zoom

When model capabilities exceed human expert evaluative bandwidth, achieving Recursive Self-Improvement (RSI) without external verifiers or compilers becomes the paramount frontier of AGI. While reinforcement learning with verifiable rewards (RLVR) thrives in formal code and math, open-ended research and complex reasoning tasks lack scalar ground truth, and solitary models evaluating their own outputs suffer from homogeneous confirmation bias. Sakana AI introduces MASS (arXiv:2610.12176, project: sakana.ai/mass), an RSI framework anchored in Multi-Agent Self-Supervision. Driven by a single base model (such as Qwen3.6-27B), MASS tasks the model with concurrently acting as task solver, workflow optimizer, and self-evaluator. The framework alternates between evolutionary multi-agent workflow discovery and supervised fine-tuning (SFT) over synthesized trajectories. Across two successive MASS self-evolution cycles, performance per output token increases by 1.2x to 1.6x across open-ended research suites (MLR-Bench, ScienceAgentBench, DSBench, AstaBench). Head-to-head win rates against the base model surge from 53.9% in round 1 to 69.9% in round 2, while a student model trained on multi-agent traces decisively beats a single-agent baseline trained on 1.4x more tokens.

Zoom

Reinforcement Learning with Verifiable Rewards (RLVR) powers modern reasoning LLMs by translating post-interaction scalar outcomes into optimization signals. However, in group-relative objectives (e.g., GRPO), reward signals vanish completely when all rollouts within a group share the same score (such as widespread mutual failures), discarding rich diagnostic feedback regarding task requirements and failure modes. In smaller models (such as 2B parameters), up to 98% of rollout groups result in universal failure, leaving standard RLVR stranded at 0% task success. Salesforce AI Research introduces Self-Retrospection Distillation (SRD, arXiv:2610.08077, repo: github.com/SalesforceAIResearch/SRD). SRD instantiates prospective learning, using privileged post-hoc trajectories to supervise pre-interaction foresight representations within the same policy without adding inference latency. Across 10 tool-integrated reasoning and long-horizon agentic benchmarks, SRD delivers gains of up to 24.2 percentage points over baseline RLVR and self-distillation. Critically, in 2B model regimes where 98% of groups are all-failure and standard RLVR flatlines at 0.0%, adding SRD reaches a 60.6% success rate under identical rollout budgets.

Zoom

Reinforcement learning (RL) post-training is foundational to scaling reasoning capabilities in large language models. However, its memory footprint remains a prohibitive bottleneck: storing full-sized gradient buffers alongside multi-state Adam optimizer moments frequently triggers out-of-memory (OOM) failures when attempting to train large models (such as 27B parameters) on standard compute hardware. In newly released research (arXiv:2610.06647), researchers introduce LoGRA, an efficient RL post-training framework available within the open-source Molt library (github.com/skzhang1/labs-molt). LoGRA compresses training representations by retaining useful learning signals inside compact low-rank gradient sketches accumulated directly during backpropagation, facilitating both memory-efficient parameter updates and streamlined policy synchronization. To counteract instability from gradient compression, LoGRA introduces a Predicted-KL step control mechanism that forecasts policy change magnitudes prior to parameter updates, dynamically attenuating disruptive updates. Evaluated across reasoning benchmarks, LoGRA reduces average training memory by up to 45.7% with zero degradation in task performance, successfully sustaining stable RL training of a 27B model for over 1,100 steps on a single eight-GPU node where standard dense Adam crashes with OOM.

Zoom

Conventional multi-step agents are trained on flat action trajectories, where SFT and RLVR apply uniform loss weights across tokens while ignoring recurring sub-procedures that facilitate top-down hierarchical human reasoning. External skill-retrieval mechanisms fail to bake capabilities into model parameters and falter out-of-domain. Researchers from the University of Waterloo and collaborating institutions introduce X-Tree, an algorithmic framework that tokenizes reusable agent experience analogous to text BPE tokenizers without expensive LLM prompts. By scoring action spans via reusability and compiling canonicalized action flows into a hierarchical eXperience Tree, X-Tree integrates seamlessly into Offline RL, Online RLVR, and On-Policy Self-Distillation, delivering up to a 5.8% task success rate improvement across WebArena, ScienceWorld, and WebShop.

Zoom

Conventional LLM agents treat multi-step trajectories as flat token sequences, applying uniform loss weights and ignoring the hierarchical sub-routines that humans reuse across complex tasks. While prompt-based skills retrieve procedures in-context, they leave model weights unaltered. Inspired by text tokenizers that construct vocabularies through frequency statistics without LLM calls, researchers from Waterloo and UIUC introduce X-Tree. By scoring action-span reusability and merging recurring steps into an eXperience tree, X-Tree integrates into offline RL, online RLVR, and on-policy self-distillation, improving task success by up to 4.5% on WebArena, 5.8% on ScienceWorld, and 4.1% on WebShop.

Zoom

Recent leaps in language model performance stem predominantly from verifiable agentic training data rather than architectural revisions alone. However, generating verifiable tasks for supervised fine-tuning and reinforcement learning still relies heavily on expensive human-in-the-loop engineering, hindering recursive self-improvement (RSI). Researchers from CUHK and collaborating organizations introduce AutoDataBench, the first benchmark evaluating an agent's capability to autonomously synthesize verified training tasks against industrial production acceptance standards. Unlike traditional evaluation schemes requiring expensive end-to-end retraining runs, AutoDataBench directly inspects generated tasks across validity, novelty, difficulty, and behavioral coverage. Evaluated on three executable benchmarks, existing frontier agents score below 20/100 within a 45-minute budget; however, quadrupling the test-time budget significantly boosts success rates while maintaining unit costs, proving that autonomous synthetic data pipelines are viable when scaled with compute.

Zoom

Existing Computer-Use Agents (CUAs) face a dilemma: relying strictly on graphical user interface (GUI) interactions is slow and error-prone, while building per-application API tools demands unsustainable engineering and lacks generalizability. Researchers from Zhejiang University introduce HybridCUA, combining the UI versatility of GUIs with the execution speed and determinism of the Command Line Interface (CLI). Supported by the newly constructed HybridCUA-8K dataset (5K hybrid trajectories and 3K RLVR tasks), HybridCUA employs a two-stage training paradigm: supervised fine-tuning across interleaved GUI-CLI modalities, followed by CLI-aware reinforcement learning that rewards reliable, selective shell execution. On the OSWorld benchmark, HybridCUA-9B achieves 53.6% accuracy, improving over the base model by 14.8 percentage points, while delivering a 4.0-point gain on WindowsAgentArena.

Zoom

In reinforcement learning with verifiable rewards (RLVR), rollout trajectories are generated by high-throughput inference engines while backward gradients are computed by training engines, introducing subtle floating-point probability discrepancies that trigger unbounded variance under exact importance sampling. Researchers identified the invariance of logit displacement distributions and introduced Calibrated Importance Sampling (CIS), featuring confidence-aware truncation that replaces unbounded second moments with constant bounds, establishing top scores across five mathematical benchmarks.

Zoom