Models25d ago
HKUST Uncovers the Mechanistic Reality of Reasoning Distillation: On-Policy Distillation Teaches Compositional Skills, Not Factual Knowledge
The emergence of reasoning models such as OpenAI o1/o3 and DeepSeek-R1 sparked widespread efforts to distill dense reasoning traces into smaller 7B and 14B student models, unlocking striking performance leaps on math and coding benchmarks. However, a fundamental mechanistic question has lingered: does On-Policy Distillation (OPD) inject new parametric factual knowledge into the student, or does it merely impart compositional reasoning skills? Researchers from Hong Kong University of Science and Technology (HKUST) address this mystery in a landmark study (arXiv:2610.09639). Utilizing a controlled synthetic framework across four models from three distinct architectures, the authors independently isolate factual memory from multi-step compositional ability. The experiments establish that reverse-KL OPD reliably transfers compositional problem-solving skills across unseen task structures, but transfers minimal factual knowledge. Replacing reverse-KL with forward-KL restores factual memorization, while student rollouts specifically refine multi-step logical execution. These findings demonstrate that on-policy distillation does not expand a model's knowledge boundary, but rather teaches it to mobilize and organize facts it already possesses.
Research32d ago
CorrGRPO Solves Multi-Reward Imbalance in GRPO: Pearson Correlation Normalization Boosts Code & Tool Calling
Group Relative Policy Optimization (GRPO) forms the reinforcement learning foundation for reasoning models like DeepSeek-R1 by normalizing group rewards to derive advantage estimates. However, practical LLM alignment requires balancing multiple competing rewards, such as code execution correctness, format compliance, tool-calling precision, and safety. Standard GRPO computes advantage normalization across aggregate variance, causing large-scale, highly correlated reward components to overshadow critical fine-grained signals. HKUST researchers propose CorrGRPO (Correlation-Normalized GRPO), which normalizes reward covariances into Pearson correlation coefficients. Across code generation, tool invocation, and agent safety benchmarks spanning 0.5B to 8B models, CorrGRPO systematically outperforms standard GRPO.
Tooling70d ago
Codeium Unveils Windsurf Enterprise: Self-Hosted Cloud Clusters & SOC2 / HIPAA Compliance
Codeium released Windsurf Enterprise Edition supporting air-gapped on-premise VPC clusters and SOC2 compliance.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.