Scaling Properties of Same-Family On-Policy Distillation: Compact RL Teachers Transfer Better to Larger Students via Power Laws

核心背景与行业痛点

While reinforcement learning empowers models to execute complex chain-of-thought deductions, transferring this capability across distinct model scales via on-policy distillation (OPD) has historically relied on empirical heuristics. Critical theoretical questions have remained unanswered: Can a compact RL-specialized checkpoint supervise a larger base model (weak-to-strong OPD)? How do transfer velocity and peak accuracy scale with parameter counts? The absence of formal scaling laws forces engineering teams to execute compute-intensive trial-and-error distillation runs.

架构亮点与底层机制

Researchers from Zhejiang University and collaborating teams present the scaling properties of same-family on-policy distillation (arXiv:2609.32722):

1. Universal Useful-Transfer Regime: Identifies that early-stage OPD dynamics adhere to a strict linear trajectory where held-out task accuracy (gold score G) scales linearly with the square root of token-level reverse KL divergence from initialization.
2. Weak-to-Strong Superiority: Proves that in every observed weak-to-strong pairing, the larger student's peak score strictly exceeds that of its RL supervisor, demonstrating that compact RL checkpoints unlock latent capacity in larger models.
3. Empirical Power Laws for OPD: Formalizes quantitative power laws modeling peak score and transfer slope as explicit functions of student parameters, teacher parameters, and teacher accuracy.
4. Supervision Density Paradox: Reveals that teacher scale saturates when approaching student scale; for a matched accuracy score, smaller teachers transfer capabilities more efficiently with lower representational noise.

权威 Benchmark 与实测跑分对比

Evaluated across complex arithmetic, scientific reasoning, and algorithmic coding benchmarks:

1. Consistent Super-Teacher Student Peaks: Larger student models outscore their supervising RL teachers by 5.4 to 8.2 points across diverse weak-to-strong configurations.
2. High-Fidelity Power Law Accuracy: The fitted scaling laws predict final convergence performance with an error margin below 3.5% across held-out training runs.
3. Bootstrapping Weak-to-Strong Efficiency: Multi-stage weak-to-strong bootstrapping attains 98% of full RL reasoning parity while reducing compute expenditure by over 85%.

开发者实战落地与开箱指南

This study provides an actionable economic roadmap for LLM distillation. Instead of conducting costly reinforcement learning directly on 70B+ parameters, practitioners should run sample-efficient RL loops on compact 3B/7B models, transferring the acquired reasoning patterns onto large base models via OPD to slash compute costs by orders of magnitude.