On-Policy vs Off-Policy Distillation Dynamics: Cambridge Empirical Study Reveals Forward KL Invariance and Reverse KL Sensitivity

核心背景与行业痛点

On-policy learning is widely heralded as a foundational breakthrough in post-training, praised for curbing catastrophic forgetting and yielding superior generalization compared to offline supervised fine-tuning (SFT). However, prevailing empirical literature conflates multiple training hyperparameters simultaneously—varying rollout policies, token-level divergence objectives, and learning rate dynamics concurrently—leaving the true mechanistic drivers of distillation success obscured.

架构亮点与底层机制

Researchers from the University of Cambridge led by Mihaela van der Schaar present a rigorous controlled study (arXiv:2609.35259) across Llama-3 and Qwen-2.5 architectures:

1. Orthogonal Variable Disentanglement: Systematically isolates the standalone effects of rollout policies (along a continuous student-to-teacher spectrum), token-level KL divergence directions, and optimization learning rates.
2. Invariance of Forward-KL: Proves that Forward KL divergence is remarkably robust to rollout distributions, achieving stable and high performance whether fed by offline teacher rollouts or on-policy student rollouts.
3. Extreme Sensitivity of Reverse-KL: Demonstrates that Reverse KL is fragile, strictly demanding student-generated rollouts to avoid optimization divergence.
4. Learning Rates Govern Forgetting: Confirms that catastrophic forgetting and parameter update sparsity are primarily dictated by learning rate magnitudes and scheduler annealing rather than the on-policy nature of rollouts.

权威 Benchmark 与实测跑分对比

Evaluated on reasoning tasks spanning scientific deduction, clinical medicine, and complex arithmetic:

1. Flat Performance Curves Across Rollout Spectrum: Forward-KL maintains invariant accuracy across the entire spectrum of teacher-student rollout ratios, refuting the dogma that on-policy data is unconditionally superior.
2. Fragile Out-of-Distribution Advantages: While on-policy rollouts provide an edge on hard out-of-distribution Countdown arithmetic, this benefit does not reliably survive subsequent RLVR alignment.
3. Universal Robustness: Findings hold consistently across unclipped gradients, sampled Monte-Carlo KL estimators, and extended multi-step reasoning traces.

开发者实战落地与开箱指南

This study yields crucial practical cost-savings for distillation workflows. Teams operating with constrained inference compute budgets can confidently deploy offline teacher traces paired with Forward-KL loss functions, achieving parity with on-policy distillation without incurring the massive overhead of continuous online student rollouts.