The Teacher Is a Direction, Not a Destination: RIDE Extrapolates RL Representation Residuals to Surpass Teacher Models

核心背景与行业痛点

On-policy distillation (OPD) trains student policies by aligning next-token distributions on student-sampled rollouts with an RL-privileged teacher. While generalized variants seek to allow students to surpass teachers via output-space reward extrapolation, they are hindered by intrinsic mechanics: language-model projection heads attenuate representation shifts anisotropically, and token-level log-probability ratios severely amplify sampling noise, inducing training divergence.

架构亮点与底层机制

Researchers introduce RIDE (RL-Induced Direction Extrapolation, arXiv:2609.36484), establishing that 'the teacher is a direction, not a destination':

1. Geometric Representation Shift: Demonstrates that reinforcement learning applies consistent geometric representation shifts across layers relative to base model weights.
2. Layer-Wise Residual Extrapolation: RIDE calculates the vector residual between the RL teacher and base model across all hidden states and token steps, regressing student representations toward targets extrapolated beyond the teacher along this directional vector.
3. Directional Linear Reward with Quadratic Penalty: Formulates the regression objective as maximizing a directional reward under a quadratic constraint centered on the teacher, enforcing stability while steering the student onward.
4. Circumvention of LM-Head Attenuation: Operates entirely within representation space, eliminating logit noise amplification.

权威 Benchmark 与实测跑分对比

Evaluated across four diverse base/RL-teacher pairs spanning distinct architectures and pretraining genealogies:

1. Consistently Surpasses RL-Trained Teachers: RIDE matches or outperforms the RL-trained teacher across every evaluated model pair, standing as the only method whose mean performance surpasses the privileged supervisor.
2. Overcomes Output-Space Collapse: Where output-space extrapolation degrades student performance when the teacher remains close to base weights, RIDE maintains consistent positive transfer.
3. Robust Reasoning Generalization: Delivers higher precision on multi-step reasoning rollouts without hallucination degradation.

开发者实战落地与开箱指南

RIDE is open-sourced at https://github.com/xixixixixxxx/RIDE. For ML engineers optimizing post-training pipelines and compact reasoning models, replacing output-space KL divergence with RIDE representation residual extrapolation offers an elegant pathway to create student models that outshine their RL teachers without executing expensive secondary RL loops.