Domain-Normalized MOPD: Rescaling Teacher Feedback Prevents Instruction Dominance in Multi-Expert Distillation

核心背景与行业痛点 / Background & Pain Points Reinforcement learning easily produces domain-specific specialist foundation models (e.g., mathematics, software engineering, instruction alignment), yet real-world applications demand unified generalist systems. Multi-Teacher On-Policy Distillation (MOPD) routes prompts to corresponding specialists whose token-level feedback guides a unified student model. However, standard MOPD students frequently underperform individual specialists and suffer catastrophic capability degradation on mathematical and algorithmic reasoning. ### 架构亮点与底层机制 / Architectural Highlights Researchers from SUTD and NTU uncovered the underlying mechanism and introduced Domain-Normalized MOPD (DN-MOPD): 1. The Instruction Dominance Trap: Discovers that supervisory feedback from instruction-following teachers exhibits several times higher variance and dispersion than feedback from mathematical specialists, overwhelming the student's shared parameter updates and washing out specialized reasoning priors; 2. Variance-Normalized Gradient Rescaling: Retains domain routing while dynamically normalizing teacher feedback by its empirical distribution spread, scaling down overly dispersed instruction feedback to establish parity with dense mathematical gradients; 3. Targeted Calibration: Proves that successful multi-specialist fusion requires calibrating feedback intensity rather than merely routing prompt assignments. ### 权威 Benchmark 与实测跑分对比 / Benchmark & Evaluation Extensive experiments on Qwen3.5 checkpoints across three model tiers demonstrated decisive advantages: - Consistent Cross-Benchmark Superiority: Outperforms conventional MOPD on six public benchmarks across all parameter sizes, three distinct random seeds, and both tight and generous output token constraints; - Full Mathematical Capability Recovery: Successfully preserves deep mathematical logic alongside conversational alignment, recovering gains previously lost during unnormalized multi-teacher runs; - Convergence Stability: Suppresses training instability and loss spikes caused by conflicting cross-domain gradient magnitudes. ### 开发者实战落地与开箱指南 / Developer Practical Guide - Preprint Citation: Detailed in arXiv preprint 2609.35347; - Training Recipe Adoption: Teams employing multi-teacher distillation in frameworks like TRL and OpenRLHF should introduce domain-variance scaling factors before backward passes; - Zero Compute Overhead: Implementation requires only scalar normalization during loss aggregation with zero impact on training throughput.