OPPD: On-Policy Power Distillation Sharpens Sequence-Level Reasoning in a Single Pass, Matching 64-Candidate Search

Background and the Problem

Large language models performing multi-step formal reasoning rely heavily on inference-time compute scaling (Best-of-N, MCTS, beam search). Autoregressive generation dilutes the target mode: while the true solution has higher singular probability than any individual alternative, sub-optimal paths collectively absorb the majority of output probability mass. Power sampling sharpens output modes by scaling sequence probabilities by an exponent greater than one, but demands sampling and scoring dozens of trajectories per request.

Architecture and How It Works

OPPD (arXiv:2610.06804) resolves inference scaling constraints by distilling sequence-level power distributions directly into model parameters:

1. Search-Free Single Pass Generation: Compresses multi-sample search advantages into single forward passes, removing inference-time candidate sampling.
2. Sequential Monte Carlo (SMC) Teacher Distillation: The student generates candidates while a frozen teacher weights rollouts under a sequence-level power distribution, optimizing maximum-likelihood updates.
3. Reference-Free On-Policy Optimization: Operates without ground-truth answers or external reward models, deriving supervisory signals directly from calibrated model self-consistency.
4. Scalable Sharpening Modulation: A single loss hyperparameter calibrates the absorbed sharpening exponent between 1.19 and 2.02.

Benchmarks and Measured Results

Benchmarked across MATH500, GSM8K, AIME, and HumanEval:

1. Outperforms 64-Candidate Power Search: Increases single-generation accuracy by 23.0 points on MATH500 and 27.3 on GSM8K, outscoring published 64-candidate power sampling by 2.4 and 3.5 points.
2. Beats Verified-Reward GRPO: Surpasses GRPO trained with ground-truth verification by 3.8 (MATH500), 4.0 (GSM8K), and 5.4 points (AIME) under matched budgets without reference solutions. Stacking OPPD on post-GRPO checkpoints yields up to a 9.3-point cumulative boost.
3. Zero-Shot Transfer to Code Generation: Training on mathematics generalizes directly to code, lifting HumanEval pass@1 by 5.3 points.

Getting Started for Developers

The OPPD training framework is open-sourced at github.com/ArminAzizi98/OPPD. Teams deploying low-latency coding agents and quantitative reasoning services can replace costly multi-sample inference orchestrations with OPPD post-training, compressing multi-candidate search performance into single-pass inference.