The Sharpening Tax in Post-Training: How RL Alignment Sacrifices Agentic Solution Coverage and How PTGS Restores It

核心背景与行业痛点

Reinforcement learning (RL) post-training is broadly accepted as the default recipe to impart agentic decision-making capabilities. However, practitioners observe a persistent pathology: while post-trained models boast high single-shot accuracy (Pass@1), their performance plateaus quickly under repeated rollouts (Pass@K). When confronting complex bugs or open-ended multi-turn workflows, aligned agents repetitively converge on identical dead-ends, failing to produce alternative solutions despite generous test-time search budgets.

架构亮点与底层机制

Researchers from Stanford University and UW-Madison introduce 'Sharpening Tax in Post-Training' (arXiv:2610.01509), demystifying this post-training paradox:

1. Base Models Possess Superior Solution Coverage: Demonstrates that raw pretrained base LLMs equipped with lightweight execution harnesses serve as capable agents. Although trailing in Pass@1, base models consistently outperform post-trained variants in solution coverage (Pass@K) given sufficient sampling budgets.
2. The Sharpening Tax Formalization: Uncovers that RL post-training drives tasks toward binary extremes (always solved or never solved), trading off test-time diversity for sampling consistency. The authors formulate the Sharpening Tax metric to quantify this lost scalability.
3. Posterior-Tempered Group Sampling (PTGS): Proposes a plug-and-play Bayesian sampling strategy that dynamically adapts generation temperature per prompt according to estimated task difficulty.
4. Dual-Objective Synergy: Eliminates the diversity-accuracy compromise without architectural modifications, preserving high-entropy exploratory capacity on demanding tasks.

权威 Benchmark 与实测跑分对比

Evaluated across 14 base/post-trained checkpoint pairs from four model families across three interactive agent benchmarks (42 evaluations in total):

1. Universal Prevalence of the Sharpening Tax: Confirms that post-training reduces test-time scalability across all evaluated open-weights architectures, prematurely truncating the Pass@K saturation curve.
2. 18.4% Greater Problem Resolution: Applying PTGS enables agents to solve 18.4% more long-tail tasks under repeated sampling compared to standard fixed-temperature baselines.
3. Simultaneous Boost to Single-Shot Pass@1: Along with broad coverage expansion, PTGS improves single-shot Pass@1 accuracy by 3.2 percentage points on average.

开发者实战落地与开箱指南

This study provides an essential architectural guideline for teams architecting coding agents and system copilots. Optimizing solely for Pass@1 risks crippling the model's test-time search potential. Engineering teams should deploy adaptive Bayesian sampling mechanisms like PTGS and preserve un-sharpened representation paths to ensure agents retain the divergent creativity required to resolve mission-critical edge cases.