Tokenizing Reusable Experience: X-Tree Recovers Hierarchical Skill Vocabularies to Boost Autonomous Agent Success by up to 5.8%

核心背景与行业痛点

Training multi-step agents across web automation, code synthesis, and embodied environments is hampered by low sample efficiency. Established paradigms, including supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), model action trajectories as flat token streams. This uniform treatment ignores recurrent sub-procedures that underpin human hierarchical planning. Recent attempts utilizing context-based skill retrieval (RAG) keep routines in the prompt rather than baking them into weights, failing to generalize across out-of-distribution environments.

架构亮点与底层机制

Researchers from the University of Waterloo and collaborating teams introduce X-Tree, applying tokenization principles to agentic decision-making:

1. Prompt-Free Experience Tokenization: Drawing inspiration from subword BPE tokenizers, X-Tree operates without expensive LLM prompts. It measures co-occurrence frequency and causal outcome gains to compute reusability scores across canonicalized action spans, automatically merging high-value spans into structured nodes.
2. Hierarchical eXperience Tree: Compiles action sequences into an explicit tree where each node formalizes how compound actions decompose into modular sub-skills.
3. Universal Tri-Paradigm Integration:

• Offline RL: Formulates each X-Tree node as a distinct structured training sample.
• Online RLVR: Infuses an adaptive skill bonus to reward structured exploration.
• On-Policy Self-Distillation: Equips privileged teacher models with X-Tree context to accelerate student convergence.

1. Parameter-Level Native Alignment: Bakes hierarchical routines directly into model weights, requiring zero external retrieval components at runtime.

权威 Benchmark 与实测跑分对比

Evaluated across three model scales on WebArena, ScienceWorld, and WebShop under matched data and compute budgets:

1. Consistent Success Rate Gains: X-Tree boosts task success rate (SR) by 4.5% on WebArena, 5.8% on ScienceWorld, and 4.1% on WebShop.
2. 35% Faster Exploration in RLVR: Integrating adaptive skill bonuses accelerates RL convergence, reaching target milestones in 35% fewer environment rollout interactions.
3. Robust Out-of-Distribution Transfer: The structured skill vocabulary transfers reliably to novel unseen tasks without prompt brittleness.

开发者实战落地与开箱指南

The X-Tree tokenizer and training recipes are publicly accessible. AI teams developing web agents, GUI automation copilots, or robotics foundations can parse raw trajectory logs through X-Tree's tokenizer to extract hierarchical skill trees, seamlessly augmenting existing post-training pipelines with superior sample efficiency.