Researchers have introduced World Action Agent (WAA, arXiv: 2609.29964), a multi-agent harness enabling general-purpose Vision-Language Models (VLMs) to directly manipulate robotic arms within an interactive visual workspace. Overcoming the fragility of open-loop code-as-policy approaches, WAA equips VLMs with contact-centric dynamic geometry, an Imagination Agent for iterative action rehearsal, and in-view closed-loop trajectory correction. On the challenging LIBERO-Pro manipulation suite, WAA achieves a state-of-the-art 75.6% task success rate, outperforming end-to-end VLAs and visual harness baselines.

Key Takeaways

  • ✓Bypasses brittle code-as-policy generation, enabling VLMs to pilot physical robots directly through a visual-action harness.
  • ✓Implements Action Rehearsal with an Imagination Agent, iteratively previewing and refining trajectory proposals prior to physical actuation.
  • ✓Introduces in-view closed-loop correction, autonomously eliminating residual physical offsets directly within the visual scene view.
  • ✓Achieves a record 75.6% task success rate on LIBERO-Pro; lifts Qwen3.5-9B out-of-domain performance from 1.7% to 43.3%.
  • ✓Full preprint, multi-agent harness architecture, and trajectory collection released openly on arXiv and Hugging Face.
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🌐
ArXiv 论文预印本 (2609.29964)
arxiv.org/abs/2609.29964
↗
📝
Hugging Face Daily Papers 讨论主页
huggingface.co/papers/2609.29964
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points Vision-Language Models possess broad commonsense and spatial reasoning, yet bridging them to real-world robotic manipulators remains plagued by fragility. Systems either delegate VLMs to write static Python scripts (code-as-policy)—failing upon slight contact slips or object shifts—or rely on monolithic end-to-end Vision-Language-Action (VLA) architectures requiring millions of human demonstrations that struggle outside the training distribution. ### Architecture Highlights & Internals World Action Agent (WAA) introduces an interactive multi-agent harness: (1) Contact Views automatically crop and isolate the local geometric neighborhood around active end-effector interactions; (2) Action Rehearsal engages an Imagination Agent to preview, simulate, and edit motion proposals before physical execution; and (3) In-View Correction continually closes the perception-action loop, canceling metric drifts directly within the camera frame. ### Authoritative Benchmarks & Measured Scores On the demanding LIBERO-Pro robotic manipulation benchmark: WAA establishes a state-of-the-art 75.6% success rate, significantly surpassing end-to-end VLAs and program-synthesis baselines without requiring environment-specific retuning. Training traces generated by WAA boost lightweight Qwen3.5-9B zero-shot transfer success from 1.7% to 43.3% across out-of-distribution manipulation suites. ### Developer Hands-on Guide Robotics developers can review the multi-agent harness specifications, observation view selection algorithms, and prompt templates on arXiv and Hugging Face Papers.