SpatialOPSD: Distilling Spatial Intelligence from Verified Coding Agent Traces into Tool-Free Multimodal LLMs

Background and the Problem

Endowing Multimodal Large Language Models (MLLMs) with 3D spatial intelligence is a prerequisite for physical interaction in embodied robotics. Conventional paradigms rely on spatial coding agents—prompting MLLMs to generate and run external Python geometry scripts (e.g., point cloud processing, monocular depth estimation, and 3D bounding projections) during inference. However, this tool-reliant architecture incurs massive execution latency, complex sandbox runtime dependencies, and high failure rates in dynamic real-world environments.

Architecture and How It Works

To resolve this operational overhead, researchers introduce SpatialOPSD (arXiv:2610.11366, repo: github.com/vermouth599/Spatial-OPSD), an on-policy self-distillation framework that internalizes agentic spatial tools directly into standalone MLLMs:

1. On-Policy Self-Distillation via Privileged Traces: Leverages summarized execution traces from validated spatial coding agents as privileged supervision during training, guiding a tool-free student model to mimic the geometric representations of tool-augmented teacher views.
2. Repetition-Aware Distillation: Preventing privileged information leakage from causing degenerated repetitive text generation. Pairs repetition masking with unlikelihood regularization to penalize repetitive loops and force the student model to internalize compressed causal spatial reasoning.
3. Pure Standalone Tool-Free Inference: Completely eliminates runtime Python environments and code execution sandboxes at test time, producing direct 3D spatial predictions in a single forward pass.

Benchmarks and Measured Results

Evaluated on demanding 3D spatial reasoning, object localization, and robotic affordance benchmarks:

1. Superior Standalone Performance: Substantially outperforms standard tool-free vision-language models, demonstrating internal representations of depth and 3D geometry.
2. Over 75% Latency Reduction: Bypassing the code-generation, sandboxing, and execution feedback loop slashes end-to-end response times by over 75%, matching the real-time requirements of robotics.
3. 86.4% Drop in Repetitive Hallucination: The repetition-aware distillation mechanism curtails degenerative token repetition by 86.4% compared to naive distillation baselines.

Getting Started for Developers

The SpatialOPSD training pipeline and evaluation configurations are available on GitHub (github.com/vermouth599/Spatial-OPSD). Robotics and edge AI engineers can utilize heavy coding agents offline to produce validated spatial traces, distilling their capabilities into compact multimodal models for high-speed, zero-dependency onboard deployment.