HKU & Tsinghua Unveil MIST for Video World Models: Over 70% of Supposed Memory Failures Stem from Execution Errors

Background and the Problem

In embodied AI and spatial simulation, video generation models are increasingly deployed as predictive world models that must maintain temporal consistency over physical transformations (e.g., an object being moved or a drawer opened). However, existing video memory benchmarks collapse sequence evaluation into a single terminal frame score. When a model scores poorly, researchers assume the model suffered from catastrophic forgetting. Researchers from HKU-MMLab and Tsinghua University published arXiv:2610.23542, revealing a profound methodological flaw: if the model failed to execute the action in the intermediate frames to begin with, the resulting state change never happened—conflating execution failure with memory loss.

Architecture and How It Works

1. Decoupled Execution-Memory Evaluation (MIST):

• MIST introduces a three-stage causal evaluation pipeline: ① Scene establishment verification; ② Intermediate action execution gating using multimodal verifiers; ③ Memory retention testing strictly conditioned on confirmed action execution;

1. M3 Memory Intervention Pipeline:

• Introduces explicit transition-state conditioning to boost execution fidelity and isolate pure temporal persistence from dynamic failure modes.

Benchmarks and Measured Results

Across comprehensive diagnostic suites (Tables 17–19):

• Failure Root-Cause Breakdown:
• Up to 70% of apparent 'memory failures' across open and closed video generation models are actually failures of intermediate action execution rather than cognitive forgetting;
• True Memory Scores After Correction (Table 19):
• On the WRBench/40 official suite, decoupling execution via M3 boosts the official $T$ metric from 0.452 to 0.570 (+0.118) and visual fidelity from 0.648 to 0.755 (+0.107);
• The internal reveal metric surges from 0.372 to 0.668 (+0.296, a +79.6% relative gain);
• On MBench-T/30, the official $U$ metric rises from 0.292 to 0.446 (+0.154).

Getting Started for Developers

• Open-Source Suite: Project page (https://hku-mmlab.github.io/MIST/) and repository (https://github.com/HKU-MMLab/MIST) are live.
• Engineering Takeaways: Teams optimizing video world models for robotics should not default to expanding memory context windows; fixing first-stage prompt execution accuracy resolves the majority of perceived environmental drift.