Shanghai AI Laboratory has open-sourced AV-GRPO (arXiv: 2609.29816, GitHub: zhiyuxu03/AV-GRPO), a modality-anchored online diffusion reinforcement learning framework for joint audio-video generation. Addressing the persistent audio-visual desynchronization, semantic mismatch, and high compute overhead in coupled diffusion models, AV-GRPO decouples multimodal preference optimization into manageable unimodal subproblems via modality-anchored rollouts and trajectory-locked frozen-tower RL. It delivers state-of-the-art cross-modal synchronization on JavisBench and VABench, substantially outperforming LTX-2.3.

Key Takeaways

  • ✓Extends Group Relative Policy Optimization (GRPO) to continuous joint audio-visual diffusion reinforcement learning.
  • ✓Implements modality-anchored rollouts and trajectory-locked frozen towers to resolve multimodal credit assignment bottlenecks.
  • ✓Releases 5DAV, the first systematically decoupled 5-dimensional difficulty-controllable training dataset.
  • ✓Outperforms LTX-2.3 on JavisBench and VABench, boosting cross-modal synchronization by 28.4% and semantic alignment by 19.6%.
  • ✓Training scripts, inference recipes, LoRA checkpoints, and datasets published under open-source licenses on GitHub.
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🐙
AV-GRPO GitHub 官方开源仓库
github.com/zhiyuxu03/AV-GRPO
↗
🌐
ArXiv 论文预印本 (2609.29816)
arxiv.org/abs/2609.29816
↗
📝
Hugging Face Daily Papers 主页
huggingface.co/papers/2609.29816
↗
🐙
GitHub: zhiyuxu03/AV-GRPO%60%EF%BC%9B
github.com/zhiyuxu03/AV-GRPO`；
↗
🐙
GitHub: zhiyuxu03/AV-GRPO%60
github.com/zhiyuxu03/AV-GRPO`
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points Joint audio-video diffusion models represent the next frontier in multimodal generation, yet existing pipelines frequently suffer from jarring audio-visual desynchronization—such as delayed engine roar during high-speed vehicle maneuvers or misaligned lip movements. Directly porting diffusion reinforcement learning (RL) to joint audio-video towers introduces severe credit assignment ambiguities: heterogeneous acoustic and visual reward signals entangle, making it impossible to determine whether policy gradients should update the audio head or the visual backbone. ### Architecture Highlights & Internals Shanghai AI Laboratory introduces AV-GRPO, an online diffusion RL framework grounded in modality-anchored rollouts. By anchoring one modality trajectory while exploring candidate variations in the peer modality, the framework decouples joint preference optimization into distinct unimodal subproblems. Trajectory-locked frozen-tower optimization slashes training compute, while the accompanying 5DAV dataset introduces five difficulty-controlled dimensions (motion velocity, reverberation, occlusions). Dynamically tuned noise perturbations accommodate the divergent temporal dynamics of audio waveforms and visual latents. ### Authoritative Benchmarks & Measured Scores Evaluated on JavisBench and VABench benchmarks: AV-GRPO outperforms LTX-2.3 across semantic alignment and physical plausibility, driving a 28.4% gain in cross-modal synchronization and a 41.2% reduction in temporal offset errors. Decoupled optimization cuts peak VRAM consumption during RL backpropagation by 52%, enabling single-GPU LoRA fine-tuning. ### Developer Hands-on Guide Researchers can clone https://github.com/zhiyuxu03/AV-GRPO to access training recipes, download pre-trained LoRA weights from Hugging Face, and inspect the 5DAV evaluation dataset.