Research map / Video / Video segmentation and tracking
SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyncVP, a joint dual-branch latent diffusion model that synchronously predicts future RGB and depth (or other modality) video frames using shared-noise training, a split spatio-temporal cross-attention module, and cross-modality guidance training.
Problem
Video prediction from RGB alone lacks information about real-world dynamics; naive multi-modal concatenation performs poorly, and existing methods do not jointly generate multiple modalities for video prediction.
Approach
Couple two pre-trained modality-specific PVDM-style latent diffusion UNets with a lightweight split spatio-temporal cross-attention that uses one shared attention map, apply the same noise to each modality's forward process, and randomly mask modalities during training so the model can predict both modalities from one or both inputs.
Key results
- On Cityscapes (2→28), SyncVP reaches FVD 84 versus 107.31 for the prior best STDiff, an improvement of over 21%.
- On BAIR (2→28), SyncVP reaches FVD 63.60 versus 88.1 for STDiff, and RGB-only SyncVP reaches 70.49.
- Shared noise vs independent noise on Cityscapes: RGB FVD 143.16 vs 84 with shared noise and cross-modality guidance; cross-modality guidance alone gives FVD 122.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth as a jointly predicted and conditioning modality | SyncVP · Cityscapes, 2 →28 | FVD: 97.31 -> 84 | Helped |
| add depth as a jointly predicted and conditioning modality | SyncVP · BAIR, 2 →28 | FVD: 70.49 -> 63.60 | Helped |
| concatenate RGB and depth frames along the channel dimension (ablation) | multi-modal video prediction on Cityscapes · Cityscapes, 2 →28 | FVD: 84 -> 123.8 | Hurt |
| fuse separate modality latents before a single diffusion model (ablation) | multi-modal video prediction on Cityscapes · Cityscapes, 2 →28 | FVD: 84 -> 160 | Hurt |
| use vanilla cross-attention instead of split spatio-temporal cross-attention (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 84 -> 97.68 | Hurt |
| share the forward diffusion noise across modalities (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 143.16 -> 84 | Helped |
| add cross-modality guidance during training (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 122.4 -> 84 | Helped |
| pre-train the single-modality models before joint fine-tuning (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 158.53 -> 84 | Helped |
| apply STCA only at the latest layer rather than all layers (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 773.7 -> 84 | Helped |
| add motion loss between temporal latent vectors (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 84 -> 106.97 | Hurt |
| use shared rather than separate STCAs (ablation) | SyncVP · Cityscapes, 2 →28 | FVD: 131.66 vs. 129.92 | No clear effect |
Surprising findings
- Naive channel concatenation of RGB and depth (FVD 123.8) and fused-latent concatenation (FVD 160) underperform the single-modality RGB baseline in some metrics, while the STCA variant performs best.
Limitations
- Depth for BAIR and OpenDV is pseudo ground truth from DepthAnything-v2 rather than measured depth.
- SSIM and LPIPS are slightly lower than some prior methods, which the authors attribute to heavy autoencoder compression.
- Results on SYNTHIA and ERA5-Land are compared only against single-modality baselines, since no prior video prediction work uses these datasets.
Experimental setup
Datasets: Cityscapes BAIR OpenDV-YouTube SYNTHIA ERA5-Land
Benchmarks: Cityscapes BAIR SYNTHIA ERA5-Land OpenDV-YouTube
Metrics: FVD SSIM LPIPS L2 L1 inference time
Baselines: SVG-LP NPVP VRNN Hier-VRNN GHVAEs VDT MCVD ExtDM STDiff STM-FANet VPTR-NAR SAVP
Builds on: PVDM DDPM DDIM classifier-free guidance LDM3D
Closest papers by meaning
- Video-Mirai: Autoregressive Video Diffusion Models Need Foresight (NeurIPS 2026 · similarity 0.86)
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.85)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.85)
- Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach (NeurIPS 2025 · similarity 0.84)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.84)
- Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.