Research map / Video / Video generation
MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskGWM, a DiT-based (SD3-initialized) driving world model trained with an added mask-reconstruction objective using diffusion-related mask tokens and row-wise temporal masking, with single-view and multi-view variants.
Problem
Driving world models built on diffusion video prediction struggle with long-horizon prediction and zero-shot generalization across datasets and viewpoints.
Approach
Combine diffusion generation loss with MAE-style mask reconstruction in a DiT: diffusion-related mask tokens (noise-embedding plus learnable, weighted by timestep) reconcile the v-prediction objective with feature reconstruction, and a row-wise shifted temporal mask avoids 3D attention masks while enabling temporal context learning.
Key results
- Single-view MaskGWM-long reaches FID 5.6 and FVD 92.5 on nuScenes, versus VISTA's 6.9 FID and 89.4 FVD (FVD slightly worse).
- Multi-view MaskGWM-mview achieves FID 8.9 and FVD 65.4 on nuScenes, beating Drive-WM (15.8 / 122.7) without future layouts.
- On zero-shot Waymo (600 videos), MaskGWM-long gets FVD 118.83 vs VISTA's 176.56, with FID 9.55 vs 9.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video mask reconstruction to diffusion training | MaskGWM-long · nuScenes single-view | FID: 6.9 -> 5.6 | Helped |
| add video mask reconstruction to diffusion training | MaskGWM-long · nuScenes single-view | FVD: 89.4 -> 92.5 | Hurt |
| add multi-view generation with cross-view row-wise attention | MaskGWM-mview · nuScenes multi-view | FID: 11.2 -> 8.9 | Helped |
| add multi-view generation with cross-view row-wise attention | MaskGWM-mview · nuScenes multi-view | FVD: 55.7 -> 65.4 | Hurt |
| transfer the model zero-shot to Waymo | MaskGWM-long · Waymo validation | FVD: 176.56 -> 118.83 | Helped |
| transfer the model zero-shot to Waymo | MaskGWM-long · Waymo validation | FID: 9.76 -> 9.55 | Helped |
| combine noise embedding fm(ϵ) with learnable parameters p in the mask token (ablation) | MaskGWM · OpenDV-2K validation | FVD: 126.71 -> 105.52 | Helped |
| combine noise embedding fm(ϵ) with learnable parameters p in the mask token (ablation) | MaskGWM · OpenDV-2K validation | FID: 7.35 -> 5.69 | Helped |
| add view mask reconstruction at r=0.25 (ablation) | MaskGWM-mview · nuScenes multi-view | FID: 9.2 -> 8.9 | Helped |
| add view mask reconstruction at r=0.25 (ablation) | MaskGWM-mview · nuScenes multi-view | FVD: 65.9 -> 65.4 | Helped |
Surprising findings
- Masking on the view dimension was not detrimental; random row-wise masking across views even improved results.
- Unshifted temporal masking (requiring 3D attention masks) hurt FVD notably versus row-wise shifted masking, despite the same mask ratio.
Limitations
- Single-view model FVD on nuScenes (92.5) is slightly worse than VISTA (89.4).
- Long-horizon evaluation uses only 300 sampled videos due to computational cost.
Experimental setup
Datasets: OpenDV-2K nuScenes Waymo
Benchmarks: nuScenes OpenDV-2K validation Waymo validation
Metrics: FID FVD
Baselines: VISTA GenAD Drive-WM DriveDreamer-2 DiVE Panacea MagicDrive DrivingDiffusion DriveGAN
Builds on: Stable Diffusion 3 VISTA MaskDiT SD-DiT Rectified Flow
Closest papers by meaning
- Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion (ICLR 2024 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.87)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.87)
- VDT: General-purpose Video Diffusion Transformers via Mask Modeling (ICLR 2024 · similarity 0.87)
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024 · similarity 0.87)
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.