Research map / Vision-language and multimodal / Human and embodied modeling
Generating Multimodal Driving Scenes via Next-Scene Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMGen, an autoregressive framework that jointly generates ego-action, traffic map, agents, and camera image per scene, with TAR/OAR decomposition and an Action-aware Map Alignment module for controllable driving scene sequences.
Problem
Existing driving scene generators cover only a limited set of modalities (e.g., no map evolution or no agent forecasting), limiting controllable scene generation for evaluating autonomous driving systems; naive AR over all tokens of multimodal long sequences is computationally prohibitive.
Approach
Generate scenes frame-by-frame with a two-stage decomposition: Temporal AutoRegressive (causal attention over same-position tokens across past frames) for inter-frame dynamics, and Ordered AutoRegressive (GPT-like decoding in fixed modality order) for intra-frame alignment, plus an affine Action-aware Map Alignment that warps map features by the predicted ego motion.
Key results
- On WOMD initial-scene generation, UMGen reaches MMD position 0.0730 and heading 0.0550, lower than UniGen (0.1208 / 0.1104) and TrafficGen (0.1451 / 0.1325).
- On nuPlan, UMGen improves over reproduced TrafficGen on all four MMD attributes (e.g., position 0.0828 vs 0.0926).
- Removing the OAR module (UMGen-T) raises agent collision rate from 5.1% to 5.6% and worsens MMD across attributes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UMGen instead of UniGen | Initial scene generation · WOMD | Position MMD: 0.1208 -> 0.0730 | Helped |
| Use UMGen instead of UniGen | Initial scene generation · WOMD | Heading MMD: 0.1104 -> 0.0550 | Helped |
| Use UMGen instead of UniGen | Initial scene generation · WOMD | Size MMD: 0.0815 -> 0.0533 | Helped |
| Use UMGen instead of UniGen | Initial scene generation · WOMD | Velocity MMD: 0.1591 -> 0.1502 | Helped |
| Use UMGen instead of TrafficGen* | Initial scene generation · nuPlan | Position MMD: 0.0926 -> 0.0828 | Helped |
| Use UMGen instead of TrafficGen* | Initial scene generation · nuPlan | Heading MMD: 0.0799 -> 0.0682 | Helped |
| Use UMGen instead of TrafficGen* | Initial scene generation · nuPlan | Size MMD: 0.0856 -> 0.0674 | Helped |
| Use UMGen instead of TrafficGen* | Initial scene generation · nuPlan | Velocity MMD: 0.0988 -> 0.0760 | Helped |
| Add the OAR module (ablation) | UMGen-T · nuPlan validation generated scenes | Collision rate (%): 5.6 -> 5.1 | Helped |
| Add the OAR module (ablation) | UMGen-T · nuPlan validation generated scenes | Position MMD: 0.040 -> 0.038 | Helped |
| Add the AMA module (ablation) | UMGen · Map prediction under ego-vehicle motion | Map-ego coherence | Helped |
| Use TAR rather than vanilla autoregressive attention (ablation) | UMGen · Long-horizon scene generation | Inference efficiency | Helped |
Limitations
- Image generation quality is limited; the paper notes diffusion conditioning as a remedy (results in appendix).
- Evaluation of sequence realism is mainly qualitative visualization; quantitative sequence metrics are limited in the main text.
Experimental setup
Datasets: nuPlan Waymo Open Motion Dataset
Benchmarks: Waymo Open Motion Dataset nuPlan
Metrics: MMD (position, heading, size, velocity) average agent collision rate peak GPU memory per-token inference time
Baselines: TrafficGen LCTGen SceneGen UniGen GAIA-1 (vanilla AR) UMGen-T
Builds on: GAIA-1 VQGAN GPT-2 TrafficGen
Closest papers by meaning
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.89)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.89)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.89)
- Panacea: Panoramic and Controllable Video Generation for Autonomous Driving (CVPR 2024 · similarity 0.88)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.88)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.