Research map / Image generation / Autoregressive and latent generation
Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Scenario Dreamer, a fully data-driven generative driving simulator that pairs a vectorized latent diffusion model for initial lane-graph and agent-box generation with a return-conditioned autoregressive Transformer (CtRL-Sim) for closed-loop agent behaviour.
Problem
Existing generative driving simulators rasterize scenes, wasting computation on empty pixels, and either use rule-based agent behaviours or only produce short non-reactive scenarios, limiting realism and length.
Approach
A vectorized latent diffusion model operates directly on lane and object vector elements (factorized lane/object attention autoencoder with a latent Transformer diffusion model, plus recursive lane ordering for permutation ambiguity), enabling inpainting-based scene extrapolation and combined with a return-conditioned autoregressive behaviour model that supports adversarial tilting.
Key results
- On nuPlan, Scenario Dreamer (B) with 377M params beats SLEDGE DiT-XL (769M) on all lane-graph metrics, with ~2x fewer parameters, ~7x lower latency, and 10x fewer GPU training hours.
- On Waymo, Scenario Dreamer (L) reduces the Urban Planning Connectivity metric to 0.03 vs 4.53 for privileged DriveSceneGen, and reduces agent collision rate to 4.8% vs 0.2% for the privileged baseline (note: lower collision rate is not better for DriveSceneGen here)
- PPO planners trained on Waymo scenes achieve lower success on 100m Scenario Dreamer routes (38.2%) than on Waymo test (63.8%) with 55m routes, and negative tilting lowers success further to 32.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use vectorized latent diffusion instead of rasterized scene generation | Scenario Dreamer · nuPlan lane graph generation | FD: 1.44 -> 0.67 | Helped |
| use vectorized latent diffusion instead of rasterized scene generation | Scenario Dreamer · Waymo lane graph generation | FD: 40.59 -> 1.38 | Helped |
| use vectorized latent diffusion instead of rasterized scene generation | Scenario Dreamer · nuPlan agent generation | Nearest Dist. JSD: 0.49 -> 0.09 | Helped |
| use vectorized latent diffusion instead of rasterized scene generation | Scenario Dreamer · Waymo agent generation | Nearest Dist. JSD: 0.63 -> 0.06 | Helped |
| remove lane ordering (ablation) | Scenario Dreamer · nuPlan lane graph generation | FD: 1.05 -> 1.36 | Hurt |
| replace learned lane connectivity with heuristic topology labels (ablation) | Scenario Dreamer · nuPlan lane graph generation | Reach: 0.07 -> 0.12 | Hurt |
| replace learned lane connectivity with heuristic topology labels (ablation) | Scenario Dreamer · nuPlan lane graph generation | Connectivity: 0.28 -> 0.28 | No clear effect |
| use non-factorized diffusion processing (ablation) | Scenario Dreamer · nuPlan lane graph generation | FD: 1.05 -> 1.21 | Hurt |
| use non-factorized diffusion processing (ablation) | Scenario Dreamer · nuPlan lane graph generation | Per-scene generation time (s): 0.09 -> 0.20 | Hurt |
| replace log replay with reactive CtRL-Sim behaviors | PPO planner · Waymo Test | Success rate (%): 63.8±0.8 -> 59.4±0.4 | Mixed |
| evaluate on Scenario Dreamer-generated 55m routes instead of Waymo test scenes | PPO planner · 55m routes | Success rate (%): 59.4±0.4 -> 59.8±1.7 | No clear effect |
| increase Scenario Dreamer route length from 55m to 100m | PPO planner · Scenario Dreamer environments | Success rate (%): 59.8±1.7 -> 38.2±1.9 | Hurt |
Surprising findings
- The privileged DriveSceneGen baseline had the lowest collision rate on Waymo (0.2%), while Scenario Dreamer had higher collision rates (4.8-5.4%), though the paper does not discuss this trade-off in depth.
- The authors found DriveSceneGen's diffusion model did not converge, so they used a privileged variant instead.
Limitations
- Traffic light signaling does not always provide valid traffic logic.
- Only generates centerline maps; other road elements such as road edges and crosswalks are not modeled.
- Some collision-rate results are higher than the privileged DriveSceneGen baseline.
Experimental setup
Datasets: Waymo Open Motion Dataset nuPlan
Benchmarks: Waymo Open Motion Dataset test nuPlan test
Metrics: Frechet distance Urban Planning metrics (connectivity, density, reach, conven route length endpoint distance JSD collision rate collision rate (RL) offroad rate success rate generation latency
Baselines: SLEDGE (DiT-L) SLEDGE (DiT-XL) DriveSceneGen IDM Trajeglish Log Replay
Builds on: SLEDGE CtRL-Sim DiT GPUDrive Trajeglish
Closest papers by meaning
- SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model (CVPR 2025 · similarity 0.89)
- SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout (NeurIPS 2024 · similarity 0.88)
- Generative Scenario Rollouts for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.87)
- Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner (AAAI 2025 · similarity 0.87)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.