Research map / Image generation / Fast diffusion sampling
SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneDiffuser, a scene-level diffusion prior for traffic simulation that unifies scene initialization and closed-loop rollout via amortized diffusion, generalized hard constraints, and LLM-prompted constraint generation.
Problem
Traffic simulation for autonomous vehicle testing needs realistic, controllable agents for both initial scene generation and closed-loop rollout, where diffusion models are costly and suffer compounding errors.
Approach
Amortized diffusion aligns diffusion denoising steps with physical timesteps via a monotonic noise schedule over a future buffer, so each simulation step needs only one denoising evaluation while reusing prior predictions; generalized hard constraints clip the scene tensor at every denoising step.
Key results
- Amortized AR at 10 Hz achieves a WOSAC composite score of 0.703 with 16x fewer model inference calls per step than Full AR at 10 Hz.
- Ours (One-Shot) reaches composite 0.736 and Ours (Amortized) 0.703, versus VBD 0.720 and Trajeglish 0.735 on the WOSAC test split; the amortized model is best among diffusion models in closed loop.
- LLM-generated constrained scene control succeeded in 40 of 100 sampled examples (qualitative success criteria).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use amortized autoregressive rollout instead of Full AR at 10 Hz | SceneDiffuser · WOSAC | simulation realism: 16x less model inference per step | Helped |
| increase the replan rate under Full AR | SceneDiffuser Full AR · WOSAC closed-loop rollout | simulation realism | Hurt |
| increase temporal resolution by decreasing temporal patch size (ablation) | SceneDiffuser · WOMD scene generation | realism | Helped |
| increase model parameter count (ablation) | SceneDiffuser · WOMD scene generation | realism | Helped |
| co-train on behavior prediction, SceneGen, and random control masks instead of behavior prediction only (ablation) | SceneDiffuser · simulation agent rollout | collision rate | Helped |
| jointly learn agent features including size and type instead of pose-only features (ablation) | SceneDiffuser · simulation agent rollout | performance | No clear effect |
| replace AdaLN-Zero conditioning with cross attention (ablation) | SceneDiffuser · WOMD scene generation | realism: 7.99% | Hurt |
| remove the agent-wise spatial attention layer (ablation) | SceneDiffuser · WOMD scene generation | collision rate | Hurt |
| apply hard constraints on collisions during scene generation | SceneDiffuser · WOMD scene generation | composite metric | Helped |
| apply hard constraints after each diffusion step rather than only post-diffusion (ablation) | SceneDiffuser · unconditional scene generation | trajectory realism | Helped |
Surprising findings
- LLM-prompted constrained scene generation succeeded in only 40/100 manually judged cases.
- Replacing AdaLN-Zero conditioning with cross attention reduced realism by 7.99%, mainly through higher collision and offroad rates.
Limitations
- Does not exceed SOTA of other autoregressive models on WOSAC.
- Validity masks are not modeled and logged validity is used as input.
- LLM-based constraint generation has a low manual success rate (40/100).
Experimental setup
Datasets: Waymo Open Motion Dataset v1.2.0
Benchmarks: Waymo Open Sim Agents Challenge (WOSAC) WOMD validation scene generation realism
Metrics: WOSAC composite metric minADE ADE collision rate offroad rate realism per-component likelihood inference calls per step
Baselines: SceneDMF VBD MTR+++ MVTE Trajeglish Const. Velocity Random Agent
Builds on: DPM++ TEDi Perceiver IO Adafactor dynamic thresholding
Closest papers by meaning
- SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model (CVPR 2025 · similarity 0.90)
- DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (CVPR 2025 · similarity 0.89)
- Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments (CVPR 2025 · similarity 0.88)
- DiffSF: Diffusion Models for Scene Flow Estimation (NeurIPS 2024 · similarity 0.88)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.88)
- Causal Composition Diffusion Model for Closed-loop Traffic Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.