Research map / Image generation / Autoregressive and latent generation
SimGen: Simulator-conditioned Driving Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimGen, a simulator-conditioned cascade diffusion model that converts simulator depth/segmentation into realistic conditions and generates diverse driving images, trained on the new DIVA dataset mixing YouTube and MetaDrive data.
Problem
Controllable driving scene image generation is limited by small, low-diversity datasets like nuScenes and by the sim-to-real gap when simulator layouts are used as conditions.
Approach
A two-stage cascade: a text-to-real-condition diffusion model (CondDiff) is used to convert noised simulator depth/segmentation into realistic conditions via SDEdit-style intermediate-time injection without sim-specific training, followed by an image diffusion model (ImgDiff) with a unified masked adapter that fuses multimodal conditions and resolves conflicts.
Key results
- SimGen improves D_pix by +6.5 over DrivingDiffusion and surpasses prior methods in FID and diversity on nuScenes.
- With SimGen-generated augmentation, BEVFusion single-frame vehicle map mIoU rises to 39.0 (+4.4 over the real-only baseline of 34.6).
- Cascade CondDiff conversion reduces FID by 2.3 in the ablation; unified adapter further improves FID by 0.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SimGen-generated images to augment training data | single-frame BEVFusion · nuScenes train + synthetic data | mIoU Vehicle: +4.4 | Helped |
| add the cascade pipeline to transform SimCond into RealCond (ablation) | SimGen · DIVA subset, 30K frames | FID: -2.3 | Helped |
| add simulator-pulled ExtraCond to the control conditions (ablation) | SimGen · DIVA subset, 30K frames | AP Car: +1.3 | Helped |
| add simulator-pulled ExtraCond to the control conditions (ablation) | SimGen · DIVA subset, 30K frames | FID: +0.5 | Hurt |
| add a Unified Adapter (ablation) | SimGen with ExtraCond · DIVA subset, 30K frames | FID: -0.8 | Helped |
| train SimGen on DIVA rather than nuScenes only | SimGen · novel appearances | appearance diversity | Helped |
| use SimGen rather than DrivingDiffusion | driving scene generation · nuScenes-specific comparison | D_pix: +6.5 | Helped |
Surprising findings
- Simulator depth and segmentation conditions are less suited than real conditions for controlling real image generation, so the sim-to-real conversion is necessary.
Limitations
- Single-view only; no multi-view generation for BEV models.
- Long inference time due to diffusion sampling with SD-2.1.
- Video extension is only preliminary.
Experimental setup
Datasets: DIVA-Real DIVA-Sim nuScenes Waymo Open Dataset
Benchmarks: nuScenes
Metrics: FID pixel variance diversity (D_pix) mIoU AP
Baselines: BEVGen MagicDrive DrivingDiffusion Panacea SimGen-nuSc ControlNet
Builds on: ControlNet Stable Diffusion 2.1 SDEdit UniControl MetaDrive
Closest papers by meaning
- Glad: A Streaming Scene Generator for Autonomous Driving (ICLR 2025 · similarity 0.89)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.88)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.87)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.87)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.86)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.