Research map / 3D vision / 3D generation
UniScene: Unified Occupancy-centric Driving Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniScene, a unified occupancy-centric hierarchical framework that generates semantic occupancy from BEV layouts, then video (via Gaussian-based joint rendering) and LiDAR (via prior-guided sparse modeling) conditioned on that occupancy.
Problem
Existing driving scene generators produce only a single data form (usually RGB video) directly from coarse layouts, which limits downstream applicability and struggles to model the direct layout-to-data distribution.
Approach
Use semantic occupancy as an intermediate meta representation: generate occupancy sequences from BEV layouts with an occupancy DiT, then condition video generation on Gaussian-splatted semantic and depth maps and LiDAR generation on occupancy-guided sparse ray sampling.
Key results
- Video generation reaches 71.94 FVD with generated occupancy (70.52 with GT occupancy), vs. 139.00 for Panacea and 87.52 ablated variant.
- LiDAR generation achieves MMD 2.40e-4 and JSD 0.108 with 0.47 s generation time, surpassing LiDARDM's MMD 3.51e-4 by 31.62%.
- Augmenting NuScenes with UniScene data improves CONet camera occupancy mIoU from 12.8 to 16.5 and BEV segmentation road mIoU from 74.30 to 81.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rendered semantic and depth maps (Gaussian-based joint rendering) as occupancy conditions in place of spatial-temporal attention (ablation) | video generation (SVD-based) · NuScenes video generation | FVD: FVD 110.87 -> 70.52 (about 34.66% and 31.15% gains from semantic/depth maps) | Helped |
| Add geometry-aware (depth-based warp) noise prior (ablation) | video generation · NuScenes video generation | FVD: FVD 87.52 -> 70.52 | Helped |
| Prior-guided sparse sampling instead of uniform sampling (ablation) | LiDAR generation · NuScenes LiDAR generation | JSD: JSD 0.075 -> 0.072, memory 16.66 -> 6.84 GB, time 0.30 -> 0.25 s | Helped |
| Add ray-dropping head (ablation) | LiDAR generation · NuScenes LiDAR generation | MMD and JSD: JSD 0.100 -> 0.072; MMD 3.25 -> 1.53 | Helped |
| Replace Sparse UNet with single submanifold convolution layer (ablation) | LiDAR generation · NuScenes LiDAR generation | JSD: JSD 0.097 -> 0.072 (-25.77%) | Helped |
| Add temporal 3D axial attention to occupancy VAE decoder (ablation) | occupancy generation · NuScenes occupancy generation | MMD: MMD 17.10 -> 10.60 (38.01% reduction) | Helped |
| Add temporal attention layer to Occupancy DiT (ablation) | occupancy generation · NuScenes occupancy generation | F3D: F3D 176.74 -> 158.55 (10.29%) | Helped |
| Add spatial attention layer to Occupancy DiT (ablation) | occupancy generation · NuScenes occupancy generation | F3D: F3D 261.03 -> 158.55 (39.26%) | Helped |
| Use continuous occupancy VAE instead of VQVAE tokenizer | occupancy reconstruction · NuScenes-Occupancy key-frame val | mIoU: 92.1 at ratio 32 and 72.9 at ratio 512 vs OccWorld VQVAE 65.7 | Helped |
| Occupancy-conditioned multi-view video generation (UniScene) vs prior methods | video generation · NuScenes video generation | FVD: FVD 71.94 vs Vista* 112.65 and Drive-WM 122.70 | Helped |
| Prior-guided sparse LiDAR generation vs LiDARDM | LiDAR generation · NuScenes LiDAR generation | MMD: MMD 2.40 vs 3.51 (10^-4); time 0.47 s vs 45.12 s | Helped |
| Add UniScene-generated data to training for occupancy prediction | CONet occupancy prediction · NuScenes occupancy prediction, camera/LiDAR/C&L | mIoU: camera +3.7 (12.8 -> 16.5), LiDAR +3.5 (15.8 -> 19.3), C&L +3.8 (20.1 -> 23.9) | Helped |
Surprising findings
- Occupancy VAE at compression ratio 512 reconstructs with 72.9 mIoU, still outperforming VQVAE-based OccWorld at ratio 16 (65.7 mIoU) in the reported comparison.
- Spatial-temporal attention alone for multi-view consistency performs worse than occupancy-based rendering maps (FVD 110.87 vs 70.52).
Limitations
- Evaluated only on NuScenes, a single dataset.
- Video generation relies on text prompts and rendered maps; multi-view consistency depends on occupancy quality.
Experimental setup
Datasets: NuScenes NuScenes-Occupancy
Benchmarks: NuScenes-Occupancy NuScenes
Metrics: mIoU IoU F3D MMD FID FVD JSD mAP NDS BEV segmentation IoU
Baselines: BEVGen BEVControl DriveGAN DriveDreamer Vista WoVoGen Panacea MagicDrive Drive-WM OccLLama OccWorld OccSora
Builds on: Stable Video Diffusion ControlNet OccWorld LiDARDM CONet
Closest papers by meaning
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.91)
- ODG: Occupancy Prediction Using Dual Gaussians (NeurIPS 2025 · similarity 0.89)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.89)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.88)
- OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving (AAAI 2025 · similarity 0.88)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.