Research map / 3D vision / 3D generation
Towards Realistic Scene Generation with LiDAR Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiDAR Diffusion Models (LiDMs), which run latent diffusion over a range-image autoencoder with curve-wise compression, point-wise coordinate supervision, and patch-wise encoding to generate realistic, conditional LiDAR scenes much faster than point-based diffusion.
Problem
Diffusion models operating on raw LiDAR points struggle to preserve curve-like patterns and 3D geometry, produce noisy scenes, and are computationally slow, and prior LiDAR generators lack conditional generation.
Approach
Latent diffusion on range images, with an autoencoder tailored to LiDAR: horizontal-only curve-wise downsampling and 1x4 kernels to keep scan-line patterns, a 3D coordinate reconstruction loss to keep scene geometry, and patch-wise downsampling for full object context, enabling multimodal conditioning (semantic maps, camera views, text).
Key results
- On 64-beam KITTI-360 unconditional generation with 50 steps, LiDM improves FRID to 158.8 vs 199.5 for Latent Diffusion (about 20% relative), and is competitive with LiDARGen at 1160 steps.
- On Semantic-Map-to-LiDAR (SemanticKITTI), LiDM reaches FRID 22.9 vs 42.5 for LiDARGen, and on Camera-to-LiDAR it outperforms LiDARGen by over 36% across metrics.
- LiDM achieves about 107x higher throughput (1.603 vs 0.015 samples/sec) than point-based LiDARGen on one RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LiDM instead of LDM with the same 50 diffusion steps | Unconditional LiDAR generation · KITTI-360 64-beam | FRID: 199.5 -> 158.8; 20.4% | Helped |
| Use LiDM instead of LDM with the same 50 diffusion steps | Unconditional LiDAR generation · KITTI-360 64-beam | FSVD: 70.7 -> 53.7; 24.0% | Helped |
| Use LiDM instead of LDM with the same 50 diffusion steps | Unconditional LiDAR generation · KITTI-360 64-beam | FPVD: 61.9 -> 42.7; 31.0% | Helped |
| Use LiDM instead of LDM with the same 50 diffusion steps | Unconditional LiDAR generation · KITTI-360 64-beam | JSD: 0.236 -> 0.213; 9.7% | Helped |
| Use LiDM instead of LDM with the same 50 diffusion steps | Unconditional LiDAR generation · KITTI-360 64-beam | MMD: 5.06 -> 4.46; 11.9% | Helped |
| Use LiDM instead of Latent Diffusion | Semantic-Map-to-LiDAR generation · SemanticKITTI | FRID: 24.0 -> 22.9 | Helped |
| Use LiDM instead of Latent Diffusion | Semantic-Map-to-LiDAR generation · SemanticKITTI | FSVD: 21.3 -> 20.2 | Helped |
| Use LiDM instead of Latent Diffusion | Semantic-Map-to-LiDAR generation · SemanticKITTI | FPVD: 20.3 -> 17.7 | Helped |
| Use LiDM instead of Latent Diffusion | Semantic-Map-to-LiDAR generation · SemanticKITTI | JSD: 0.088 -> 0.072 | Helped |
| Use LiDM instead of Latent Diffusion | Semantic-Map-to-LiDAR generation · SemanticKITTI | MMD: 3.73 -> 3.16 | Helped |
| Use LiDM instead of Latent Diffusion | Camera-to-LiDAR generation · KITTI-360 | FRID: 50.2 -> 44.9 | Helped |
| Use LiDM instead of Latent Diffusion | Camera-to-LiDAR generation · KITTI-360 | FSVD: 35.9 -> 32.5 | Helped |
Surprising findings
- Curve-wise encoding generally outperforms patch-wise encoding in the scale-factor study, so patch-wise compression is only kept for efficiency and competitiveness.
Limitations
- Visual gap to real LiDAR data, attributed mainly to autoencoder reconstruction quality
- Autoencoders may produce blurry object-background boundaries that appear unreasonable in 3D
- Text-to-LiDAR struggles with complex prompts due to limited training data
Experimental setup
Datasets: KITTI-360 nuScenes SemanticKITTI
Benchmarks: KITTI-360 nuScenes SemanticKITTI
Metrics: FRID FSVD FPVD JSD MMD throughput inference speed
Baselines: LiDARGen LiDARGAN LiDARVAE Latent Diffusion
Builds on: Latent Diffusion Models LiDARGen PatchGAN VQ-VAE
Closest papers by meaning
- Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion (CVPR 2024 · similarity 0.91)
- Spiral: Semantic-Aware Progressive LiDAR Scene Generation and Understanding (NeurIPS 2025 · similarity 0.88)
- LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance (NeurIPS 2026 · similarity 0.87)
- Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata (CVPR 2024 · similarity 0.87)
- LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation (CVPR 2025 · similarity 0.87)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.