Research map / 3D vision / 3D generation
SemCity: Semantic Scene Generation with Triplane Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemCity, a triplane-based diffusion model that generates semantic 3D outdoor scenes on SemanticKITTI and CarlaSC, with triplane manipulation for inpainting, outpainting, and SSC refinement.
Problem
Generating semantic 3D scenes for real-world outdoor environments, where sensor limitations produce many empty voxels that voxel-based diffusion models handle poorly.
Approach
Factorize a voxelized semantic scene into a triplane latent via an autoencoder with an implicit MLP decoder, train a DDPM on triplanes, and manipulate triplanes during sampling (RePaint-style trimask and overlap conditioning) for inpainting, outpainting, and SSC refinement.
Key results
- On SemanticKITTI, SemCity reaches FID 56.55 versus 112.82 for SSD; on CarlaSC FID 40.63 versus 87.39.
- SSC refinement improves MonoScene IoU from 37.12 to 50.44 and SCPNet IoU from 50.24 to 59.25 on SemanticKITTI validation.
- Outpainting extends a 256x256x32 scene to 1792x3328x32 while maintaining road connectivity and object coherence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SemCity instead of SSD for scene generation | SemanticKITTI · SemanticKITTI | FID: 112.82 -> 56.55 | Helped |
| use SemCity instead of SSD for scene generation | CarlaSC · CarlaSC | FID: 87.39 -> 40.63 | Helped |
| refine MonoScene predictions with SemCity | MonoScene · SemanticKITTI validation | IoU: 37.12 -> 50.44 | Helped |
| refine OccDepth predictions with SemCity | OccDepth · SemanticKITTI validation | IoU: 41.60 -> 50.20 | Helped |
| refine SSA-SC predictions with SemCity | SSA-SC · SemanticKITTI validation | IoU: 58.25 -> 60.71 | Helped |
| refine SCPNet predictions with SemCity | SCPNet · SemanticKITTI validation | IoU: 50.24 -> 59.25 | Helped |
| remove positional embedding (ablation) | SemCity · SemanticKITTI | FID: 56.55 -> 59.14 | Hurt |
| remove positional embedding (ablation) | SemCity · SemanticKITTI | precision: 0.39 -> 0.30 | Hurt |
| replace the triplane representation with an xy-plane (ablation) | SemCity · SemanticKITTI | FID: 56.55 -> 75.56 | Hurt |
| replace the triplane representation with a 3D volume (ablation) | SemCity · SemanticKITTI | FID: 56.55 -> 136.50 | Hurt |
Surprising findings
- Voxel-based SSD performs well on synthetic CarlaSC but poorly on real SemanticKITTI, highlighting the difficulty of real-world empty space.
Limitations
- Struggles to model areas occluded from the sensor, such as building rears.
- Incomplete building heights due to driving-view data.
- Traces of moving objects remain from dataset frame merging.
Experimental setup
Datasets: SemanticKITTI CarlaSC
Benchmarks: SemanticKITTI CarlaSC
Metrics: FID KID IS precision recall IoU mIoU
Baselines: SSD MonoScene OccDepth SSA-SC SCPNet
Builds on: DDPM RePaint DDPS triplane representation SSD
Closest papers by meaning
- CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation (NeurIPS 2025 · similarity 0.90)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.88)
- SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation (CVPR 2025 · similarity 0.88)
- DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis (CVPR 2024 · similarity 0.88)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.88)
- DORSal: Diffusion for Object-centric Representations of Scenes $\textit{et al.}$ (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.