Research map / 3D vision / 3D generation
CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SketchSem3D, a sketch and pseudo-labeled satellite annotation benchmark for 3D outdoor semantic scene generation, and CymbaDiff, a latent diffusion model with cylinder Mamba blocks that improves spatial coherence.
Problem
Generating realistic 3D outdoor semantic urban scenes from user-friendly inputs is hard because no public large-scale benchmark exists and Cartesian scan orders distort spatial neighborhoods in voxel grids.
Approach
Combines a Cartesian Triple Mamba branch with a cylinder Mamba branch that scans voxels in (theta, r, z) order, fused into a latent diffusion denoiser, conditioned on freehand sketches and CLIP/SAM-derived pseudo-labeled satellite annotations.
Key results
- CymbaDiff reaches FID 40.67 and MMD 0.04 on Sketch-based SemanticKITTI, versus 56.55 FID for Semcity (2D FID) and 138.86 for 3D DiT.
- On SemanticKITTI validation, CymbaDiff attains 43.2 IoU and 14.6 mIoU using only sketch and PSA inputs, exceeding the best monocular baseline by 1.1 mIoU.
- Cross-dataset test trained on SemanticKITTI and evaluated on KITTI-360 yields FID 107.53 vs 272.83 for 3D DiT on the KITTI-360 subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CymbaDiff instead of 3D DIT | 3D semantic scene generation · Sketch-based SemanticKITTI | FID: 138.86 -> 40.67 | Helped |
| Use CymbaDiff instead of 3D DIT | 3D semantic scene generation · Sketch-based SemanticKITTI | MMD: 0.08 -> 0.04 | Helped |
| Use CymbaDiff instead of 3D DIT | 3D semantic scene generation · Sketch-based KITTI-360 | FID: 272.83 -> 107.53 | Helped |
| Use CymbaDiff instead of 3D DIT | 3D semantic scene generation · Sketch-based KITTI-360 | MMD: 0.11 -> 0.08 | Helped |
| Add CSCB (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | FID: 90.53 -> 40.67 | Helped |
| Add CSCB (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | MMD: 0.06 -> 0.04 | Helped |
| Add DDCB (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | FID: 76.57 -> 40.67 | Helped |
| Add DDCB (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | MMD: 0.06 -> 0.04 | Helped |
| Add C-Mamba alongside Triple Mamba layers (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | FID: 74.09 -> 40.67 | Helped |
| Add C-Mamba alongside Triple Mamba layers (ablation) | CymbaDiff · Sketch-based SemanticKITTI test | MMD: 0.05 -> 0.04 | Helped |
| Use sketch and pseudo-labeled satellite annotations instead of the best reported stereo method | Semantic scene completion · SemanticKITTI validation | mIoU: 13.7 -> 14.6 | Helped |
| Use sketch and pseudo-labeled satellite annotations instead of the leading monocular method | Semantic scene completion · SemanticKITTI validation | mIoU: 13.5 -> 14.6 | Helped |
Surprising findings
- Semcity and CityDreamer denoisers, retrained in the authors' framework, reach FID near 950-990, far worse than CymbaDiff, and full SemCity pipeline training diverged to NaN.
Limitations
- Sketches are Canny edge maps derived from BEV projections of ground truth, not authentic human drawings.
- Pseudo-labeled satellite annotations are automatic and noisy, causing confusion such as vegetation vs buildings.
- Small or occluded objects are sometimes not reconstructed well.
Experimental setup
Datasets: SemanticKITTI SSCBench-KITTI-360 KITTI KITTI-360 NuScenes
Benchmarks: SketchSem3D (Sketch-based SemanticKITTI) Sketch-based KITTI-360 SemanticKITTI validation set
Metrics: 3D FID MMD IoU mIoU
Baselines: SSD SemCity 3D Latent Diffusion 3D DiT MonoScene VoxFormer OccFormer CityDreamer UrbanDiff
Builds on: Mamba Latent Diffusion SCPNet UrbanDiff Segment Anything Model CLIP
Closest papers by meaning
- SemCity: Semantic Scene Generation with Triplane Diffusion (CVPR 2024 · similarity 0.90)
- Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata (CVPR 2024 · similarity 0.88)
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity (CVPR 2026 · similarity 0.86)
- CityDreamer: Compositional Generative Model of Unbounded 3D Cities (CVPR 2024 · similarity 0.86)
- USCILab3D: A Large-scale, Long-term, Semantically Annotated Outdoor Dataset (NeurIPS 2024 · similarity 0.86)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.