Research map / Image generation / Autoregressive and latent generation
Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sat2Scene generates 3D urban street-view scenes from satellite-inferred geometry by running a 3D sparse diffusion model to colorize a point cloud, then feed-forward neural rendering to produce consistent arbitrary views.
Problem
Generating consistent, photorealistic urban street-view scenes and arbitrary-view renderings directly from satellite imagery, which faces large view changes and scene scale; prior 2D/GAN/video methods lack inter-frame consistency and per-scene optimization methods are inflexible.
Approach
Applies diffusion directly in a 3D sparse setting (sparse convolutions over occupied surface voxels) to denoise per-point colors on a fixed geometry, then extracts point-anchored features with a feed-forward encoder and renders via volume rendering with point-wise feature aggregation, avoiding test-time optimization and dense voxel memory costs.
Key results
- On HoliCity, Sat2Scene achieves FVD 20.30 vs 22.79 for MVDiffusion and 37.06 for Sat2Vid, and PSNR 31.54 vs 17.56 for MVDiffusion.
- FID is 71.98 for Sat2Scene, second to MVDiffusion's 50.78, so single-frame FID is not best.
- In a 32-person user study, 81.46% of participants preferred Sat2Scene videos over baselines (MVDiffusion 15.62%, Sat2Vid 2.92%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Sat2Scene instead of MVDiffusion | HoliCity video generation · HoliCity | FVD: 22.79 -> 20.30 | Helped |
| use Sat2Scene instead of MVDiffusion | HoliCity video generation · HoliCity | FID: 50.78 -> 71.98 | Hurt |
| use Sat2Scene instead of the strongest listed baseline | HoliCity video generation · HoliCity | PSNR: 17.56 -> 31.54 | Helped |
| use Sat2Scene instead of the strongest listed baseline | HoliCity video generation · HoliCity | SSIM: 0.593 -> 0.956 | Helped |
| use Sat2Scene instead of the strongest listed baseline | HoliCity video generation · HoliCity | LPIPS: 0.259 -> 0.237 | Helped |
| use Sat2Scene instead of MVDiffusion | HoliCity video generation · HoliCity | User study preference: 15.62% -> 81.46% | Helped |
| add point resampling to balance point-cloud density (ablation) | Sat2Scene · HoliCity | FID: 131.38 -> 71.98 | Helped |
| add point aggregation in rendering (ablation) | Sat2Scene · HoliCity | FID: 85.58 -> 71.98 | Helped |
| add depth supervision when training the neural renderer (ablation) | Sat2Scene · HoliCity | FID: 80.40 -> 71.98 | Helped |
Surprising findings
- Sat2Scene does not win on single-frame FID/KID, where MVDiffusion is better, despite winning on video metrics and user preference.
- Point resampling is critical: removing it raises FID from 71.98 to 131.38.
Limitations
- Single-frame image quality (FID/KID) trails MVDiffusion, and generated building textures are less diverse.
- Requires point clouds from geometry; dynamic objects like cars lack geometry and need confidence masking.
- Training is per-stage on a single GPU with voxel sizes adjusted to fit memory.
Experimental setup
Datasets: HoliCity OmniCity Cityscapes
Benchmarks: HoliCity OmniCity
Metrics: FVD KVD FID KID PSNR SSIM LPIPS depth RMSE user study preference
Baselines: MVDiffusion Sat2Vid InfiniCity
Builds on: DDPM DDIM Latent Diffusion Models NeRF Minkowski Engine
Closest papers by meaning
- CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation (NeurIPS 2025 · similarity 0.84)
- SemCity: Semantic Scene Generation with Triplane Diffusion (CVPR 2024 · similarity 0.83)
- GPS as a Control Signal for Image Generation (CVPR 2025 · similarity 0.82)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.82)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.82)
- DiffusionSat: A Generative Foundation Model for Satellite Imagery (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.