Research map / 3D vision / 3D generation
StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds StreetCrafter, a LiDAR-point-cloud-rendering-conditioned video diffusion model for controllable street view synthesis, and distills it into a dynamic 3DGS for real-time novel view rendering and scene editing.
Problem
Novel view synthesis for autonomous driving scenes degrades sharply when the viewpoint deviates from the recorded trajectory, and existing video diffusion models lack fine-grained camera control for street scenes.
Approach
Condition a video diffusion model (initialized from Vista) on pixel-level LiDAR point cloud renderings aggregated over adjacent frames instead of camera pose embeddings, then distill its generated novel-trajectory views into dynamic 3D Gaussian Splatting as extra supervision.
Key results
- On Waymo lane shift at 3m, Ours-V achieves FID 73.49 versus 93.38 for Street Gaussians and 122.52 for 3DGS.
- Ours-G renders at 58.17 FPS-class speed comparable to Street Gaussians (71.42 FID@2m vs 58.17 for Ours-G) while improving lane-shift FID on Waymo.
- StreetCrafter's LiDAR condition reduces random-set FID to 55.53 versus 92.75 for Plücker embedding conditioning in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LiDAR conditions with Plücker camera-ray embeddings (ablation) | StreetCrafter · Waymo random set | PSNR: 22.88 vs 27.00 | Hurt |
| replace LiDAR conditions with camera pose and projected 3D boxes (ablation) | StreetCrafter · Waymo random set | PSNR: 24.54 vs 27.00 | Hurt |
| use single-frame rather than aggregated LiDAR (ablation) | StreetCrafter · Waymo random set | PSNR: 25.57 vs 27.00 | Hurt |
| project aggregated LiDAR without assigning each point a fixed screen-space radius (ablation) | StreetCrafter · Waymo random set | PSNR: 26.15 vs 27.00 | Hurt |
| add novel-view LPIPS loss rather than using L1 loss for novel views (ablation) | StreetCrafter-to-3DGS distillation · Waymo distillation sequences | PSNR: 30.80 -> 30.88 | Helped |
| keep input and novel views independently weighted rather than setting λnovel to 1.0 (ablation) | StreetCrafter-to-3DGS distillation · Waymo distillation sequences | PSNR: 29.38 -> 30.88 | Helped |
| progressively reduce the noise scale during training rather than keep s at 1 (ablation) | StreetCrafter-to-3DGS distillation · Waymo distillation sequences | PSNR: 30.86 -> 30.88 | Helped |
| distill StreetCrafter into 3DGS rather than use the strongest listed baseline | Waymo street view synthesis · Waymo lane shift 2m | FID: 71.42 -> 58.17 | Helped |
| distill StreetCrafter into 3DGS rather than use the strongest listed baseline | PandaSet street view synthesis · PandaSet lane shift 3m | FID: 86.44 -> 78.88 | Helped |
Surprising findings
- Training only on single-lane driving sequences still yields high-quality synthesis across multiple lanes at test time when conditions are changed.
Limitations
- Requires costly LiDAR and object tracklet data for training.
- Video diffusion inference is far from real-time (about 0.2 fps at 576x1024).
Experimental setup
Datasets: Waymo Open Dataset PandaSet
Benchmarks: Waymo Open Dataset PandaSet
Metrics: PSNR LPIPS FID FPS
Baselines: 3DGS Street Gaussians EmerNeRF UniSim NeuRAD Plücker embedding conditioning pose + 3D box conditioning
Builds on: Vista Stable Video Diffusion 3D Gaussian Splatting Street Gaussians ControlNet
Closest papers by meaning
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.92)
- LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes (CVPR 2024 · similarity 0.90)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.88)
- Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata (CVPR 2024 · similarity 0.87)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.87)
- Towards Realistic Scene Generation with LiDAR Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.