Research map / 3D vision / 3D generation
SceneCrafter: Controllable Multi-View Driving Scene Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneCrafter, a controllable multi-view diffusion editor for driving scenes that edits weather, time of day, and agents, trained on synthetic paired data generated by teacher models with an adapted Prompt-to-Prompt and masked-training/multi-view repaint.
Problem
Editing real multi-camera driving footage with cross-camera 3D consistency, learning empty-street priors despite populated training logs, and obtaining paired training data for global (weather, time of day) and local (agent insertion/removal) edits.
Approach
Adapts Prompt-to-Prompt by replacing image-to-image self-attention (not cross-attention) to keep geometry consistent across multi-view synthetic pairs, uses masked training with multi-view repaint to learn empty-street priors, and alpha-blends empty and populated scenes to create local-edit pairs; a student editor is then distilled from these teachers.
Key results
- On time-of-day editing, SceneCrafter reaches FID 37.2 vs 46.8 for P2P* and 60.4 for SDEdit, with 83.6% user preference.
- On local editing, SceneCrafter achieves FID 23.5 (removal) and 21.7 (insertion) vs 26.0/28.5 for MV-RePaint.
- On generation, full SceneCrafter achieves FID 36.2 and 3D LPIPS 0.187, close to real data's 0.186, versus CAT3D's 121.3 FID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace cross-attention weights with self-attention weights (ablation) | Prompt-to-Prompt global-edit data generation · Time of day editing | FID: 57.1 -> 41.5 | Helped |
| replace cross-attention weights with self-attention weights (ablation) | Prompt-to-Prompt global-edit data generation · Time of day editing | CLIP Score: 0.204 -> 0.202 | No clear effect |
| add agent features and HD maps (ablation) | Prompt-to-Prompt global-edit data generation · Time of day editing | CLIP Score: 0.202 -> 0.214 | Helped |
| use daytime-only source images (ablation) | Prompt-to-Prompt global-edit data generation · Time of day editing | FID: 39.9 -> 36.2 | Helped |
| use daytime-only source images (ablation) | Prompt-to-Prompt global-edit data generation · Time of day editing | CLIP Score: 0.214 -> 0.223 | Helped |
| concatenate source-image latents instead of conditioning through cross-attention (ablation) | SceneCrafter source-image conditioning · Time of day editing | FID: 50.3 -> 37.2 | Helped |
| concatenate source-image latents instead of conditioning through cross-attention (ablation) | SceneCrafter source-image conditioning · Time of day editing | CLIP Score: 0.203 -> 0.220 | Helped |
| add conditioning signals (ablation) | SceneCrafter generation model · Multi-view generation | FID: 68.5 -> 36.2 | Helped |
| add conditioning signals (ablation) | SceneCrafter generation model · Multi-view generation | 3D LPIPS: 0.254 -> 0.187 | Helped |
| use SceneCrafter instead of SDEdit or P2P* | Global editing · Time of day editing | FID: 60.4 -> 37.2; 46.8 -> 37.2 | Helped |
| use SceneCrafter instead of P2P* | Global editing · Time of day editing | CLIP Score: 0.223 -> 0.220 | Hurt |
| use SceneCrafter instead of SDEdit or P2P* | Global editing · Weather editing | User Study: 1.8% -> 85.5%; 12.7% -> 85.5% | Helped |
Surprising findings
- Replacing image-to-image self-attention rather than the usual cross-attention in Prompt-to-Prompt gave far more geometrically consistent pairs.
- Concatenating source images as pixel-level conditioning beat cross-attention conditioning (FID 37.2 vs 50.3).
Limitations
- Synthetic pair quality depends on teacher model and Prompt-to-Prompt attention replacement
- Evaluation relies on an internal proprietary dataset
Experimental setup
Datasets: Waymo internal multi-camera driving logs (13.9M segments)
Benchmarks: Waymo internal driving logs held-out 1% test split
Metrics: FID CLIP Score user study preference 3D LPIPS
Baselines: SDEdit P2P* 2D-RePaint MV-RePaint CAT3D
Builds on: Prompt-to-Prompt CAT3D RePaint InstructPix2Pix SDEdit
Closest papers by meaning
- DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes (AAAI 2025 · similarity 0.89)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.89)
- SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model (CVPR 2025 · similarity 0.89)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.89)
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.88)
- SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.