Research map / 3D vision / 3D generation
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DriveEditor, a diffusion-based framework that unifies repositioning, insertion, replacement, and deletion of objects in driving videos via depth-aware 3D box control and three-level appearance control.
Problem
Object-level editing in driving videos is hard: existing diffusion video editors lack precise 3D position control and struggle to preserve high-fidelity object appearance from a reference image, limiting data augmentation for vision-centric autonomous driving.
Approach
A single diffusion video model (fine-tuned from SVD) is trained by reconstructing occluded objects; editing tasks share inputs (reference image, 3D box, masked video). Position is controlled by depth-aware 6-face projections injected into ResBlocks; appearance is kept via cut-and-paste, CLIP cross-attention, and multi-view features from a frozen SV3D through a 3D Information Fusion Module.
Key results
- Deletion: DriveEditor FID 30.17 / FVD 228 vs ProPainter 36.13 / 365 and SD 30.89 / 457.
- Replacement: DriveEditor FID 11.83 / FVD 39 vs T2V 14.22 / 151 and TAV 58.25 / 783.
- Augmenting 50% of nuScenes with generated repositioning and replacement data raises StreamPETR NDS from 0.569 to 0.581.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DriveEditor instead of SD for deletion | Deletion · nuScenes validation set | FVD: 457 -> 228 | Helped |
| Use DriveEditor instead of ProPainter for deletion | Deletion · nuScenes validation set | FID: 36.13 -> 30.17 | Helped |
| Use DriveEditor instead of T2V for replacement | Replacement · nuScenes validation set | FVD: 151 -> 39 | Helped |
| Use DriveEditor instead of TAV for replacement | Replacement · nuScenes validation set | FID: 58.25 -> 11.83 | Helped |
| Add a non-depth Pose Controller (PC) (ablation) | DriveEditor · Reconstruction | mAOE: 0.197 -> 0.148 | Helped |
| Add a non-depth Pose Controller (PC) (ablation) | DriveEditor · Reconstruction | FID: 6.56 -> 6.59 | Hurt |
| Add Depth-aware Projection (DP) to PC (ablation) | DriveEditor · Reconstruction | mAOE: 0.148 -> 0.061 | Helped |
| Add Depth-aware Projection (DP) to PC (ablation) | DriveEditor · Reconstruction | FID: 6.59 -> 6.49 | Helped |
| Add 3D priors through 3D-IFM to PC and DP (ablation) | DriveEditor · Reconstruction | FID: 6.49 -> 6.36 | Helped |
| Add 3D priors through 3D-IFM to PC and DP (ablation) | DriveEditor · Reconstruction | mAOE: 0.061 -> 0.043 | Helped |
| Add repositioning-augmented data | StreamPETR training · nuScenes detection | mAOE: 0.378 -> 0.340 | Helped |
| Add replacement-augmented data | StreamPETR training · nuScenes detection | mATE: 0.615 -> 0.588 | Helped |
Surprising findings
- Adding positional control without depth information (PC alone) slightly degrades image quality metrics compared with the baseline, which the authors attribute to interference from injected position features.
Limitations
- Low-contrast or small objects (rain, nighttime) can yield blurred or incomplete repositioned objects due to segmentation difficulty.
- Large appearance mismatch between user-specified object and scene (e.g., daytime object in nighttime scene) causes visual distortions.
- Trained on 10-frame clips; long videos require iterative editing.
Experimental setup
Datasets: nuScenes Waymo
Benchmarks: nuScenes validation set editing Waymo (zero-shot qualitative)
Metrics: FID FVD CLIP-I PSNR LPIPS mRecall mATE mAOE mAP NDS
Baselines: ProPainter Stable Diffusion Inpainting Tune-A-Video Text2Video-Zero
Builds on: Stable Video Diffusion Stable Video 3D ControlNet Tune-A-Video Panacea
Closest papers by meaning
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.89)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.87)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.87)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.86)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.85)
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.