Research map / 3D vision / 3D generation
CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes InstructPix2Pix on a single edited reference image (with prior preservation and affine augmentation) and uses it in a two-stage, edited-image-buffer optimization of deformable 3D Gaussians for controllable dynamic 3D scene editing.
Problem
Controllable and temporally consistent local editing of dynamic 3D scenes is hard because editing backbones are limited and tracking edit regions across frames is difficult.
Approach
Personalize InstructPix2Pix from one user-edited reference frame (any 2D editor) so it learns the edit and its region, reducing dynamic scene editing to 2D editing; then optimize deformable 3D Gaussians in two stages with an edited-image buffer and temporal loss.
Key results
- On Portrait, Cat, and Steak scenes, the method achieves CLIP scores of 27.75, 31.81, 28.52 vs IN4D's 27.38, 31.72, 28.23.
- Subject consistency is 0.953, 0.968, 0.988 vs IN4D's 0.933, 0.964, 0.983.
- Total runtime is 60, 60, and 40 minutes vs 2 hours for IN4D, under half the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Portrait | CLIP score: 27.38 -> 27.75 | Helped |
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Portrait | Consistency: 0.933 -> 0.953 | Helped |
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Cat | CLIP score: 31.72 -> 31.81 | Helped |
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Cat | Consistency: 0.964 -> 0.968 | Helped |
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Steak | CLIP score: 28.23 -> 28.52 | Helped |
| use CTRL-D instead of Instruct 4D-to-4D | Dynamic scene editing · Steak | Consistency: 0.983 -> 0.988 | Helped |
| add affine data augmentation during IP2P fine-tuning (ablation) | CTRL-D · Dynamic scene editing | Editing quality and consistency | Helped |
| remove affine data augmentation during IP2P fine-tuning (ablation) | CTRL-D · Dynamic scene editing | Editing quality and consistency | Hurt |
| use an edited image buffer to select frames for optimization (ablation) | CTRL-D · Dynamic scene editing | Editing result | Helped |
| replace edited-buffer frame selection with random frame selection (ablation) | CTRL-D · Dynamic scene editing | Editing result | Hurt |
Limitations
- Inherits reconstruction failures of the 3D Gaussian backbone, producing blurry edits for moving parts such as hands.
- Inherits IP2P difficulty generating 3D-consistent content in empty regions, e.g., adding new objects like a bag.
- Relies on a single reference image, so one-shot fine-tuning risks overfitting, mitigated only by affine augmentation.
Experimental setup
Datasets: DyCheck N3DV in-house human-centric dynamic scenes
Benchmarks: DyCheck N3DV
Metrics: CLIP score subject consistency (VBench) running time
Baselines: Instruct 4D-to-4D AnyV2V
Builds on: InstructPix2Pix DreamBooth 3D Gaussian Splatting Instruct-NeRF2NeRF Dynamic 3D Gaussians
Closest papers by meaning
- Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion (CVPR 2024 · similarity 0.91)
- Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separation (CVPR 2025 · similarity 0.90)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.90)
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting (ICLR 2025 · similarity 0.89)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.