Research map / 3D vision / 3D generation
ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConsistDreamer, an IN2N-style instruction-guided 3D scene editing framework that makes InstructPix2Pix-based 2D diffusion 3D-consistent via structured noise, surrounding-view inputs, and self-supervised consistency-enforcing training.
Problem
Instruction-guided 3D scene editing by distilling 2D diffusion models into NeRFs suffers from cross-view and cross-batch inconsistency, which causes blurred results and loss of regular patterns like checkered textures.
Approach
Augment the 2D editing diffusion model with 3D awareness and consistency by (1) fixing 3D-consistent structured Gaussian noise rendered per view, (2) feeding surrounding-view composed images containing a main view and reference views, and (3) training with a self-supervised loss against depth-warped, weighted-averaged consistent target views, with a ControlNet 3D embedding condition and LoRA.
Key results
- On ablation tasks (A)-(D), the full ConsistDreamer achieves the lowest distillation fidelity score (e.g., 27.4 vs 35.0 without structured noise on task A, and 62.7 vs 83.1 on task C).
- Qualitatively, ConsistDreamer is the first to successfully edit plaid/checkered patterns and produces sharper textures than IN2N and ViCA-NeRF on the Fangzhou scene and ScanNet++ scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add structured noise instead of independently generated noise (ablation) | ConsistDreamer · Tasks A–D | DFS: 35.0±0.5 -> 27.4±0.4; 234.8±2.4 -> 159.8±2.1; 83.1±2.2 -> 62.7±1.9; 217.8±2.0 -> 132.8±1.2 | Helped |
| Add consistency-enforcing training (ablation) | ConsistDreamer · Tasks A–D | DFS: 34.6±1.1 -> 27.4±0.4; 221.8±2.0 -> 159.8±2.1; 80.4±2.0 -> 62.7±1.9; 201.8±1.9 -> 132.8±1.2 | Helped |
| Add structured noise and consistency-enforcing training (ablation) | ConsistDreamer with surrounding views · Tasks A–D | DFS: 34.0±0.3 -> 27.4±0.4; 262.0±2.7 -> 159.8±2.1; 83.9±2.2 -> 62.7±1.9; 214.4±2.0 -> 132.8±1.2 | Helped |
| Add structured noise, surrounding views, and consistency-enforcing training (ablation) | IN2N variant · Tasks A–D | DFS: 91.0±1.2 -> 27.4±0.4; 255.8±2.0 -> 159.8±2.1; 90.9±1.5 -> 62.7±1.9; 222.4±2.0 -> 132.8±1.2 | Helped |
Surprising findings
- Varying the noise across iterations leads to highly different generation results, so prior distillation methods cannot even reproduce identical images for the same view.
Limitations
- Quantitative evaluation is limited; baselines compared mainly qualitatively, and the DFS metric measures fidelity to the base diffusion model rather than edit quality.
- Shape-changing edits require a coarse-to-fine strategy because structured noise depends on geometry.
- Editing is bounded by the capability of the underlying InstructPix2Pix model.
Experimental setup
Datasets: IN2N scenes ScanNet++
Benchmarks: IN2N scenes ScanNet++ scenes
Metrics: distillation fidelity score (FID-based)
Baselines: Instruct-NeRF2NeRF ViCA-NeRF NeRF-Art CSD DreamEditor EN2N PDS GE
Builds on: Instruct-NeRF2NeRF InstructPix2Pix DreamFusion ControlNet Nerfstudio
Closest papers by meaning
- SyncNoise: Geometrically Consistent Noise Prediction for Instruction-based 3D Editing (AAAI 2025 · similarity 0.91)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.89)
- Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion (CVPR 2024 · similarity 0.88)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.88)
- ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion (CVPR 2024 · similarity 0.88)
- PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.