Research map / 3D vision / 3D generation
SyncNoise: Geometrically Consistent Noise Prediction for Instruction-based 3D Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyncNoise, which edits multi-view renderings with an instruction-based 2D diffusion model while enforcing geometric consistency on U-Net noise-prediction features and propagating anchor views via depth-filtered reprojection, to produce consistent, detailed 3D (NeRF/Gaussian) edits.
Problem
Instruction-based text-driven 3D scene editing with 2D diffusion models suffers from multi-view inconsistency; iterative dataset update (IN2N) is slow and over-smooths, while latent-level consistency (EN2N) blurs high-frequency details.
Approach
Align U-Net decoder noise-prediction features across views using SfM-supervised depth and reprojection/cycle-consistency filtered correspondences (instead of latents), then propagate CLIP-selected anchor views to neighbors via pixel reprojection, with masked diffusion refinement to remove artifacts.
Key results
- SyncNoise (NeRF) reaches 30.86% CLIP score-type fidelity and 18.31% text-image direction similarity, vs 29.18%/16.49% for IN2N, with 23 min vs 57 min editing time.
- SyncNoise on Gaussian splatting attains 18.14% CLIP text-image direction similarity in 7 min, beating GaussianEditor's 17.04% in 11 min.
- Ablation: dense alignment of U-Net features yields consistent edits but fails to change geometry and over-smooths; sparse alignment edits geometry while keeping consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add geometrically consistent noise alignment and cross-view pixel-level projection | SyncNoise on NeRF · Four-scene, 10-prompt evaluation | CLIP Text-Image Direction Similarity: 17.04% -> 18.31% | Helped |
| Add geometrically consistent noise alignment and cross-view pixel-level projection | SyncNoise on NeRF · Four-scene, 10-prompt evaluation | MUSIQ: 64.425 -> 66.668 | Helped |
| Add cross-view pixel-level projection to noise alignment (ablation) | SyncNoise on NeRF · Four-scene, 10-prompt evaluation | CLIP Score: 30.64% -> 30.86% | Helped |
| Add cross-view pixel-level projection to noise alignment (ablation) | SyncNoise on NeRF · Four-scene, 10-prompt evaluation | CLIP-IQA: 0.559 -> 0.540 | Hurt |
| Add geometrically consistent noise alignment to pixel projection (ablation) | SyncNoise on NeRF · Four-scene, 10-prompt evaluation | CLIP Score: 29.50% -> 30.86% | Helped |
| Use dense rather than sparse feature alignment (ablation) | SyncNoise · Multi-view 3D scene editing | Texture detail | Hurt |
| Add masked diffusion refinement after cross-view pixel projection (ablation) | SyncNoise · Novel views after reprojection | Artifact reduction | Helped |
| Align decoder features rather than skip features (ablation) | SyncNoise · Multi-view image editing | Multi-view consistency | Helped |
Surprising findings
- Aligning U-Net decoder features rather than latent maps avoids the blurring seen in latent averaging, and skip-feature alignment has minor impact.
Limitations
- Quantitative evaluation covers only four scenes and ten prompts.
- Reprojection can introduce artifacts in novel views, requiring masked refinement.
- Relies on SfM keypoints for depth supervision, and correspondence quality decreases as view span increases.
Experimental setup
Datasets: multi-view scenes from 3D editing benchmarks (bear, face, fangzhou, person)
Benchmarks: own 3D editing scenes (4 scenes, 10 text prompts)
Metrics: CLIP score CLIP text-image direction similarity CLIP temporal direction similarity CLIP-IQA MUSIQ running time
Baselines: Instruct-NeRF2NeRF Efficient-NeRF2NeRF GaussianEditor DreamEditor
Builds on: InstructPix2Pix Instruct-NeRF2NeRF Efficient-NeRF2NeRF Plug-and-Play Structure-from-Motion
Closest papers by meaning
- ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing (CVPR 2024 · similarity 0.91)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.88)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.87)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.87)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.