Research map / 3D vision / Gaussian splatting
DIFIX3D+: Improving 3D Reconstructions with Single-Step Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the single-step SD-Turbo diffusion model into DIFIX, an artifact fixer for NeRF/3DGS renderings, and uses it both to clean pseudo-training views during progressive 3D updates and as a real-time post-render enhancer (DIFIX3D+).
Problem
Novel-view synthesis from NeRF and 3D Gaussian Splatting produces artifacts (spurious geometry, missing regions, blur) in underconstrained regions and at extreme novel viewpoints, especially with sparse inputs or imperfect poses, and existing diffusion-prior methods are slow because they query the d
Approach
Fine-tune a single-step diffusion model (SD-Turbo, with LoRA decoder and reference-view mixing attention) to map artifact-laden renderings to clean images using data curated from NeRF/3DGS failure modes, then distill its outputs back into the 3D representation via incremental pose-perturbed updates and apply it at render time.
Key results
- On DL3DV, DIFIX3D+ (3DGS) reaches 17.99 PSNR and 40.86 FID versus 17.18 PSNR and 107.23 FID for the 3DGS baseline.
- On the Nerfbusters dataset, DIFIX3D+ (Nerfacto) improves PSNR from 17.29 to 18.32 and FID from 134.65 to 49.44.
- On the in-house RDS driving dataset, Nerfacto + DIFIX3D+ improves PSNR from 19.95 to 21.75 and FID from 91.38 to 73.08; the paper reports a 76 ms render-time cost per view on an A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DIFIX directly to rendered views, without 3D updates (ablation) | Nerfacto · Nerfbusters | FID: 134.65 -> 49.87 | Helped |
| Distill DIFIX outputs with a non-incremental 3D update (ablation) | Nerfacto + DIFIX · Nerfbusters | FID: 49.87 -> 75.94 | Hurt |
| Apply 3D updates incrementally (ablation) | Nerfacto + DIFIX with 3D updates · Nerfbusters | FID: 75.94 -> 63.77 | Helped |
| Add DIFIX as a post-rendering step (ablation) | DIFIX3D · Nerfbusters | FID: 63.77 -> 49.44 | Helped |
| Use noise level τ=200 instead of τ=1000 (ablation) | DIFIX · Nerfbusters | LPIPS: 0.3810 -> 0.3190 | Helped |
| Add Gram loss (ablation) | DIFIX with τ=200 · Nerfbusters | LPIPS: 0.3190 -> 0.3064 | Helped |
| Condition on reference views (ablation) | DIFIX with τ=200 and Gram loss · Nerfbusters | LPIPS: 0.3064 -> 0.2996 | Helped |
| Use τ=200 instead of τ=1000 (ablation) | DIFIX · Nerfbusters | FID: 108.86 -> 61.80 | Helped |
| Add Gram loss (ablation) | DIFIX with τ=200 · Nerfbusters | FID: 61.80 -> 55.45 | Helped |
| Condition on reference views (ablation) | DIFIX with τ=200 and Gram loss · Nerfbusters | FID: 55.45 -> 47.87 | Helped |
Surprising findings
- Directly applying the fixer to rendered views without 3D updates gives inconsistent results across poses, and adding all pseudo-views at once degrades LPIPS and FID relative to incremental updates.
Limitations
- Generative hallucination can cause inconsistencies across poses in under-observed regions, which requires the progressive update scheme.
- Some regions remain blurry because reconstruction methods have limited capacity to represent sharp details.
- The driving dataset is in-house and not released.
Experimental setup
Datasets: DL3DV-10K Nerfbusters Internal real driving scene (RDS) dataset
Benchmarks: Nerfbusters dataset DL3DV dataset RDS driving dataset
Metrics: PSNR SSIM LPIPS FID
Baselines: Nerfbusters GANeRF NeRFLiX Nerfacto 3DGS pix2pix-Turbo
Builds on: SD-Turbo Pix2pix-Turbo Instruct-NeRF2NeRF NeRF 3D Gaussian Splatting
Closest papers by meaning
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.90)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (CVPR 2024 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- ReconFusion: 3D Reconstruction with Diffusion Priors (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.