Research map / 3D vision / 3D generation
Enhancing Novel View Synthesis via Geometry Grounded Set Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SetDiff, a geometry-grounded set diffusion enhancer that restores 3DGS novel-view renderings by conditioning a single-step SD-Turbo UNet on coordinate maps and Plücker ray embeddings while jointly processing variable-size reference and target view sets.
Problem
Novel view synthesis from 3D Gaussian Splatting degrades when rendering views far from training trajectories (extrapolation), producing artifacts, floaters, and hallucinations in autonomous driving and general scenes.
Approach
Condition an image diffusion enhancer on pixel-aligned 3DGS coordinate maps and pose-aware Plücker embeddings, and replace self-attention with a set mixer that performs joint attention across all reference and target tokens so the model handles arbitrary numbers and orders of views.
Key results
- On EUVS Setting 3, SetDiff reaches 16.60 PSNR / 0.745 SSIM / 0.291 LPIPS versus 15.91 / 0.743 / 0.335 for DiFix3D++.
- On nuScenes extrapolated trajectories, SetDiff reaches 21.58 PSNR and 0.303 LPIPS over OmniRe's 17.15 PSNR and 0.448 LPIPS, beating DiFix3D++ at 18.74 PSNR.
- On DL3DV 3-view sparse setting, SetDiff reaches 16.87 PSNR and 0.335 LPIPS, above DiFix3D++ (15.93 / 0.361).
Surprising findings
- Diffusion enhancers improve image metrics substantially but give only limited gains in neural reconstruction consistency (Warp error), since geometry-based reconstruction already enforces cross-view consistency.
- On EUVS Setting 2, VEGS reaches the highest PSNR (23.33) yet has worse SSIM/LPIPS than SetDiff, and SetDiff's PSNR is below DiFix3D++ on EUVS Setting 1.
Limitations
- Degrades when reference-target geometric overlap is very limited (extreme viewpoint shifts or heavy occlusion), relying more on the diffusion prior and risking hallucinations.
- C-maps become noisy and unreliable at long temporal extrapolation with low SNR.
- Operates as a post-processor and does not update 3DGS geometry.
Experimental setup
Datasets: EUVS Para-Lane nuScenes DL3DV-10K
Benchmarks: EUVS Para-Lane nuScenes DL3DV-10K
Metrics: PSNR SSIM LPIPS Warp consistency error
Baselines: 3DGS 2DGS GSPro PGSR 3DGM Feature 3DGS Zip-NeRF Instant-NGP VEGS DiFix3D DiFix3D++ Scaffold-GS
Builds on: 3D Gaussian Splatting DiFix3D SD-Turbo Stable Diffusion 3DGS-Enhancer
Closest papers by meaning
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.89)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.88)
- Novel View Synthesis Under Large-Deviation Viewpoint for Autonomous Driving (AAAI 2025 · similarity 0.88)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.88)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.