Research map / 3D vision / Depth and stereo
Novel View Synthesis with View-Dependent Effects from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NVSVDE-Net, a self-supervised single-image novel view synthesis method that models view-dependent effects as negative-disparity samples along epipolar lines and uses a relaxed volumetric rendering approximation computed in one backbone pass.
Problem
Single-image novel view synthesis typically ignores view-dependent effects (specularities/reflections), and existing methods need depth or pose annotations and are slow due to per-sample MLP evaluation.
Approach
Model VDEs as negative disparities induced by target camera motion, infusing them into the input image by aggregating colors along the negative-depth region of epipolar lines, combined with a relaxed volumetric rendering that predicts ray point weights in a single backbone pass and a refined sampler, trained only on image sequences with an improved coarse-to-fine PoseNet.
Key results
- On RealEstate10k, NVSVDE-Net reaches 24.31 PSNR and 30.25 PSNRlf vs 23.61 PSNR and 28.96 PSNRlf for SceneRF.
- On MannequinChallenge, NVSVDE-Net reaches 22.43 PSNR vs 21.60 for SceneRF.
- Runtime is 42 ms per 640x360 image on an A6000 (about 33 FPS), versus 160 ms for SceneRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VDE synthesis (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.1679 -> 24.3131 | Helped |
| Enable VDE estimation (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.2092 -> 24.3131 | Helped |
| Add depth and VDE logit recalibration blocks (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 23.9710 -> 24.3131 | Helped |
| Use adaptive sampling with N=16 instead of 48 coarse samples and no adaptive samples (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 23.9613 -> 24.3131 | Helped |
| Use 16 adaptive samples instead of 32 (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.3007 -> 24.3131 | No clear effect |
| Use fine-grained output instead of coarse synthetic output (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.1020 -> 24.3131 | Helped |
| Estimate ray point weights instead of density σ (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.1729 -> 24.3131 | Helped |
| Use the full method's positional encoding instead of periodic γ (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.2506 -> 24.3131 | No clear effect |
| Use resize-crop augmentation and test at 1/2 resolution (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.1030 -> 24.3131 | Helped |
| Use ResNet34 instead of ResNet18 (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.0340 -> 24.3131 | Helped |
| Use ResNet34 instead of Swin-t (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 24.1917 -> 24.3131 | Helped |
| Use the improved PoseNet instead of [49]'s PoseNet (ablation) | NVSVDE-Net · RealEstate10k | PSNR: 23.8736 -> 24.3131 | Helped |
Surprising findings
- Estimating density sigma instead of T(t)sigma(t) was more expensive and gave slightly worse metrics.
- The PSNR of the coarse output is lower than the fine output, but higher than a coarse-only model with more samples, showing joint supervision helps.
Limitations
- VDE modeling is limited to simple glossy/diffuse specular reflections plausible from single images.
- Evaluation uses PSNRlf, a custom metric, and SSIM/LPIPS show low sensitivity to VDEs.
- Baselines were retrained under the authors' self-supervised setup, and some were unstable early in training.
Experimental setup
Datasets: RealEstate10k MannequinChallenge
Benchmarks: RealEstate10k MannequinChallenge
Metrics: MAE PSNR PSNRlf SSIM LPIPS
Baselines: PixelNeRF BehindScenes MINE Single-view MPI SceneRF
Builds on: NeRF PixelNeRF BehindTheScenes SceneRF MINE
Closest papers by meaning
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.89)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.87)
- Pseudo-Generalized Dynamic View Synthesis from a Video (ICLR 2024 · similarity 0.87)
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.87)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.87)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.