Research map / 3D vision / 3D generation
NViST: In the Wild New View Synthesis from a Single Image with Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NViST, a transformer encoder-decoder that predicts a vector-matrix radiance field from a single in-the-wild image, trained on MVImgNet with relative 6-DoF pose and no canonicalization.
Problem
Generalizable novel-view synthesis from a single image for real-world, casually captured scenes with diverse backgrounds and scale ambiguities, where prior methods rely on synthetic object-centred data, aligned poses, or 3-DoF camera models.
Approach
A fine-tuned MAE encoder feeds features into a decoder where learnable output tokens attend to image features via cross-attention and to each other via self-attention; camera focal length and distance are injected via adaptive layer normalisation; the output is a TensoRF-style vector-matrix radiance field trained with relative 6-DoF poses only.
Key results
- On MVImgNet test scenes, NViST achieves 20.83 PSNR, 0.57 SSIM and 0.29 LPIPS, outperforming PixelNeRF (17.02) and VisionNeRF (19.82).
- On ShapeNet-SRN, NViST reaches 23.91 PSNR on cars and 24.50 on chairs, comparable to baselines, while using only relative pose.
- Ablation: removing end-to-end encoder updating drops PSNR from 20.83 to 18.54, and replacing VM with triplane drops it to 19.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train NViST on MVImgNet with the proposed transformer and VM radiance-field design | NViST · MVImgNet test | PSNR: 19.82 -> 20.83 | Helped |
| Train NViST on MVImgNet with the proposed transformer and VM radiance-field design | NViST · MVImgNet test | LPIPS: 0.47 -> 0.29 | Helped |
| add LPIPS perceptual loss (ablation) | NViST · MVImgNet test | LPIPS: 0.46 -> 0.29 | Helped |
| add camera conditioning (ablation) | NViST · MVImgNet test | PSNR: 20.24 -> 20.83 | Helped |
| use adaptive layer normalization instead of concatenating camera parameters to output tokens (ablation) | NViST · MVImgNet test | PSNR: 20.81 -> 20.83 | No clear effect |
| use cross-attention instead of self-attention over concatenated feature and output tokens (ablation) | NViST · MVImgNet test | PSNR: 20.74 -> 20.83 | No clear effect |
| use VM representation instead of triplane representation (ablation) | NViST · MVImgNet test | PSNR: 19.60 -> 20.83 | Helped |
| update encoder weights during end-to-end training (ablation) | NViST · MVImgNet test | PSNR: 18.54 -> 20.83 | Helped |
Surprising findings
- Updating the MAE encoder end-to-end gives a large gain (PSNR 18.54 without vs 20.83 with), so the pretrained features alone are insufficient.
- On ShapeNet-SRN, NViST does not clearly beat specialised baselines despite using only relative pose, performing similarly rather than better.
Limitations
- Images downsampled by x12 and training on a fraction of MVImgNet, causing loss of sharpness.
- No GAN or SDS losses, which may reduce detail.
- Single-image input only; no multiview extension.
Experimental setup
Datasets: MVImgNet ShapeNet-SRN
Benchmarks: MVImgNet held-out test scenes ShapeNet-SRN cars and chairs
Metrics: PSNR SSIM LPIPS
Baselines: PixelNeRF VisionNeRF Zero-1-to-3 VD SSDNeRF
Builds on: MAE TensoRF NeRF PixelNeRF LPIPS
Closest papers by meaning
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.87)
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.87)
- One View Is Enough: In-the-Wild Monocular Pretraining for Novel View Generation (NeurIPS 2026 · similarity 0.87)
- WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild (NeurIPS 2025 · similarity 0.87)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.