Research map / 3D vision / NeRF and radiance fields
NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeRF-VPT, a cascading view prompt tuning method in which RGB renderings from the previous NeRF stage are fed back as view prompts concatenated to the direction encoding, iteratively refining novel view synthesis as a plug-and-play add-on.
Problem
Novel view synthesis with NeRF still struggles to produce high-quality images with fine details, good textures and high PSNR, and it depends heavily on dense views.
Approach
Render images with a pre-trained NeRF, then retrain stage by stage with the previous stage's RGB renderings concatenated to the direction encoding as a view prompt, so each stage gets progressively more accurate prior knowledge; the method is model-agnostic and plugs into NeRF, Mip-NeRF and TensoRF.
Key results
- On Realistic Synthetic 360, NeRF-VPT Stage 6 reaches 34.95 PSNR / 0.977 SSIM / 0.042 LPIPS versus TensoRF's 33.14 / 0.963 / 0.044.
- On Replica, applying view prompt tuning raises Mip-NeRF from 33.12 to 35.63 PSNR and TensoRF from 33.17 to 34.62 PSNR.
- With 45 views on Replica, NeRF-VPT Stage 1 reaches 31.56 PSNR, close to NeRF's 31.22 with 180 views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cascading view prompt tuning to NeRF | NeRF · Realistic Synthetic 360 | PSNR: 33.14 -> 34.95 | Helped |
| Add cascading view prompt tuning to NeRF | NeRF · Real Forward-Facing | PSNR: 27.13 -> 28.95 | Helped |
| Add view prompt tuning to Mip-NeRF | Mip-NeRF · Replica | PSNR: 33.12 -> 35.63 | Helped |
| Add view prompt tuning to TensoRF | TensoRF · User-Captured Dataset | PSNR: 27.01 -> 28.89 | Helped |
| Use 45-view NeRF-VPT instead of 180-view NeRF | NeRF · Replica | PSNR: 31.22 -> 31.56 | Helped |
| Add cascading view prompt tuning | NeRF · Replica | depth distance: 0.02 -> 0.02 | No clear effect |
| Use fused scene features as prior (ablation) | NeRF-VPT · Replica | PSNR: 31.22 -> 31.76 | Helped |
| Use Gaussian noise as direction prior (ablation) | NeRF-VPT · Replica | PSNR: 31.22 -> 11.58 | Hurt |
| Use ground-truth images as direction prior (ablation) | NeRF-VPT · Replica | PSNR: 31.22 -> 55.83 | Helped |
| Add last-stage images to position encoding rather than direction encoding (ablation) | NeRF-VPT · Replica | PSNR: 33.96 -> 32.88 | Hurt |
| Increase prompt-tuning stages from Stage 0 to Stage 6 (ablation) | NeRF-VPT · Replica | PSNR: 31.22 -> 35.39 | Helped |
Surprising findings
- Using ground-truth images as the view prompt yields a degenerate 55.83 PSNR, showing the prompt can leak answers, while Gaussian noise drops PSNR to 11.58.
- Depth quality does not improve (depth distance 0.02 for both NeRF and NeRF-VPT), despite the large image-quality gains.
Limitations
- Multi-stage training repeats the full training process, so total training time grows despite decreasing per-stage time.
- Depth estimation gains are limited; the method does not improve predicted depth.
- Stopping criterion relies on empirical observation (five to six stages), and the method is only evaluated on a few scene benchmarks.
Experimental setup
Datasets: Realistic Synthetic 360 Real Forward-Facing Replica User-captured dataset
Benchmarks: Realistic Synthetic 360 Real Forward-Facing Replica User-captured dataset
Metrics: PSNR SSIM LPIPS depth distance parameter number FPS
Baselines: NeRF IBRnet Recursive-NeRF Mip-NeRF TensoRF
Builds on: NeRF Mip-NeRF TensoRF visual prompt tuning
Closest papers by meaning
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.91)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.91)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.91)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.91)
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods (NeurIPS 2025 · similarity 0.90)
- CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.