Research map / 3D vision / NeRF and radiance fields
Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generalizable neural Point Field (GPF), a point-based alternative to image-based generalizable NeRFs, using visibility-oriented feature fetching, density-guided log sampling, and a feature-augmented learnable kernel.
Problem
Generalizable NeRFs built on image-based rendering suffer from occlusion-induced inconsistent feature matching, distortions at geometric discontinuities from independent per-point processing and rough feature aggregation, and degradation when source views are far from the target view.
Approach
Builds the generalizable neural field on a point scaffold (initialized by PatchmatchMVS) rather than image features, computes explicit geometric visibility scores from a point-density-derived depth prior to guide feature fetching, samples points nonuniformly in log-spaced positions around the estimated surface, and aggregates neighbor features with a learnable kernel augmented by neural features.
Key results
- In the generalization setting, GPF reaches 29.31 PSNR on NeRF Synthetic, 27.67 on DTU, and 26.65 on BlendedMVS, above Neuray (27.07, 25.94, 23.24).
- In the finetuning setting, GPF reaches 33.28 PSNR on NeRF Synthetic and 31.65 on DTU, versus 30.26 and 29.15 for Neuray.
- Log sampling with 16 points reaches 27.67 PSNR at 1.04s rendering versus 25.08 PSNR for uniform 128-point sampling at 103.2s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPF instead of the strongest listed baseline, Neuray | generalizable radiance-field rendering · NeRF Synthetic generalization | PSNR: 27.07 -> 29.31 | Helped |
| Use GPF instead of Neuray | generalizable radiance-field rendering · DTU generalization | PSNR: 25.94 -> 27.67 | Helped |
| Use GPF instead of Neuray | generalizable radiance-field rendering · BlendedMVS generalization | PSNR: 23.24 -> 26.65 | Helped |
| Use GPF instead of Neuray | finetuned radiance-field rendering · NeRF Synthetic finetuning | PSNR: 30.26 -> 33.28 | Helped |
| Use GPF instead of Neuray | finetuned radiance-field rendering · DTU finetuning | PSNR: 29.15 -> 31.65 | Helped |
| Use GPF instead of Neuray | finetuned radiance-field rendering · BlendedMVS finetuning | PSNR: 26.71 -> 28.78 | Helped |
| Add perturbation ε to log sampling (ablation) | GPF sampling · DTU generalization | PSNR: 27.13 -> 27.67 | Helped |
| Replace coarse-to-fine uniform sampling with log sampling (ablation) | GPF sampling · DTU generalization | PSNR: 26.56 -> 27.67 | Helped |
| Use log sampling instead of two surface samples (ablation) | GPF sampling · DTU generalization | PSNR: 24.10 -> 27.67 | Helped |
| Use two-point surface sampling instead of log sampling (ablation) | GPF sampling · DTU generalization | rendering time: 1.04s -> 0.62s | Helped |
| Add the feature-augmented learnable kernel instead of inverse distance weights (ablation) | GPF feature aggregation · DTU generalization | PSNR: 23.61 -> 27.67 | Helped |
| Add the feature-augmented learnable kernel instead of a radial basis kernel (ablation) | GPF feature aggregation · DTU generalization | PSNR: 24.75 -> 27.67 | Helped |
Surprising findings
- PointNeRF collapses in the generalization setting on NeRF Synthetic (6.12 PSNR), despite performing well on DTU after pretraining.
Limitations
- Requires PatchmatchMVS to initialize the point scaffold in the generalization stage.
- Point2Pix was reimplemented by the authors because no source code was available.
Experimental setup
Datasets: DTU NeRF Synthetic BlendedMVS LLFF
Benchmarks: NeRF Synthetic DTU BlendedMVS LLFF
Metrics: PSNR SSIM LPIPS RMSE Acc.T training time rendering time
Baselines: IBRNet MVSNeRF ENeRF Neuray PointNeRF Point2Pix
Builds on: NeRF PointNeRF PatchmatchMVS IBRNet ENeRF
Closest papers by meaning
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.91)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.90)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.90)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.90)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.90)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.