Research map / 3D vision / NeRF and radiance fields

InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules

ICLR 2024 · 3D vision › NeRF and radiance fields · approach: Sparse-input NeRF regularization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InsertNeRF, which inserts plug-and-play HyperNet modules (with sampling-aware graph filters, dynamic MLPs, and dynamic FiLM activations) into vanilla NeRF-like frameworks, fed by a multi-layer dynamic-static multi-view aggregation, to generate scene-specific NeRF weights for generalizable novel view synthesis.

Problem

Generalizing NeRF to unseen scenes without retraining, while preserving the original NeRF-like framework (vanilla NeRF, mip-NeRF, NeRF++) and its efficiency, as existing generalizable NeRF methods rely on transformers or cost volumes that are hard to attach to NeRF derivatives and slow.

Approach

Instead of replacing NeRF's network with transformers, insert HyperNet modules before each NeRF MLP layer that predict scene-specific MLP weights, biases and dynamic activations from multi-layer global-local reference features aggregated with a dynamic-static weighting scheme, with graph-based sampling-aware filters reducing epipolar noise.

Key results

• In Setting I, InsertNeRF reaches 30.35 PSNR on NeRF Synthetic, 26.44 on LLFF and 29.75 on DTU, outperforming NeuRay (28.92, 25.85, 28.30) and the other listed baselines.
• In Setting II, InsertNeRF attains 27.57 PSNR on NeRF Synthetic and 25.68 on LLFF, versus 27.29 and 25.59 for GNT.
• Insert-NeRF++ roughly doubles the PSNR of the original NeRF++ in the cross-scene setting, per the authors.

Measured effects reported in the paper

Surprising findings

• Directly using a conventional hypernetwork inside NeRF (NeRF with HyperNetwork) and removing the original NeRF MLP both hurt performance, suggesting the HyperNet modules must be combined with the original MLP layers.

Limitations

• Sparse-input results (3-view LLFF-style) are only modestly competitive: InsertNeRF (w/o retrain) reaches 19.41 PSNR, below FreeNeRF's 19.63, and the authors say fine-tuning would be needed for better results.
• Trains and evaluates on a few mainstream datasets with nearby reference views; ShapeNet gains are smaller, attributed to simple appearance.

Experimental setup

Datasets: NeRF Synthetic LLFF DTU Google Scanned Objects Tanks and Temples ShapeNet

Benchmarks: NeRF Synthetic LLFF DTU ShapeNet

Metrics: PSNR SSIM LPIPS

Baselines: PixelNeRF MVSNeRF IBRNet ContraNeRF GeoNeRF WaveNeRF NeuRay GNT DietNeRF RegNeRF GeCoNeRF FreeNeRF

Builds on: NeRF IBRNet mip-NeRF NeRF++ hypernetworks

Closest papers by meaning

• HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork (NeurIPS 2023 · similarity 0.91)
• HyperFields: Towards Zero-Shot Generation of NeRFs from Text (ICML 2024 · similarity 0.91)
• Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.90)
• Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.89)
• NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.89)
• ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.89)