Research map / 3D vision / NeRF and radiance fields
Mask-Based Modeling for Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRVM-NeRF, a latent-space masked ray and view modeling pretraining objective (BYOL-style coarse/fine branches) added to generalizable NeRFs, improving NeuRay and NeRFormer on ShapeNet, NeRF Synthetic, DTU and LLFF.
Problem
Generalizable NeRFs condition on pixel-aligned image features but lack an explicit mechanism for interacting across points along rays and across reference views, limiting precise global scene representations and generalization to novel scenes.
Approach
Apply mask-then-predict pretraining inside the NeRF latent space: randomly mask point features along each ray and per-view tokens across reference views, then align fine-branch predictions with unmasked coarse-branch latents (BYOL-style with moving-average target) jointly with the rendering loss.
Key results
- On ShapeNet-all, NeRFormer+MRVM reaches 29.25 PSNR / 0.942 SSIM / 0.060 LPIPS vs. 27.58 / 0.920 / 0.091 for NeRFormer.
- On the cross-scene generalization setting, NeuRay+MRVM reaches 29.48 PSNR on DTU vs. 26.47 for NeuRay, and 26.91 vs. 25.35 on LLFF.
- In few-shot NeRF Synthetic (10-3 views), NeuRay+MRVM improves PSNR from 22.19 to 24.03.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MRVM pretraining | NeRFormer · ShapeNet-all | PSNR: 27.58 -> 29.25 | Helped |
| Add MRVM pretraining | NeRFormer · ShapeNet-unseen | PSNR: 22.54 -> 24.08 | Helped |
| Add MRVM pretraining | NeRFormer · ShapeNet-chair 1-view | PSNR: 23.56 -> 24.65 | Helped |
| Add MRVM pretraining | NeRFormer · ShapeNet-chair 2-views | PSNR: 25.79 -> 26.87 | Helped |
| Add MRVM pretraining | NeuRay · NeRF Synthetic, DTU, and LLFF, cross-scene generalization and per-scene finetuning | PSNR: 28.29 -> 29.29; 26.47 -> 29.48; 25.35 -> 26.91; 32.35 -> 33.09; 29.79 -> 31.98; 27.06 -> 28.37 | Helped |
| Add MRVM pretraining | GNT · NeRF Synthetic and LLFF | PSNR: 27.29 -> 27.78; 25.86 -> 26.25 | Helped |
| Use RGB block masking and pixel reconstruction instead of MRVM (ablation) | NeRFormer · ShapeNet-all | PSNR: 27.58 -> 27.95 | Helped |
| Add a decoder to reconstruct masked latent features (Feat mask¹) (ablation) | NeRFormer · ShapeNet-all | PSNR: 27.58 -> 28.58 | Helped |
| Replace the coarse-branch target with a moving-average fine-branch copy (Feat mask²) (ablation) | NeRFormer · ShapeNet-all | PSNR: 27.58 -> 28.02 | Helped |
| Set the MRVM mask ratio to 0.1 rather than 0.5 (ablation) | NeRFormer+MRVM · ShapeNet-all | PSNR: 29.25 -> 27.88 | Hurt |
| Set the MRVM mask ratio to 0.25 rather than 0.5 (ablation) | NeRFormer+MRVM · ShapeNet-all | PSNR: 29.25 -> 28.54 | Hurt |
| Set the MRVM mask ratio to 0.75 rather than 0.5 (ablation) | NeRFormer+MRVM · ShapeNet-all | PSNR: 29.25 -> 28.96 | Hurt |
Surprising findings
- Masking raw RGB pixels (MIM-style) gave a smaller gain than latent feature masking, and feature masking with a fine-branch target was unstable.
Limitations
- Requires long multi-stage training (about 7-10 days on GTX-1080Ti for ShapeNet settings).
- Gains on some categories and metrics are modest (e.g., GNT+MRVM LPIPS on LLFF is slightly worse than GNT-MOVE).
Experimental setup
Datasets: NMR ShapeNet NeRF Synthetic DTU LLFF Google Scanned Objects
Benchmarks: NMR ShapeNet NeRF Synthetic DTU LLFF
Metrics: PSNR SSIM LPIPS
Baselines: NeRFormer NeuRay SRN PixelNeRF FE-NVS SRT VisionNeRF CodeNeRF MVSNeRF IBRNet NeRF GNT
Builds on: NeuRay NeRFormer BYOL MipNeRF masked image modeling
Closest papers by meaning
- NeRFLight: Fast and Light Neural Radiance Fields Using a Shared Feature Grid (CVPR 2023 · similarity 0.92)
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.91)
- Aug-NeRF: Training Stronger Neural Radiance Fields With Triple-Level Physically-Grounded Augmentations (CVPR 2022 · similarity 0.91)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.90)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.90)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.