Research map / 3D vision / NeRF and radiance fields
ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ColNeRF, a generalizable sparse-input NeRF that uses cross-view attention over source-view feature volumes and self-supervised ray regularization (depth consistency and epipolar color consistency) without auxiliary supervision.
Problem
Novel view synthesis from sparse input views with NeRF degrades due to limited supervision, and existing generalizable methods fail to maintain multi-view geometric and appearance consistency; per-scene optimization is also slow.
Approach
Integrates collaboration at input and output ends: a Collaborative Cross-View Volume Integration (CCVI) module fuses patch-level features across source views via attention, and self-supervised geometry (paired neighboring-ray depth) and appearance (epipolar-line color KL) regularization enforce multi-view consistency without depth or semantic supervision.
Key results
- On DTU 3-view, ColNeRF reaches 19.55 PSNR and 0.716 SSIM, outperforming PixelNeRF (18.74 PSNR) among generalizable methods; it trails FreeNeRF at 9-view.
- On LLFF with fine-tuning (3-view), ColNeRF ft reaches 20.97 PSNR, the best among generalizable methods and above per-scene FreeNeRF's 19.63 at 3 views.
- Data efficiency: claims ~51% fewer input views than NeRF and ~35% fewer than PixelNeRF for similar test PSNR on DTU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ColNeRF collaboration and regularization compared with PixelNeRF | DTU generalizable sparse-view NeRF · DTU 3-view | PSNR: 18.74 -> 19.55 | Helped |
| add ColNeRF collaboration and regularization compared with PixelNeRF | DTU generalizable sparse-view NeRF · DTU 6-view | PSNR: 21.02 -> 22.94 | Helped |
| add ColNeRF collaboration and regularization compared with PixelNeRF | DTU generalizable sparse-view NeRF · DTU 9-view | PSNR: 22.23 -> 23.93 | Helped |
| use ColNeRF rather than FreeNeRF | DTU sparse-view NeRF · DTU 9-view | PSNR: 24.20 -> 23.93 | Hurt |
| fine-tune ColNeRF rather than FreeNeRF | LLFF sparse-view NeRF · LLFF 3-view | PSNR: 19.63 -> 20.97 | Helped |
| fine-tune ColNeRF rather than FreeNeRF | LLFF sparse-view NeRF · LLFF 6-view | PSNR: 23.73 -> 23.32 | Hurt |
| fine-tune ColNeRF rather than FreeNeRF | LLFF sparse-view NeRF · LLFF 9-view | PSNR: 25.13 -> 23.52 | Hurt |
| add cross-view volume integration (VI) (ablation) | ColNeRF without VI or ray regularization · DTU 3-view | PSNR: 18.74 -> 19.21 | Helped |
| add geometry regularization to VI (ablation) | ColNeRF with VI · DTU 3-view | PSNR: 19.21 -> 19.39 | Helped |
| add appearance regularization to VI (ablation) | ColNeRF with VI · DTU 3-view | PSNR: 19.21 -> 19.48 | Helped |
| replace ColNeRF ray regularization with InfoNeRF regularization (ablation) | ColNeRF without RayReg · DTU 3-view | PSNR: 18.74 -> 19.32 | Helped |
| replace ColNeRF ray regularization with RegNeRF regularization (ablation) | ColNeRF without RayReg · DTU 3-view | PSNR: 18.74 -> 18.11 | Hurt |
Surprising findings
- Ablation shows the InfoNeRF regularization gives 19.32 PSNR and the RegNeRF regularization drops PSNR to 18.11, below the no-regularization baseline of 18.74 on the same backbone.
- ColNeRF ft at 3 views on LLFF achieves higher PSNR (20.97) than per-scene optimized FreeNeRF (19.63) despite no per-scene training to convergence.
Limitations
- Lightweight design (fewer sample points, small feature dimension, average pooling) degrades reconstruction accuracy and yields locally smooth renderings.
- Performance lags FreeNeRF at 9-view on DTU.
Experimental setup
Datasets: DTU LLFF
Benchmarks: DTU LLFF
Metrics: PSNR SSIM LPIPS geometric mean of MSE, sqrt(1-SSIM), LPIPS
Baselines: PixelNeRF SRF MVSNeRF DietNeRF DS-NeRF InfoNeRF RegNeRF FreeNeRF
Builds on: NeRF PixelNeRF InfoNeRF RegNeRF
Closest papers by meaning
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.90)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.89)
- AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization (AAAI 2024 · similarity 0.89)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.89)
- GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding (CVPR 2024 · similarity 0.89)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.