Research map / 3D vision / NeRF and radiance fields
CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CodecNeRF, an encoder-decoder-finetuning neural codec that generates a triplane NeRF in one forward pass, then compresses per-instance finetuning deltas via low-rank tensor factorization, LoRA on the MLP, and entropy coding.
Problem
NeRF representations are slow to encode via per-scene optimization and large in size, and feed-forward NeRF encoders lack compression and render quality; a practical NeRF codec needs fast encoding/decoding, compact codes, and high-quality rendering.
Approach
A DINO-feature unprojection encoder builds a 3D feature volume that is vector-quantized as three axis-aligned 2D planes and decoded into multi-resolution triplanes with a 3D-aware 2D convolution; the instance is then adapted at test time with low-rank tensor-factorized triplane deltas plus LoRA on the MLP, and the deltas are entropy-coded.
Key results
- On Objaverse, PEFT++ reaches 27.35 PSNR at 1000 iterations at 0.179 MB, versus 25.27 PSNR for Triplanes at 8.077 MB, roughly 45x smaller.
- Initialization on GSO: Ours 24.49 PSNR vs GeoLRM 23.73 PSNR, with a 0.015 MB code.
- On DTU, Ours (PEFT++) reaches 20.07 PSNR at 0.160 MB in 58.8 s, versus TensoRF 14.58 PSNR and 3D-GS 15.77 PSNR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CodecNeRF (PEFT) instead of per-scene Triplanes | NeRF novel-view synthesis · Objaverse, 2000 iterations | PSNR: 26.56 -> 28.57 | Helped |
| Use CodecNeRF (PEFT) instead of per-scene Triplanes | NeRF novel-view synthesis · GSO, 2000 iterations | PSNR: 31.54 -> 34.65 | Helped |
| Apply entropy coding to finetuning deltas | CodecNeRF · Objaverse, 2000 iterations | SIZE (MB): 1.024 -> 0.146 | Helped |
| Apply entropy coding to finetuning deltas | CodecNeRF · GSO, 2000 iterations | PSNR: 34.65 -> 33.87 | Hurt |
| Use CodecNeRF instead of GeoLRM | GSO novel-view synthesis initialization · GSO initialization | PSNR: 23.73 -> 24.49 | Helped |
| Use CodecNeRF (PEFT) instead of TensoRF | DTU novel-view synthesis · DTU | PSNR: 14.58 -> 20.05 | Helped |
| Use CodecNeRF (PEFT) instead of 3D-GS | DTU novel-view synthesis · DTU | Train (s): 64.0 -> 41.3 | Helped |
| Increase tensor decomposition rank from 1 to 4 (ablation) | CodecNeRF (PEFT) · Objaverse, 2000 iterations | PSNR: 29.07 -> 29.72 | Helped |
| Increase tensor decomposition rank from 4 to 8 (ablation) | CodecNeRF (PEFT) · Objaverse, 2000 iterations | PSNR: 29.72 -> 29.98 | Helped |
| Apply entropy coding to decoder LoRA weights (ablation) | CodecNeRF (PEFT++) · Objaverse, 2000 iterations | PSNR: 29.46 -> 29.43 | No clear effect |
| Add decoder LoRA finetuning to feature-map finetuning (ablation) | CodecNeRF (PEFT) · Objaverse, 2000 iterations | PSNR: 27.50 -> 29.74 | Helped |
| Add feature-map finetuning to decoder LoRA finetuning (ablation) | CodecNeRF (PEFT) · Objaverse, 2000 iterations | PSNR: 27.05 -> 29.74 | Helped |
Surprising findings
- Entropy coding of the LoRA MLP weights gave negligible PSNR change in the ablation (27.04 vs 27.00 at 500 iterations), suggesting little cost to compressing decoder weights.
Limitations
- Evaluated only on object-level and small-scene datasets; large-scale scenes such as Mip-NeRF 360 are not handled.
- Supports only the triplane-based representation; extending to instant-NGP or 3D Gaussian Splatting requires architectural changes.
- Feed-forward initialization lacks fine detail compared with GeoLRM, as the bottleneck code is small.
Experimental setup
Datasets: Objaverse Google Scanned Objects DTU ShapeNet
Benchmarks: Objaverse Google Scanned Objects DTU ShapeNet
Metrics: PSNR SSIM MS-SSIM model size (MB) training time (s)
Baselines: Triplanes (K-Planes-based) GeoLRM TensoRF 3D-GS DVGO Plenoxels CCNeRF MaskDWT VQRF BiRF MetaINR
Builds on: K-Planes LoRA NeRF DINO CompressAI
Closest papers by meaning
- NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation (CVPR 2024 · similarity 0.94)
- Neural NeRF Compression (ICML 2024 · similarity 0.92)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.90)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.89)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.89)
- ZeroRF: Fast Sparse View 360deg Reconstruction with Zero Pretraining (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.