Research map / 3D vision / NeRF and radiance fields
Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neural Directional Encoding (NDE), a learnable, spatially varying view-dependent encoding for NeRF that combines a cubemap feature grid for far-field reflections with cone-traced tri-plane near-field features for interreflections, enabling accurate specular rendering with tiny MLPs.
Problem
Novel-view synthesis of specular objects requires modeling high-frequency view-dependent appearance and near-field interreflections, which analytical directional encodings with large MLPs handle poorly and slowly.
Approach
Brings feature-grid encoding into the directional domain (mip-mapped cubemap of learnable features for far-field reflections) and parameterizes near-field interreflections as a spatio-spatial function via cone-tracing a mip-mapped spatial feature volume along the reflected direction, so the directional encoding depends on position.
Key results
- NDE reaches 37.19 mean PSNR on synthetic specular scenes, versus 35.88 for Ref-NeRF and 34.95 for ENVIDR, with LPIPS 0.022.
- NDE runs at 3.03 FPS with 75k color-MLP parameters at 800x800, and the real-time NDE-RT variant reaches 66 FPS in WebGL.
- Ablation: cubemap far-field plus cone-traced near-field features give 31.53 PSNR on Materials versus 28.54 for analytical encoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Neural Directional Encoding instead of the compared view-dependent appearance methods | NeRF specular-object view synthesis · Synthetic scenes (mean) | PSNR: 35.88 -> 37.19 | Helped |
| Use Neural Directional Encoding instead of the compared view-dependent appearance methods | NeRF specular-object view synthesis · Synthetic scenes (mean) | LPIPS: 0.053 -> 0.022 | Helped |
| Add cubemap far-field features rather than analytical encoding (ablation) | NDE · Materials | PSNR: 28.54 -> 30.27 | Helped |
| Add cone-traced near-field features to cubemap far-field features (ablation) | NDE · Materials | PSNR: 30.27 -> 31.53 | Helped |
| Replace volume-rendered near-field features with cone-traced features (ablation) | NDE · Materials | PSNR: 29.31 -> 31.53 | Helped |
| Reduce decoder MLP width from 64 to 32 (ablation) | NDE offline · Synthetic scenes (mean) | PSNR: 37.19 -> 36.69 | Hurt |
| Reduce decoder MLP width from 64 to 16 (ablation) | NDE offline · Synthetic scenes (mean) | PSNR: 37.19 -> 36.23 | Hurt |
| Reduce decoder MLP width from 64 to 32 (ablation) | NDE-RT · Synthetic scenes (mean) | PSNR: 35.48 -> 33.97 | Hurt |
| Reduce decoder MLP width from 64 to 16 (ablation) | NDE-RT · Synthetic scenes (mean) | FPS: 66 -> 331 | Helped |
| Replace mip-mapped tri-plane near-field encoding with mip-mapped hash grid (ablation) | NDE · Synthetic scenes (mean) | PSNR: 37.19 -> 36.52 | Hurt |
Surprising findings
- Tiny MLPs (2 layers, width 64) with the cubemap feature encoding match large IDE MLPs (8x256) in mirror reflections, where IDE fails at small size.
Limitations
- SDF geometry relies on positional-encoded MLPs, leading to long training times (10-18 h per scene).
- Sensitive to surface normal quality; hash-grid geometry produces corrupted normals.
- Difficult to model transparent objects and editability is limited.
Experimental setup
Datasets: Shiny Blender NeRF Synthetic (Materials scene) NeRO real scenes
Benchmarks: Shiny Blender NeRF Synthetic Materials NeRO real scenes
Metrics: PSNR SSIM LPIPS FPS #Params
Baselines: NeRO ENVIDR Ref-NeRF 3DGS IDE
Builds on: NeRF Ref-NeRF VolSDF NerfAcc Tri-MipRF
Closest papers by meaning
- SpecNeRF: Gaussian Directional Encoding for Specular Reflections (CVPR 2024 · similarity 0.92)
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (CVPR 2022 · similarity 0.91)
- NeRFLight: Fast and Light Neural Radiance Fields Using a Shared Feature Grid (CVPR 2023 · similarity 0.89)
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields (AAAI 2024 · similarity 0.88)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.88)
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.