Research map / 3D vision / NeRF and radiance fields
SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpectralNeRF, a NeRF-based model that renders per-wavelength spectrum maps (SpectralMLP) and fuses them into white-light RGB images via an attention UNet (SAUNet).
Problem
NeRF-based novel view synthesis struggles with complex scenes and textures, while physically based spectral rendering produces only single views and is slow on complex geometry; the paper seeks to combine spectral rendering physics with NeRF.
Approach
Reformulates spectral rendering (spectral power distribution to XYZ to RGB) as two steps: a SpectralMLP builds a spectral radiance field that volume-renders N spectrum maps across 380-780 nm, and a Spectrum Attention UNet (SAUNet) with attention gates and channel attention combines them into RGB, with a PSNR-weighted spectral loss.
Key results
- On 8 synthetic scenes SpectralNeRF achieves best PSNR on most scenes, e.g. Spaceship 31.951 vs Mip-NeRF 31.495 and Kitchen 35.115 vs 34.583 for NeRF; authors report ~5% average improvement over NeRF-based methods.
- On real-world scenes, Projector PSNR 31.2535 vs Aug-NeRF 30.0795; Dog doll PSNR 25.1257 vs 22.5040 for NeRF, though LPIPS is worse on Projector (0.0605 vs 0.0354 for Aug-NeRF).
- Ablation: adding 11 spectrum maps raises PSNR from 30.126 (vanilla NeRF) to 31.375; full model with attention gate and SA in first two encoders reaches 31.951.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spectral radiance fields using 11 spectrum maps (ablation) | SpectralNeRF on Spaceship · Spaceship | PSNR: 30.126 -> 31.375 | Helped |
| Add the Attention Gate (ablation) | SpectralNeRF on Spaceship · Spaceship | PSNR: 31.875 -> 31.951 | Helped |
| Add weighted spectrum-map loss weights w_s (ablation) | SpectralNeRF on Spaceship · Spaceship | PSNR: 31.601 -> 31.951 | Helped |
| Place Spectrum Attention in encoder blocks E1+E2 rather than E1 alone (ablation) | SpectralNeRF on Spaceship · Spaceship | PSNR: 31.552 -> 31.723 | Helped |
| Replace SAUNet with a fully connected layer (ablation) | SpectralNeRF · Rover car, Digger, Vintage car, Cartoon knight | PSNR: 30.086 -> 27.309; 33.378 -> 30.617; 34.480 -> 33.382; 34.915 -> 34.413 | Hurt |
| Transfer the spectral pipeline and SAUNet to Aug-NeRF (S-Aug-NeRF) (ablation) | Aug-NeRF · Digger (PNG), Vintage car, Kitchen, Projector | PSNR: +0.127; +0.491; +0.494; +2.305 | Helped |
| Reduce the number of spectrum maps from 11 to 6 (ablation) | SpectralNeRF · Synthetic datasets | PSNR | Hurt |
| Increase the number of spectrum maps from 11 to 21 (ablation) | SpectralNeRF · Synthetic datasets | PSNR | No clear effect |
| Use three RGB output channels and remove the spectral loss (ablation) | SpectralNeRF · Synthetic datasets | PSNR | Hurt |
Surprising findings
- Replacing SAUNet with a simple fully connected layer drops mean PSNR from 33.214 to 31.430 on the reported scenes, showing the fusion network matters more than expected.
- On the Projector real scene, SpectralNeRF has worse LPIPS (0.0605) than Aug-NeRF (0.0354), attributed to slightly blurry SAUNet outputs.
Limitations
- Requires many spectrum maps (11 synthetic, 8 real) which increases rendering cost; slightly slower than NeRF.
- SAUNet outputs can be blurry, giving worse LPIPS on some real scenes.
- Real-world data capture uses color absorbers over a camera lens, limiting scale and scene diversity.
Experimental setup
Datasets: 8 rendered synthetic spectral scenes (Mitsuba) 2 real-world captured scenes (Projector, Dog doll)
Benchmarks: 8 synthetic spectral scenes 2 real-world forward-facing scenes
Metrics: PSNR SSIM LPIPS L1 distance inference time
Baselines: NeRF Mip-NeRF Aug-NeRF
Builds on: NeRF Mip-NeRF Aug-NeRF Attention Gate CIE color matching functions
Closest papers by meaning
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.89)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.88)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.88)
- LAENeRF: Local Appearance Editing for Neural Radiance Fields (CVPR 2024 · similarity 0.88)
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields (AAAI 2024 · similarity 0.88)
- Bright-NeRF: Brightening Neural Radiance Field with Color Restoration from Low-Light RAW Images (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.