Research map / 3D vision / NeRF and radiance fields
SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpikeNeRF, which learns a NeRF volumetric scene representation purely from continuous spike camera streams using an integrate-and-fire spiking neuron layer with threshold variation simulation and a long-term spike rendering loss.
Problem
Reconstructing dense, photorealistic volumetric 3D scene representations from spike camera streams under noisy, varying illumination, where prior spike methods assume ideal lighting and no NeRF-based approach exists for spike data.
Approach
Supervise a NeRF MLP directly in the spike domain: a differentiable integrate-and-fire spiking neuron layer after the MLP generates spike streams with pixel-wise threshold variation, and a long-term spike rate rendering loss compares them with the recorded spikes, trained via BPTT with a surrogate gradient.
Key results
- On the synthetic dataset, SpikeNeRF reaches 20.78 PSNR (static, full image) versus 16.21 for the best TFP(32)+NeRF baseline and 13.41 for Spk2img+NeRF.
- On real spike data, SpikeNeRF achieves BRISQUE 35.94 versus 42.64 for TFP+NeRF and 47.92 for TFI+NeRF (lower is better).
- Under varying light intensities (16/32/64) on synthetic data, SpikeNeRF maintains PSNR of 18.72 to 23.89 while baselines degrade at low light.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use spike-based NeRF supervision instead of TFP (32) image reconstruction followed by NeRF | SpikeNeRF · Synthetic dataset | PSNR: 16.21/16.63 -> 20.78/22.07 | Helped |
| use spike-based NeRF supervision instead of TFP (256) image reconstruction followed by NeRF | SpikeNeRF · Synthetic dataset | PSNR: 15.07/19.24 -> 20.78/22.07 | Helped |
| use spike-based NeRF supervision instead of TFI image reconstruction followed by NeRF | SpikeNeRF · Synthetic dataset | PSNR: 14.93/19.34 -> 20.78/22.07 | Helped |
| use spike-based NeRF supervision instead of Spk2img reconstruction followed by NeRF | SpikeNeRF · Synthetic dataset | PSNR: 13.41/14.05 -> 20.78/22.07 | Helped |
| use SpikeNeRF rather than TFP + NeRF | NeRF on real-world spike data · Real-world data | BRISQUE: 42.64 -> 35.94 | Helped |
| use SpikeNeRF rather than TFP + NeRF | NeRF on real-world spike data · Real-world data | NIQE: 6.16 -> 6.09 | Helped |
| use Ls instead of Li (ablation) | SpikeNeRF · Synthetic dataset average | PSNR: 17.59/18.77 -> 18.44/19.09 | Helped |
| add Li to Ls (ablation) | SpikeNeRF · Synthetic dataset average | PSNR: 18.44/19.09 -> 18.05/19.00 | Hurt |
| replace Ls with L*i (ablation) | SpikeNeRF · Synthetic dataset average | PSNR: 18.44/19.09 -> 13.76/16.37 | Hurt |
Surprising findings
- Two-stage pipelines (spike reconstruction then NeRF) degrade sharply in low-light scenes, and the learned Spk2ImgNet baseline performs poorly on noisy spikes outside ideal illumination.
Limitations
- Optimization takes about 20 hours per scene on a single NVIDIA 3090 GPU.
- Real-world poses are estimated with COLMAP from images reconstructed from spikes, and real data lacks ground truth so evaluation relies on no-reference metrics.
- Requires a calibrated nonuniformity matrix captured from a uniform light scene.
Experimental setup
Datasets: NeRF synthetic Blender scenes (chair, ficus, hotdog, lego, materials, mic) Real-world spike camera recordings
Benchmarks: NeRF synthetic spike dataset Real-world spike camera recordings
Metrics: PSNR SSIM LPIPS BRISQUE NIQE
Baselines: TFI+NeRF TFP+NeRF Spk2img+NeRF Image+NeRF EventNeRF
Builds on: NeRF Integrate-and-Fire neuron model Spike simulator (Zhu et al. 2023) Spk2ImgNet Temporal Filtering Interval (TFI)
Closest papers by meaning
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.89)
- AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization (AAAI 2024 · similarity 0.88)
- USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting (CVPR 2025 · similarity 0.88)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.88)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.88)
- Finding Visual Saliency in Continuous Spike Stream (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.