Research map / 3D vision / NeRF and radiance fields
ResFields: Residual Neural Fields for Spatiotemporal Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ResFields, which replaces MLP linear layers with time-conditioned residual weight layers factorized as shared low-rank spanning set plus per-time coefficients, improving neural fields for video, temporal SDF, dynamic NeRF, and scene flow without widening the MLP.
Problem
Neural fields built from MLPs have limited capacity for large, complex temporal (spatiotemporal) signals, and increasing width slows inference/training and memory use, while spatial/temporal partitioning hurts generalization.
Approach
Replace each linear layer's weights W_i with W_i + W_i(t), where the time-dependent residual is a low-rank factorization: a globally shared spanning set M (R x N x M) combined with per-frame coefficients v(t) interpolated over time. This adds capacity while keeping MLP size and preserving implicit regularization and generalization.
Key results
- On 2D video approximation, Siren-512+ResFields reaches 39.21 test PSNR vs 36.37 for Siren-1024 and 34.52 for Instant NGP, with about 2.5x faster training and 30% less GPU memory than Siren-1024.
- On Owlii dynamic radiance fields, ResFields improves every baseline; e.g., TNeRF+ResFields (i=1,...,7) reaches 27.55 mean PSNR vs 26.18 for TNeRF with similar FPS.
- For temporal SDFs, Siren-128 with ResFields (rank 40) achieves 8.16 mean CD vs 9.04 for the larger Siren-256 baseline, while running about 2x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rank-10 ResFields to Siren-512 hidden layers | Siren-512 · 2D video approximation (two videos, 10% unseen pixels) | test PSNR: 31.89 -> 39.21 | Helped |
| Add rank-10 ResFields to the temporal SDF model | Siren-128 · Temporal SDF (five-sequence mean) | CD: 15.063 -> 8.158 | Helped |
| Add rank-10 ResFields to existing dynamic NeRF methods | TNeRF, DyNeRF, DNeRF, Nerfies, HyperNeRF, and NDR · Owlii dynamic NeRF (mean) | PSNR | Helped |
| Add ResFields to all middle layers | TNeRF · Three-view RGBD capture (mean) | LPIPS: 0.234 -> 0.203 | Helped |
| Add rank-10 ResFields | ReLU MLP scene-flow models · Scene flow (offset, SE(3), and DCT) | forward L1: 6.88 -> 3.85; 4.57 -> 2.64; 3.19 -> 2.18 | Helped |
| Use residual weights on layer outputs instead of adding them to layer weights (ablation) | Siren-512 video approximation · 2D video approximation | test PSNR: 39.21 -> 32.84 | Hurt |
| Multiply base weights by time-dependent weights (ablation) | Siren-512 video approximation · 2D video approximation | test PSNR: 39.21 -> 32.65 | Hurt |
| Replace base weights with time-dependent weights (ablation) | Siren-512 video approximation · 2D video approximation | test PSNR: 39.21 -> 35.17 | Hurt |
| Factorize the time-dependent residual weights instead of using unconstrained independent weights (ablation) | ResFields · 2D video approximation (30% unseen pixels) | test PSNR: 38.52 -> 41.69 | Helped |
| Use fewer coefficient factors for temporal interpolation (ablation) | Siren-512+ResFields · 2D video approximation (temporal interpolation) | test PSNR: 21.61 -> 27.34 | Mixed |
| Use multiple ResField layers rather than one higher-rank layer at equal parameter count (ablation) | Siren video approximation · 2D video approximation (equal parameter budget) | test PSNR: 37.53 -> 38.01; 38.75 -> 39.86; 39.33 -> 40.62; 39.67 -> 41.20 | Helped |
| Increase the number of ResField layers from one to three (ablation) | TNeRF · Owlii dynamic NeRF (mean) | PSNR: 94.99 -> 95.21 | Helped |
Surprising findings
- Siren-128 with ResFields (rank 40) outperforms the larger vanilla Siren-256 on temporal SDFs while being about twice as fast.
Limitations
- Does not help ill-posed monocular reconstruction where the bottleneck is lack of constraints rather than capacity.
- Per-frame independent coefficients do not support time interpolation; interpolation requires fewer factors and reduces training PSNR.
Experimental setup
Datasets: Owlii Deforming Things ReSynth bikes video cat video
Benchmarks: Owlii Deforming Things ReSynth
Metrics: PSNR SSIM LPIPS Chamfer distance normal consistency l1 flow error training it/s GPU memory
Baselines: Instant NGP Siren-1024 Siren-256 TNeRF DyNeRF DNeRF Nerfies HyperNeRF NDR HexPlane Tensor4D NeuS2
Builds on: SIREN VolSDF TNeRF Level-of-Experts
Closest papers by meaning
- How to Train Neural Field Representations: A Comprehensive Study and Benchmark (CVPR 2024 · similarity 0.87)
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.87)
- BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling (AAAI 2024 · similarity 0.87)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.87)
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.86)
- DUDF: Differentiable Unsigned Distance Fields with Hyperbolic Scaling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.