Research map / Video / Video segmentation and tracking
Progressive Fourier Neural Representation for Sequential Video Compilation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Progressive Fourier Neural Representation (PFNR), which learns sparse Fourier-space subnetworks per video session and freezes them, enabling sequential compilation of neural video representations without forgetting.
Problem
Neural implicit representations (NeRV-style) encode each video with a one-to-one model and cannot accumulate or transfer representations across sequentially arriving videos without catastrophic forgetting.
Approach
Extends the weight-selection continual learner (WSN) with a Fourier Subnetwork Operator that splits weights into real and imaginary parts, learns per-session binary masks via learned weight scores (straight-through estimator), and freezes weights of finished sessions so previous videos decode losslessly while later sessions reuse them forward.
Key results
- On UVG8, PFNR with f-NeRV3 at c=50% reaches 32.12 average PSNR and 0.93 MS-SSIM with 0.0 BWT, versus 27.66 PSNR for WSN at the same capacity.
- On UVG17, PFNR f-NeRV3 (c=30%) reaches 31.72 average PSNR versus 26.80 for WSN, with zero backward transfer.
- PFNR outperforms the multi-task upper bound (MTL, 30.78 PSNR on UVG8) and all rehearsal and regularization baselines (EWC, iCaRL, ESMER).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use f-NeRV3 instead of f-NeRV2 in PFNR | PFNR · UVG8 | average PSNR: 29.82 -> 32.12 | Helped |
| Use f-NeRV3 instead of f-NeRV2 in PFNR | PFNR · UVG8 | average MS-SSIM: 0.90 -> 0.93 | Helped |
| Use f-NeRV3 instead of f-NeRV2 in PFNR | PFNR · UVG17 | average PSNR: 29.36 -> 31.72 | Helped |
| Use f-NeRV3 instead of f-NeRV2 in PFNR | PFNR · UVG17 | average MS-SSIM: 0.90 -> 0.94 | Helped |
| Replace WSN with PFNR f-NeRV3 | Continual video representation · UVG8 | average PSNR: 27.66 -> 32.12 | Helped |
| Replace WSN with PFNR f-NeRV3 | Continual video representation · UVG17 | average PSNR: 26.80 -> 31.72 | Helped |
| Replace WSN with PFNR f-NeRV3 | Continual video representation · UVG8 | average MS-SSIM: 0.85 -> 0.93 | Helped |
| Replace WSN with PFNR f-NeRV3 | Continual video representation · UVG17 | average MS-SSIM: 0.84 -> 0.94 | Helped |
Surprising findings
- PFNR outperforms the MTL upper bound of WSN, which the authors attribute to properly selected Fourier-space weights aiding generalization.
- Rehearsal baseline ESMER did not reduce forgetting in neural representations and was worse than iCaRL in this setting.
Limitations
- Parameters of the Fourier operator grow with layer depth and input/output size, making FSO layers large (e.g., 37.8M parameters in f-NeRV3 block).
- Requires session identity at training and test time (multi-head setting).
- Performance on DAVIS50 is noted as a limitation case to investigate, and reused weights shrink as sessions grow.
Experimental setup
Datasets: UVG8 UVG17 DAVIS50 Big Buck Bunny (scikit-video)
Benchmarks: UVG8 UVG17 DAVIS50
Metrics: PSNR MS-SSIM backward transfer capacity
Baselines: STL NeRV EWC iCaRL ESMER WSN MTL upper-bound
Builds on: NeRV WSN Lottery Ticket Hypothesis Fourier Neural Operator HNeRV
Closest papers by meaning
- Boosting Neural Representations for Videos with a Conditional Decoder (CVPR 2024 · similarity 0.86)
- DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes (CVPR 2024 · similarity 0.86)
- Learning Spatially Collaged Fourier Bases for Implicit Neural Representation (AAAI 2024 · similarity 0.86)
- PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos (CVPR 2024 · similarity 0.85)
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.84)
- MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.