Research map / Video / Video segmentation and tracking
Boosting Neural Representations for Videos with a Conditional Decoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a universal conditional-decoder framework (temporal-aware affine transform, sinusoidal NeRV-like blocks, extra reconstruction losses, and consistent entropy minimization) that boosts implicit neural video representations for regression, compression, inpainting and interpolation.
Problem
Existing implicit neural video representations (INRs) underuse their capacity because intermediate features are poorly aligned with the target frame, activations yield few diverse features, losses miss high-frequency detail, and compression pipelines use inconsistent entropy models between training
Approach
A temporal-aware affine transform (TAT) without normalization modulates intermediate features using frame-index-derived embeddings, combined with a sinusoidal NeRV-like block, a frequency-domain plus MS-SSIM+L1 loss, and a network-free Gaussian entropy model (CEM) that is consistent between training and inference.
Key results
- On UVG with 3M parameters, HNeRV-Boost reaches 33.89 dB vs 32.47 dB for HNeRV, and the average over sizes improves from 33.92 to 35.68 dB.
- On DAVIS disperse-mask inpainting, boosted versions improve average PSNR by 3.92 dB (NeRV), 2.43 dB (E-NeRV), and 4.2 dB (HNeRV).
- Boosted HNeRV with CEM surpasses H.265, H.264, DCVC and DCVC-TCM in PSNR across bitrates on UVG (1080p) rate-distortion curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Boost NeRV with the conditional decoder framework | NeRV · UVG average, 3M | PSNR: 31.11 -> 32.76 | Helped |
| Boost E-NeRV with the conditional decoder framework | E-NeRV · UVG average, 3M | PSNR: 31.51 -> 33.40 | Helped |
| Boost HNeRV with the conditional decoder framework | HNeRV · UVG average, 3M | PSNR: 32.47 -> 33.89 | Helped |
| Add TAT residual blocks (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 39.12 -> 41.09 | Helped |
| Use TAT instead of AdaIN (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 38.03 -> 41.09 | Helped |
| Use TAT instead of SAF (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 40.93 -> 41.09 | Helped |
| Use SINE instead of GELU activation (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 41.00 -> 41.09 | Helped |
| Use the full reconstruction loss instead of L2 loss (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 40.19 -> 41.09 | Helped |
| Use the full reconstruction loss instead of L1+SSIM loss (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 41.00 -> 41.09 | Helped |
| Use the full reconstruction loss instead of L1 loss (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 40.37 -> 41.09 | Helped |
| Use the full reconstruction loss instead of L1+MS-SSIM loss (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 40.49 -> 41.09 | Helped |
| Add MS-SSIM to L1+frequency loss (ablation) | HNeRV-Boost · Bunny (720p), 300 epochs | PSNR: 41.08 -> 41.09 | No clear effect |
Surprising findings
- Replacing GELU with SINE improves NeRV/E-NeRV/FFNeRV clearly (~0.7 dB) but only marginally for HNeRV and SNeRV, attributed to their more uniform parameter distributions.
- Boosted models at 300 epochs outperform baselines trained for 2400 epochs on the Bosphorus video.
Limitations
- Boosted decoders incur higher decoding latency and lower FPS than the baseline INRs.
- Results are per-video overfitting; the approach does not amortize across videos.
- Entropy model requires transmitting per-weight/embedding mean and variance metadata.
Experimental setup
Datasets: Bunny UVG DAVIS validation
Benchmarks: Bunny UVG DAVIS validation
Metrics: PSNR MS-SSIM bpp encoding time decoding time FPS
Baselines: NeRV E-NeRV HNeRV FFNeRV Gomes et al. entropy minimization Maiya et al. (COOL-CHIC-style entropy coding) H.264 H.265 DCVC DCVC-TCM
Builds on: NeRV HNeRV E-NeRV AdaIN FiLM-style conditional modulation
Closest papers by meaning
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.91)
- DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes (CVPR 2024 · similarity 0.90)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.88)
- SINR: Sparsity Driven Compressed Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- Bias for Action: Video Implicit Neural Representations with Bias Modulation (CVPR 2025 · similarity 0.87)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.