Research map / Video / Video segmentation and tracking
HyperNVD: Accelerating Neural Video Decomposition via Hypernetworks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperNVD, a hypernetwork that maps a frozen VideoMAE-based video embedding to the weights and hash encodings of an INR layered video decomposition model, enabling faster convergence on unseen videos.
Problem
Layer-based neural video decomposition uses per-video implicit neural representations that must be trained from scratch for each video, which is slow (tens of minutes per 480p video) and lacks generalization.
Approach
A hypernetwork conditioned on a compressed VideoMAE embedding generates the weights and multiresolution hash encoding of the NVD model, trained over multiple videos; its output is used to initialize fine-tuning on new videos, shortening convergence.
Key results
- Fine-tuning from the 15-video metamodel improves PSNR by 0.25 to 1.65 dB over training from scratch on 10 unseen DAVIS videos and reaches the same PSNR more than 30 minutes faster.
- Single-video HyperNVD matches the Hashing-nvd baseline (e.g., 30.06 vs 29.12 PSNR on hike, 34.94 vs 35.43 on car-turn), without degrading quality.
- Scaling training from 1 to 30 videos lowers PSNR by roughly 3 dB, with a performance plateau.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize fine-tuning from the 15-video HyperNVD metamodel instead of training from scratch | NVD model · DAVIS unseen videos | PSNR: +1.36, +1.65, +1.46, +0.93, +0.89, +0.72, +0.72, +0.54, +0.48, +0.25 | Helped |
| Use VideoMAE-generated embedding rather than a jointly optimized learnable embedding (ablation) | HyperNVD · DAVIS (3-video training), hike PSNR | PSNR: 26.88 -> 27.31 | Helped |
| Use video-specific MRHE parameters generated by the hypernet rather than universal MRHE parameters (ablation) | HyperNVD · DAVIS (3-video training), car-turn PSNR | PSNR: 30.01 -> 31.87 | Helped |
| Use ground-truth masks rather than MaskRCNN masks as initial guidance (ablation) | HyperNVD · DAVIS hike | PSNR: 29.61 -> 30.06 | Mixed |
| Train with multiple videos rather than one video (ablation) | HyperNVD · DAVIS hike | PSNR: 30.06; 27.24 | Hurt |
| Initialize the NVD model with HyperNVD weights rather than random parameters | NVD model on unseen videos · Unseen videos (Figure 1) | training time to reach the same PSNR: more than 30 minutes less | Helped |
Surprising findings
- Training on more videos causes about a 3 dB PSNR drop relative to single-video training, indicating a capacity or generalization trade-off.
Limitations
- Multi-video training lowers reconstruction quality compared with single-video models, losing high-frequency detail.
- Hypernet has about 290M parameters and requires about 22GB GPU memory per video, which is heavy.
- Experiments are limited to 16-frame clips at 768x432 on DAVIS.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: PSNR SSIM
Baselines: Deformable Sprites Layered Neural Atlases Hashing-nvd CoDeF training from scratch
Builds on: Hashing-nvd Layered Neural Atlases VideoMAE HyP-NeRF RAFT
Closest papers by meaning
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.85)
- MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance (AAAI 2025 · similarity 0.84)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.84)
- Boosting Neural Representations for Videos with a Conditional Decoder (CVPR 2024 · similarity 0.84)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.84)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.