Research map / Video / Video segmentation and tracking
PNVC: Towards Practical INR-based Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PNVC, a practical INR-based video codec combining autoencoder-style pretraining with per-frame overfitting, supporting low-delay and random-access configurations with ~20+ FPS 1080p decoding.
Problem
INR-based neural video codecs overfit whole sequences with a single model, causing high system latency and incompatibility with LD/RA coding configurations, while autoencoder codecs have high decoder complexity.
Approach
Hybrid of autoencoder-based hierarchical latents and INR decoder, pretrained offline then overfitted per GOP/frame; introduces ModMixer, a reparameterized FC block with a token mixer degenerated during pretraining so it adds no inference cost, plus hierarchical quality control, modulation-based entropy model, and scale-aware hierarchical positional encoding.
Key results
- PNVC (LD) achieves -34.35% BD-rate (PSNR) vs HM 18.0 LD on UVG, and -58.97% in MS-SSIM.
- PNVC outperforms HiNeRV on UVG by roughly 9 points in PSNR BD-rate (-34.35% vs -25.42%) while supporting LD and RA modes.
- PNVC decodes at 25.3 FPS (LD) on 1080p content with about 101.1 kMACs/pixel decoding complexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reparameterization during pretraining (ablation) | PNVC · UVG and MCL-JCV | BD-rate: 6.45% (UVG); 6.34% (MCL-JCV) coding loss when removed | Helped |
| Use the overparameterized basis during online overfitting (ablation) | PNVC · UVG and MCL-JCV | BD-rate: 3.43% (UVG); 3.25% (MCL-JCV) coding loss when disregarded | Helped |
| Use reparameterization for fully connected layers (ablation) | PNVC · UVG and MCL-JCV | BD-rate: 2.22% (UVG); 2.36% (MCL-JCV) coding loss when removed | Helped |
| Add the degenerated token mixer (ablation) | PNVC · UVG and MCL-JCV | BD-rate: 4.57% (UVG); 4.49% (MCL-JCV) coding loss when removed | Helped |
| Use adaptive quality parameters in LD mode (ablation) | PNVC · UVG and MCL-JCV, LD | BD-rate: 1.21% (UVG); 1.22% (MCL-JCV) coding loss when removed | Helped |
| Use adaptive quality parameters in RA mode (ablation) | PNVC · UVG and MCL-JCV, RA | BD-rate: 2.25% (UVG); 2.23% (MCL-JCV) coding loss when removed | Helped |
| Add online overfitting in LD mode (ablation) | PNVC · UVG and MCL-JCV, LD | BD-rate: 7.95% (UVG); 7.95% (MCL-JCV) coding loss when removed | Helped |
| Add online overfitting in RA mode (ablation) | PNVC · UVG and MCL-JCV, RA | BD-rate: 9.47% (UVG); 9.34% (MCL-JCV) coding loss when removed | Helped |
| Use uneven channel grouping (ablation) | PNVC entropy model · UVG and MCL-JCV | BD-rate: 1.76% (UVG); 1.91% (MCL-JCV) coding loss when removed | Helped |
| Use temporal feature modulation instead of concatenation (ablation) | PNVC entropy model · UVG and MCL-JCV | BD-rate: 3.39% (UVG); 3.57% (MCL-JCV) coding loss with concatenation | Helped |
| Use scaled hierarchical gridding (ablation) | PNVC decoder · UVG and MCL-JCV | BD-rate: 2.39% (UVG); 2.36% (MCL-JCV) coding loss with original gridding | Helped |
| Add rescaling feature gamma (ablation) | PNVC decoder · UVG and MCL-JCV | BD-rate: 0.75% (UVG); 0.76% (MCL-JCV) coding loss when removed | Helped |
Surprising findings
- Encoding speed of PNVC (about 0.014 FPS) is much slower than autoencoder-based codecs and VTM, limiting real-time use.
Limitations
- Encoding speed is very slow, limiting real-time applications such as video conferencing.
- Relies on per-frame overfitting at encoding time, which is computationally costly.
- Evaluated only on 1080p datasets (UVG, MCL-JCV) with the Vimeo-90k pretraining set.
Experimental setup
Datasets: Vimeo-90k
Benchmarks: UVG MCL-JCV
Metrics: BD-rate (PSNR) BD-rate (MS-SSIM) FPS kMACs/pixel parameters
Baselines: HM 18.0 VTM 20.0 VCT DCVC-HEM DCVC-DC FFNeRV C3 HNeRV-Boost HiNeRV
Builds on: HiNeRV ELIC DCVC quadtree-based entropy model structural reparameterization
Closest papers by meaning
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.92)
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.90)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.90)
- On Quantizing Neural Representation for Variable-Rate Video Coding (ICLR 2025 · similarity 0.88)
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.87)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.