Research map / Video / Video segmentation and tracking
Neural Video Compression with Feature Modulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DCVC-FM, a conditional-coding neural video codec that uses learnable quantization-scaler feature modulation for a wide single-model quality range and periodic temporal feature refresh for long prediction chains, beating VTM and ECM under intra-period -1.
Problem
Neural video codecs (NVCs) lack a wide quality range in a single model (prior ~3.8 dB) and degrade badly over long prediction chains (e.g., intra-period -1), blocking practical use.
Approach
Modulate the current frame's latent feature with a learnable global and spatial-channel quantization scaler, trained with uniform sampling of quantization parameter and lambda over a wide range; periodically refresh the temporal feature from the reconstructed pixel frame, and train on longer videos.
Key results
- Under intra-period -1 (all frames, YUV420), DCVC-FM achieves 25.5% BD-rate saving over VTM-17.0 while DCVC-DC is 12.8% worse than VTM.
- DCVC-FM achieves 29.7% average bitrate saving over DCVC-DC with 16% MACs reduction (2642G to 2225G).
- Quality range per model expands from 3.8 dB (DCVC-DC) to 11.4 dB on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCVC-FM instead of ECM-11.0 | Video compression on all frames in YUV420 with intra-period –1 · Average across UVG, MCL-JCV, HEVC B–E | BD-Rate (%): –20.5 -> –25.5 | Helped |
| Use DCVC-FM instead of DCVC-DC | Video compression on all frames in YUV420 with intra-period –1 · Average across UVG, MCL-JCV, HEVC B–E | BD-Rate (%): 29.7% bitrate saving | Helped |
| Use DCVC-FM instead of DCVC-DC | Single-model quality adjustment · YUV420, all frames, intra-period –1 | PSNR range (dB): 3.8 -> 11.4 | Helped |
| Add structure optimization (ablation) | DCVC-FM · All frames, YUV420, intra-period –1 | BD-Rate (%): 0.8 | Hurt |
| Add wider quality range support (ablation) | DCVC-FM · All frames, YUV420, intra-period –1 | BD-Rate (%): 3.4 | Hurt |
| Add single-model support for RGB and YUV (ablation) | DCVC-FM · All frames, YUV420, intra-period –1 | BD-Rate (%): 4.8 | Hurt |
| Train with longer video (ablation) | DCVC-FM · All frames, YUV420, intra-period –1 | BD-Rate (%): –15.5 | Helped |
| Add periodic temporal feature refresh (ablation) | DCVC-FM · All frames, YUV420, intra-period –1 | BD-Rate (%): –29.7 | Helped |
Surprising findings
- DCVC-DC shows severe quality degradation under intra-period -1, becoming 12.8% worse than VTM on all frames, with 84.4% bitrate increase on HEVC E.
- Naive fp16 inference causes 87.3% BD-rate increase due to grid_sample absolute-coordinate precision; a relative-offset reimplementation reduces this to 0.9%.
Limitations
- Inference is still far from real-time speed.
- Floating-point inference has cross-platform issues for entropy coding.
- Rate control is only demonstrated with a simple buffer-fullness heuristic, not a dedicated algorithm.
Experimental setup
Datasets: Vimeo-90k Vimeo raw videos HEVC B-E UVG MCL-JCV
Benchmarks: HEVC B HEVC C HEVC D HEVC E UVG MCL-JCV
Metrics: BD-Rate (%) PSNR (dB) MACs encoding/decoding time quality range (dB)
Baselines: DCVC-DC DCVC-HEM DCVC-TCM DCVC CANF-VC H.266/VTM-17.0 H.265/HM-16.25 ECM-5.0 ECM-11.0
Builds on: DCVC-DC DCVC-TCM DCVC-HEM hyperprior
Closest papers by meaning
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.91)
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.91)
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.90)
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.90)
- On Quantizing Neural Representation for Variable-Rate Video Coding (ICLR 2025 · similarity 0.89)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.