Research map / Video / Video segmentation and tracking
Deep Hierarchical Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DHVC, a hierarchical probabilistic predictive video codec built on ResNet-style hierarchical VAEs that conditions multiscale latents on temporal and spatial priors, achieving better R-D performance and much faster coding than VCT and DCVC.
Problem
Learned video compression via latent probabilistic predictive coding uses single-scale latents (1/16 resolution), which limits conditional probability estimation accuracy and requires complex prediction networks.
Approach
Hierarchical VAEs with multiscale latents (1/64 to 1/8 resolution) whose priors combine same-scale temporal priors from previous frames with spatial priors from coarser scales, plus a spatial-temporal prediction module and in-loop decoding fusion, removing the need for spatial autoregression.
Key results
- DHVC outperforms learned codecs (DVC-Pro, MLVC, RLVC, DCVC, VCT) across UVG, MCL-JCV and HEVC classes in R-D curves, and beats HM-16.26 on MCL-JCV.
- Encoding/decoding takes 0.25s/0.21s per 1080p frame versus DCVC 13.78s/46.48s and VCT 1.64s/1.58s, with peak memory 4.27G versus 11.71G (DCVC) and 10.51G (VCT).
- Adding in-loop decoding fusion gives an averaged ~1 dB PSNR gain in the ablation on UVG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hierarchical multiscale probabilistic prediction instead of single-scale prediction | DHVC · UVG, MCL-JCV, HEVC Class B/C/D/E | rate-distortion performance | Helped |
| add temporal probabilistic prediction (ablation) | DHVC baseline · UVG | PSNR | Helped |
| add in-loop decoding fusion (ablation) | DHVC with temporal prediction · UVG | PSNR: 1 dB | Helped |
| fine-tune with five successive frames (ablation) | DHVC with temporal prediction and decoding fusion · UVG | rate-distortion performance | Helped |
| use DHVC rather than VCT | learned video codecs · 1080p video evaluation | encoding time (s): 1.64 -> 0.25 | Helped |
| use DHVC rather than VCT | learned video codecs · 1080p video evaluation | decoding time (s): 1.58 -> 0.21 | Helped |
| use DHVC rather than VCT | learned video codecs · 1080p video evaluation | peak memory (G): 10.51 -> 4.27 | Helped |
| use DHVC rather than DCVC | learned video codecs · 1080p video evaluation | encoding time (s): 13.78 -> 0.25 | Helped |
| use DHVC rather than DCVC | learned video codecs · 1080p video evaluation | decoding time (s): 46.48 -> 0.21 | Helped |
| use DHVC rather than DCVC | learned video codecs · 1080p video evaluation | peak memory (G): 11.71 -> 4.27 | Helped |
| use latent-space probabilistic prediction rather than motion alignment | DHVC · synthetic pixel-shifted videos | PSNR | Hurt |
Surprising findings
- Parameter count (112M) is larger than VCT (188M is larger; DHVC is 112M) and DCVC (7.6M), yet DHVC is much faster and uses less memory, indicating model size is not closely tied to runtime.
- On low-resolution HEVC Class C/D/E, the learned codecs still trail traditional codecs, and VCT performs even worse than DVC-Pro at mid-to-high bitrates.
Limitations
- Performance gap to traditional codecs remains on low-resolution HEVC Class C/D/E sequences due to downsampled latents.
- Pixel-shift content is handled worse than DCVC because no explicit motion alignment is used.
- Improvements from optimization tools (feature fusion, post-processing, bitrate allocation) were not integrated.
Experimental setup
Datasets: Vimeo-90K UVG MCL-JCV HEVC Class B HEVC Class C HEVC Class D HEVC Class E
Benchmarks: UVG MCL-JCV HEVC Class B HEVC Class C HEVC Class D HEVC Class E
Metrics: PSNR bpp encoding time decoding time peak memory parameters multiply-add operations per pixel
Baselines: x265 HM-16.26 DVC-Pro MLVC RLVC DCVC VCT
Builds on: ResNet VAE VCT hyperprior VAE ConvNeXt
Closest papers by meaning
- Hierarchical Quantized Autoencoders (NeurIPS 2020 · similarity 0.88)
- Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction (CVPR 2021 · similarity 0.88)
- Improving Inference for Neural Image Compression (NeurIPS 2020 · similarity 0.87)
- Deep Generative Video Compression (NeurIPS 2019 · similarity 0.87)
- Improved Video VAE for Latent Video Diffusion Model (CVPR 2025 · similarity 0.86)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.