Research map / Video / Video segmentation and tracking
Neural B-frame Video Compression with Bi-directional Reference Harmonization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BRHVC, a neural B-frame video compression method with Bi-directional Motion Converge and Bi-directional Contextual Fusion to balance contributions of two reference frames, outperforming DCVC-FM/DCVC-B and surpassing VTM-RA on HEVC.
Problem
Neural B-frame video compression under hierarchical random-access coding suffers from unbalanced reference contribution, since the two bi-directional references differ greatly in usefulness at large frame spans, and existing methods lag behind VTM-RA.
Approach
Converge multi-scale optical flows into a single compressed latent with bi-directional priors (BMC), and explicitly learn per-region weights for the two context-reference features guided by motion-compensated pixel similarity (BCF), so the bi-directional references are harmonized.
Key results
- BRHVC achieves 32.0% average BD-rate saving on HEVC vs VTM-LDB, versus VTM-RA's 30.9%, and 27.3% on all datasets.
- Ablation: BCF alone gives -6.6% and BMC plus BCF gives -12.3% BD-rate relative to the baseline, with DCVC-B re-trained at 0.4%.
- On HEVC Class D, BRHVC saves 44.7% bitrate vs VTM-LDB; BRHVC also outperforms DCVC-B on all datasets in the paper's tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BRHVC instead of VTM-RA | Neural B-frame video compression · HEVC Average | BD-rate (%): -30.9 -> -32.0 | Helped |
| Use BRHVC instead of DCVC-FM | Neural video compression · All Average | BD-rate (%): -20.3 -> -27.3 | Helped |
| Add BMC to the baseline (ablation) | BL · HEVC datasets | BD-rate (%): 0.0 -> -6.4% | Helped |
| Add BCF to the baseline (ablation) | BL · HEVC datasets | BD-rate (%): 0.0 -> -6.6% | Helped |
| Add both BCF and BMC to the baseline (ablation) | BL · HEVC datasets | BD-rate (%): 0.0 -> -12.3% | Helped |
| Add BMC to BCF (ablation) | BL with BCF · HEVC datasets | BD-rate (%): -6.6% -> -12.3% | Helped |
| Replace largest-scale flows with upsampled intermediate-scale flows (ablation) | BRHVC · HEVC datasets | BD-rate (%): 0 -> 6.7% | Hurt |
| Replace intermediate-scale flows with downsampled largest-scale flows (ablation) | BRHVC · HEVC datasets | BD-rate (%): 0 -> 4.2% | Hurt |
| Replace smallest-scale flows with downsampled intermediate-scale flows (ablation) | BRHVC · HEVC datasets | BD-rate (%): 0 -> 3.0% | Hurt |
| Stack 4 ResBlocks per scale for reference fusion (ablation) | BL · HEVC datasets | BD-rate (%): 0.0 -> -3.9% | Helped |
| Stack 9 encoder and 4 decoder ResBlocks (ablation) | BL · HEVC datasets | BD-rate (%): 0.0 -> -4.1% | Helped |
| Use BRHVC instead of DCVC-B | MS-SSIM-optimized video compression · HEVC Average | BD-rate (%): -42.4% -> -44.0% | Helped |
Surprising findings
- Simply stacking more ResBlocks (BL-1, BL-2) yields only about 4% gain versus 12.3% for BRHVC, suggesting the task-specific design matters more than added capacity.
- On MCL-JCV and UVG (high frame rate), BRHVC underperforms relative to VTM-RA, with DCVC-B close on MCL-JCV in the reported BD-rates.
Limitations
- Does not perform sufficiently well on high frame rate datasets such as MCL-JCV and UVG (per authors).
- Decoding time increases by about 24% with the added modules.
- Quantitative URC analysis is limited to a single baseline model and average BD-rate results.
Experimental setup
Datasets: Vimeo-90k
Benchmarks: HEVC Class B HEVC Class C HEVC Class D HEVC Class E MCL-JCV UVG
Metrics: BD-rate (PSNR) BD-rate (MS-SSIM) kMACs/pixel parameters memory encoding/decoding time
Baselines: VTM-LDB VTM-RA HM-LDB HM-RA DCVC-DC DCVC-FM DCVC-B
Builds on: DCVC-B DCVC-DC SpyNet BMRC bi-directional motion residual compression
Closest papers by meaning
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.89)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.87)
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.86)
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.86)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.85)
- ECVC: Exploiting Non-Local Correlations in Multiple Frames for Contextual Video Compression (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.