Research map / Video / Video segmentation and tracking
ECVC: Exploiting Non-Local Correlations in Multiple Frames for Contextual Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECVC, a learned video codec that mines local and non-local temporal contexts from two reference frames (MNLC with linear cross attention) and fine-tunes on long sequences with partial cascaded fine-tuning (PCFS) to reduce error accumulation.
Problem
Learned video compression inter prediction mainly captures local motion, ignores non-local correlations among frames, uses a single reference frame, and suffers error accumulation from train-test sequence length mismatch.
Approach
Multiple Frame Non-Local Context Mining combines offset-diversity local contexts from two reference frames with multi-head linear cross attention (softmax on rows and columns) for non-local contexts at linear cost; Partial Cascaded Finetuning fine-tunes on groups of a long sequence to reduce error accumulation under limited GPU memory.
Key results
- Under IP 32 (PSNR), ECVC achieves 29.3% average BD-rate saving over VTM-13.2 LDB, versus 18.8% for DCVC-FM, i.e. about 10.5% more savings than DCVC-FM.
- Under IP -1 (all frames), ECVC achieves 32.1% average BD-rate saving over VTM-13.2 LDB, versus 20.6% for DCVC-FM (about 11.5% more).
- Under MS-SSIM optimization at IP 32, ECVC gives 56.6% average BD-rate saving over VTM-13.2, 3.7% more than DCVC-DC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ECVC with multiple-frame non-local context mining and partial cascaded fine-tuning instead of DCVC-DC | Video compression · Average, IP 32, 96 frames | BD-Rate (%) vs VTM-13.2 LDB, PSNR: −22.0 -> −29.3 | Helped |
| Use ECVC instead of DCVC-FM | Video compression · Average, IP −1, all frames | BD-Rate (%) vs VTM-13.2 LDB, PSNR: −20.6 -> −32.1 | Helped |
| Add MNLC to the DCVC-DC base (ablation) | DCVC-DC · HEVC B/C/D/E, IP 32 | BD-Rate (%) vs base model: −4.0 / −9.9 / −9.0 / −9.7 | Helped |
| Add MNLC to the DCVC-DC base (ablation) | DCVC-DC · HEVC B/C/D/E, IP −1 | BD-Rate (%) vs base model: −2.5 / −6.5 / −7.9 / −16.9 | Helped |
| Add PCFS and MNLC to the DCVC-DC base (ablation) | DCVC-DC · HEVC B/C/D/E, IP 32 | BD-Rate (%) vs base model: −9.1 / −15.3 / −14.5 / −14.8 | Helped |
| Add PCFS and MNLC to the DCVC-DC base (ablation) | DCVC-DC · HEVC B/C/D/E, IP −1 | BD-Rate (%) vs base model: −18.1 / −25.7 / −24.8 / −45.1 | Helped |
| Add PCFS to DCVC-HEM (ablation) | DCVC-HEM · HEVC B/C/D/E, IP 32 | BD-Rate (%) vs VTM-13.2 LDB: 23.8 -> 14.5 | Helped |
| Add PCFS to DCVC-HEM (ablation) | DCVC-HEM · HEVC B/C/D/E, IP −1 | BD-Rate (%) vs VTM-13.2 LDB: 27.9 -> 4.5 | Helped |
| Increase fine-tuning sequence length from 6 to 55 frames (ablation) | ECVC · HEVC E, IP −1 | BD-Rate (%) vs base model: −9.5 -> −45.1 | Helped |
| Increase fine-tuning sequence length from 6 to 55 frames (ablation) | ECVC · HEVC B, IP 32 | BD-Rate (%) vs base model: −4.0 -> −9.1 | Helped |
Surprising findings
- ECVC may perform worse than VTM on out-of-domain sequences such as anime videos, since it is trained on natural videos.
- Fine-tuning on longer sequences helps most on low-motion HEVC E, e.g. PCFS with 55 frames reduces HEVC E bit-rate by 45.1% over base under IP -1 in ablation.
Limitations
- Performs worse than VTM on out-of-domain content such as anime videos because it is trained on natural videos.
- Slightly higher complexity and parameters than DCVC-DC (61.9M vs 50.8M params).
Experimental setup
Datasets: Vimeo-90K BVI-DVC
Benchmarks: HEVC B HEVC C HEVC D HEVC E UVG MCL-JCV
Metrics: BD-Rate PSNR MS-SSIM
Baselines: DCVC-DC DCVC-FM DCVC-HEM DCVC-TCM VTM-13.2 low delay B
Builds on: DCVC-DC DCVC-FM offset diversity linear attention
Closest papers by meaning
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.88)
- Augmented Deep Contexts for Spatially Embedded Video Coding (CVPR 2025 · similarity 0.86)
- Neural B-frame Video Compression with Bi-directional Reference Harmonization (NeurIPS 2025 · similarity 0.85)
- FLAVC: Learned Video Compression with Feature Level Attention (CVPR 2025 · similarity 0.85)
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.85)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.