Research map / Video / Video segmentation and tracking
Augmented Deep Contexts for Spatially Embedded Video Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEVC, a neural video codec that embeds a 4x-downsampled base DCVC-DC codec to supply spatial references, fused with temporal references via motion/feature co-augmentation, a spatial-guided latent prior, and joint spatial-temporal optimization.
Problem
Temporal-only neural video codecs struggle with large motions and emerging objects because their temporal contexts and latent priors are limited and misaligned.
Approach
Embed a low-resolution base codec to produce spatial references (base MVs, spatial feature, spatial latent) and augment deep contexts and the latent prior with them, plus jointly optimize bit allocation between base and full-resolution codecs.
Key results
- SEVC achieves -22.5% average BD-Rate vs VTM-13.2 LDB at intra period 32, versus -14.2% for DCVC-DC.
- Under intra period -1, SEVC reaches -24.5% BD-Rate vs VTM, about 11.9% more savings than DCVC-FM (-12.6%).
- On large-motion/emerging-object sequences, SEVC gives -55.9% on USTC BicycleDriving vs -17.9% for DCVC-DC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| embed a low-resolution base codec to provide spatial references | SEVC · Average, IP 32 | BD-Rate (%): –14.2 -> –22.5 | Helped |
| embed a low-resolution base codec to provide spatial references | SEVC · Average, IP –1 | BD-Rate (%): –12.6 -> –24.5 | Helped |
| remove augmentation and send base MVs and spatial feature directly to context fusion (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 17.5 | Hurt |
| use one instead of two Augment Stages in each scale (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 4.7 | Hurt |
| use three instead of two Augment Stages in each scale (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> –1.9 | Helped |
| add multiple temporal latent representations to the prior (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 1.5 | Hurt |
| replace Transformers with convolution networks in M2 (ablation) | SEVC · HEVC datasets | BD-Rate (%): 1.5 -> 6.5 | Hurt |
| discard spatial latent representation in prior generation (ablation) | SEVC · HEVC datasets | BD-Rate (%): 6.5 -> 8.4 | Hurt |
| disable joint spatial-temporal optimization and fix base codec parameters during finetuning (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 5.1 | Hurt |
| set wl to 0 (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 11.6 | Hurt |
| set wl to 0.01 (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 5.3 | Hurt |
| set wl to 0.1 (ablation) | SEVC · HEVC datasets | BD-Rate (%): 0.0 -> 1.2 | Hurt |
Surprising findings
- The base codec's bit share is not fixed; joint optimization lets the base layer receive more bits for low-quality frames and fewer for high-quality frames.
Limitations
- The base codec is limited to an existing NVC (DCVC-DC), and spatial references are reconstructions of the low-resolution video, which constrains their characteristics.
- Higher computational complexity than DCVC-DC and DCVC-FM.
- Emerging objects still show large residuals in some regions even with spatial references.
Experimental setup
Datasets: Vimeo-90k
Benchmarks: UVG MCL-JCV HEVC Class B-E USTC-TD
Metrics: BD-Rate (PSNR) MACs encoding time decoding time
Baselines: DCVC DCVC-TCM DCVC-HEM DCVC-DC Sheng-2024 DCVC-FM VTM-13.2 LDB
Builds on: DCVC-DC DCVC-HEM Transformer-based latent alignment
Closest papers by meaning
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.88)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.86)
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.86)
- Combining Frame and GOP Embeddings for Neural Video Representation (CVPR 2024 · similarity 0.86)
- ECVC: Exploiting Non-Local Correlations in Multiple Frames for Contextual Video Compression (CVPR 2025 · similarity 0.86)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.