Research map / Video / Video segmentation and tracking
Offline and Online Optical Flow Enhancement for Deep Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Enhances optical flow for DCVC deep video compression in two stages: offline fine-tuning of Spynet using VTM motion vectors, and online gradient-based optimization of optical flow latents per video, yielding 12.8% BD-rate savings over DCVC.
Problem
Pre-trained optical flow networks used for motion estimation in deep video compression are trained for prediction accuracy rather than rate-distortion efficiency, trained on synthetic data, and are not adapted to individual videos.
Approach
Fine-tune Spynet with block-level motion vectors searched by VTM (which achieve a better RD trade-off) as offline supervision, and online-optimize the optical flow latent features (y and z) per video with gradient descent on the RD loss while keeping all network weights fixed, so decoder complexity is unchanged.
Key results
- Offline plus online enhancement achieves on average 12.8% BD-rate saving over DCVC across test datasets.
- Online enhancement alone gives 10.6% average BD-rate saving; offline alone gives 3.9%.
- Enhanced Spynet warp frames improve average PSNR by 1.15 dB (33.23 dB vs 32.08 dB) on JVET CTC sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add offline Spynet fine-tuning with VTM motion vectors (ablation) | DCVC · All testing datasets (Average) | BD-Rate (%): -3.9 | Helped |
| Add online optical-flow latent optimization (ablation) | DCVC · All testing datasets (Average) | BD-Rate (%): -10.6 | Helped |
| Add offline and online optical-flow enhancement (ablation) | DCVC · All testing datasets (Average) | BD-Rate (%): -12.8 | Helped |
| Increase online updating times from 0 to 2500 (ablation) | DCVC + Offline on HEVC Class C and D · HEVC Class C and HEVC Class D | BD-Rate (%): Class C: 0.0 -> -11.6; Class D: 0.0 -> -9.2 | Helped |
| Increase multi-frame window size from 2 to 5 (ablation) | DCVC + Offline + Online · HEVC Class C and HEVC Class D | BD-Rate (%): Class C: 0.0 -> -0.8; Class D: 0.0 -> -0.8 | Mixed |
| Offline-enhance Spynet using VTM motion vectors | Spynet · JVET CTC test sequences | Warp-frame PSNR (dB): 32.08dB vs. 33.23dB; 1.15dB | Helped |
Surprising findings
- Multi-frame online optimization with larger window sizes gives only marginal gains (about 0.8% at W=5) while greatly increasing encoding time.
Limitations
- Online optimization greatly increases encoding time (N=1500 iterations, e.g., 388.73 s vs 2.71 s on Class C).
- Multi-frame online optimization improves little relative to its encoding cost.
- Offline stage relies on VTM motion vectors and its block-level MV is interpolated to dense flow.
Experimental setup
Datasets: BVI-DVC Vimeo-90k JVET CTC UVG HEVC HEVC RGB
Benchmarks: HEVC Class B HEVC Class C HEVC Class D HEVC Class E HEVC RGB UVG JVET CTC
Metrics: BD-rate (PSNR) encoding time decoding time warp PSNR
Baselines: DCVC DVC Cheng2020Anchor
Builds on: DCVC Spynet H.266/VVC online encoder updating
Closest papers by meaning
- DVC: An End-To-End Deep Video Compression Framework (CVPR 2019 · similarity 0.89)
- DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework (CVPR 2025 · similarity 0.87)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.87)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.85)
- Plug-and-Play Versatile Compressed Video Enhancement (CVPR 2025 · similarity 0.85)
- Taming generative video models for zero-shot optical flow extraction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.