Research map / Video / Video segmentation and tracking
Task-Aware Encoder Control for Deep Video Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a task-aware encoder controller (dynamic vision mode prediction plus GoP selection network) that steers pre-trained deep video codecs (FVC, DCVC, TCM) for machine tasks without changing the decoder.
Problem
Deep video compression for machine vision usually requires a dedicated codec and decoder per task; the paper asks whether a single pre-trained DVC decoder can serve human viewing and multiple machine tasks by controlling only the encoder.
Approach
Control the encoder rather than the decoder: a hyperprior-guided mode prediction network skips non-essential motion/residual feature elements (creating a Pm frame), and a GoP selection network chooses per-GoP P vs Pm frame types at encoding time, all while keeping pre-trained DVC decoders fixed.
Key results
- Ours+DCVC achieves about 41.82% bitrate savings vs DCVC at equal MOTA; Ours+FVC about 27.54% vs FVC; Ours+TCM about 12.64% vs TCM on MOT17.
- On ImageNet VID with YOLOV, bitrate reductions of about 21.76% vs FVC and 20.42% vs TCM at equal mAP50.
- On UCF101 action recognition with TSM, bitrate savings of about 21.81% for TCM and 26.03% for FVC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply task-aware encoder control to DCVC | DCVC · MOT17 | BD-RATE (MOTA): -41.82 | Helped |
| Apply task-aware encoder control to FVC | FVC · MOT17 | BD-RATE (MOTA): -34.28 | Helped |
| Apply task-aware encoder control to TCM | TCM · MOT17 | BD-RATE (mAP): -45.10 | Helped |
| Use DivGoP with Pm frames instead of Pr frames (ablation) | DCVC · MOT17 | mAP: 28.68% vs 32.67% bitrate savings | Helped |
| Use DivGoP with Pm frames instead of Pr frames (ablation) | FVC · MOT17 | mAP: 16.24% vs 19.43% bitrate savings | Helped |
| Use GoP Selection instead of the hand-crafted DivGoP structure (ablation) | DCVC · MOT17 | mAP: 32.67% vs 39.43% bitrate saving | Helped |
| Use DivGoP encoding for machine vision rather than the original codec structure (ablation) | TCM · HEVC Class B and Class C | PSNR: about 1dB | Hurt |
| Fine-tune the FVC encoder or full codec for the detection task (ablation) | FVC · MOT17 | bitrate | Hurt |
Surprising findings
- Simply fine-tuning the FVC encoder/codec for detection markedly increases bitrate, because decoded-frame fidelity degradation propagates across inter-frame references.
- Using pretrained-free Pr (warped prediction) frames instead of learned Pm frames gives only about 28.68% (DCVC) and 16.24% (FVC) savings, lower than Pm frames.
Limitations
- Encoding latency increases (e.g., 0.116s to 0.159s for FVC), and the DFS-optimal GoP search is too slow for real-time use.
- Controlling for machine vision reduces human-viewing PSNR by about 1 dB for TCM, and more for FVC.
Experimental setup
Datasets: MOT17 ImageNet VID UCF101 HEVC Class B HEVC Class C
Benchmarks: MOT17 ImageNet VID UCF101 HEVC Class B HEVC Class C
Metrics: bpp MOTA MOTP mAP mAP50 FN Top1 accuracy BD-rate PSNR
Baselines: HEVC DCVC FVC TCM DivGoP
Builds on: FVC DCVC TCM ByteTrack YOLOV
Closest papers by meaning
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.85)
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.84)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.84)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.83)
- Test-Time Fine-Tuning of Image Compression Models for Multi-Task Adaptability (CVPR 2025 · similarity 0.83)
- AdaDPCC: Adaptive Rate Control and Rate-Distortion-Complexity Optimization for Dynamic Point Cloud Compression (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.