Research map / Video / Video segmentation and tracking
Plug-and-Play Versatile Compressed Video Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a codec-aware, plug-and-play compressed video enhancer that uses H.264 bitstream priors (CRF, motion vectors, partition maps) to adaptively generate frame-wise enhancement network parameters, improving quality and downstream tasks across compression levels.
Problem
Video compression degrades visual quality and hurts downstream vision models, while existing compressed-video enhancers need separate models per compression level, are compression-agnostic, and ignore downstream task assistance.
Approach
A compression-aware adaptation network hierarchically (sequence-level then frame-level CRF) generates parameters of a bitstream-aware enhancement network, which uses motion-vector alignment of reference frames and partition-map-guided region-aware sparse convolutions to enhance frames using codec information with little overhead.
Key results
- On REDS4 at CRF15, the method achieves 42.22 dB PSNR vs 41.04 dB input, a 1.2 dB gain, while MFQE 2.0 and Metabit give no improvement.
- As ×4 VSR pre-enhancement with BasicVSR++ on CRF35, gains up to 0.62 dB PSNR.
- On STCN at CRF35 for VOS, average accuracy improves from 79.20% to 81.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add codec-aware enhancement before BasicVSR++ | BasicVSR++ · REDS4, ×4 VSR, CRF35 | PSNR: 23.38 -> 24.00 | Helped |
| Add codec-aware enhancement before DEQ | DEQ · KITTI-2015 optical flow, CRF35 | EPE: 13.94 -> 13.09 | Helped |
| Add codec-aware enhancement before STCN | STCN · DAVIS-17 val VOS, CRF35 | Avg: 79.20 -> 81.57 | Helped |
| Replace CRFi with slice type for frame adaptation (ablation) | Ours · REDS4 quality enhancement, CRF15/25/35/18/28/38 | PSNR: 42.22 -> 42.24; 35.90 -> 35.91; 30.17 -> 30.19; 40.17 -> 40.18; 34.16 -> 34.17; 28.49 -> 28.52 | No clear effect |
| Add motion-vector alignment (ablation) | Baseline enhancement network · REDS4, CRF15/25/35 | PSNR: 41.04 -> 41.69; 34.92 -> 35.59; 29.25 -> 29.95 | Helped |
| Add region-aware refinement after MV alignment (ablation) | Baseline enhancement network with MV alignment · REDS4, CRF15/25/35 | PSNR: 41.69 -> 42.04; 35.59 -> 35.70; 29.95 -> 30.00 | Helped |
| Add sequence adaptation (ablation) | Baseline enhancement network with MV alignment and region-aware refinement · REDS4, CRF15/25/35 | PSNR: 42.04 -> 42.08; 35.70 -> 35.76; 30.00 -> 30.04 | Helped |
| Add frame adaptation (ablation) | Baseline enhancement network with MV alignment, region-aware refinement, and sequence adaptation · REDS4, CRF15/25/35 | PSNR: 42.08 -> 42.14; 35.76 -> 35.81; 30.04 -> 30.09 | Helped |
| Pre-enhance with MFQE 2.0 | BasicVSR++ · REDS4, ×4 VSR, CRF15 | PSNR: 29.61 -> 29.66 | Helped |
| Pre-enhance with Metabit | BasicVSR++ · REDS4, ×4 VSR, CRF15 | PSNR: 29.61 -> 29.61 | No clear effect |
| Pre-enhance with MFQE 2.0 | RAFT · KITTI-2015 optical flow, CRF15 | EPE: 5.26 -> 5.32 | Hurt |
| Pre-enhance with STDF | DEQ · KITTI-2015 optical flow, CRF15 | EPE: 3.99 -> 4.08 | Hurt |
Surprising findings
- Metabit and MFQE 2.0 yield no quality improvement over compressed input in several settings, and pre-enhancement with some methods (STDF, Metabit) hurts downstream optical flow on CRF15.
Limitations
- Evaluated only on H.264; other codecs (H.265/H.266) are only claimed to provide similar priors, not tested.
- Requires access to codec side information (CRF, motion vectors, partition maps) from the bitstream, which may be unavailable in some settings.
- Frame-level CRF may be unavailable; slice type substitution is used instead with slight loss.
Experimental setup
Datasets: REDS DAVIS
Benchmarks: REDS4 KITTI-2015 DAVIS-17 val
Metrics: PSNR SSIM EPE F1-all J F Avg
Baselines: MFQE 2.0 STDF S2SVR Metabit BasicVSR IconVSR BasicVSR++ RAFT DEQ KPAFlow STCN DeAoT
Builds on: MFQE 2.0 STDF DRConv Gainfune-style test-time adaptation (GainTune) Charbonnier loss
Closest papers by meaning
- Offline and Online Optical Flow Enhancement for Deep Video Compression (AAAI 2024 · similarity 0.85)
- RivuletMLP: An MLP-based Architecture for Efficient Compressed Video Quality Enhancement (CVPR 2025 · similarity 0.84)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.84)
- CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement (CVPR 2024 · similarity 0.83)
- GVCC: Zero-Shot Video Compression via Codebook-Driven Stochastic Rectified Flow (NeurIPS 2026 · similarity 0.83)
- Make Lossy Compression Meaningful for Low-Light Images (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.