Research map / Video / Video segmentation and tracking
Perceptual Video Compression with Neural Wrapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes neural pre- and post-processors ("neural wrappers") trained end-to-end around a differentiable neural codec proxy of AV1/VVC, improving perceptual rate-quality performance of standard codecs without changing the codec.
Problem
Standard video codecs optimize PSNR, which does not align with perceptual visual quality; existing perceptual optimizations require multiple passes, target a single metric, or yield only small BD-rate gains.
Approach
A neural video codec (scale-space flow based) is aligned to a target standard codec via a two-phase pretraining (distortion alignment to the codec output, then unsupervised rate alignment with an affine rate map), and pre/post-processors are trained end-to-end through it using iterative codec-model refinement and a stop-gradient trick that feeds the real codec output to the postprocessor.
Key results
- Average BD-rate of -18.5% across objective scores versus the underlying codecs (abstract); on the Gaming set, AV1+Proposed reaches -19.7% ALL and VVC+Proposed -14.6% ALL.
- Proposed AV1 wrapper gives -30.0% BD-rate on VMAF on Gaming and -41.8% on HEVC-B, outperforming DCVC-DC and DCVC-FM on most metrics.
- Two-phase pretraining raises rate/PSNR PCC between codec model and target codec from 0.56/0.60 to 0.88/0.85; a 5-scale ACR MOS test under ITU-T P.910 also shows gains over both AV1 and VVC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add neural pre- and postprocessor wrapper | SVT-AV1 · Gaming, XIPH, UVG, HEVC-B | BD-rate averaged over all quality scores: -19.7 / -19.2 / -20.0 / -21.2 | Helped |
| add neural pre- and postprocessor wrapper | VVC · Gaming, XIPH, UVG, HEVC-B | BD-rate averaged over all quality scores: -14.6 / -12.8 / -11.2 / -29.4 | Helped |
| add pre+post processor wrapper (joint training) (ablation) | SVT-AV1 · Gaming | BD-rate SSIM/MS-SSIM/AVQT/VMAF: -25.6 / -20.8 / -21.4 / -20.5 | Helped |
| use only postprocessor (no preprocessor) (ablation) | SVT-AV1 · Gaming | BD-rate SSIM: +17.9 vs -25.6 full | Hurt |
| use only preprocessor (no postprocessor) (ablation) | SVT-AV1 · Gaming | BD-rate SSIM: -2.9 vs -25.6 full | Hurt |
| remove iterative codec model update (ablation) | end-to-end training · Gaming | BD-rate SSIM: -23.1 vs -25.6 full | Hurt |
| remove stop-gradient operator in end-to-end training (ablation) | end-to-end training · Gaming | BD-rate SSIM: +16.4 vs -25.6 full | Hurt |
| remove two-phase codec model pretraining (ablation) | end-to-end training · Gaming | BD-rate SSIM: -11.2 vs -25.6 full | Hurt |
| add stop-gradient and iterative codec updates to end-to-end training (ablation) | neural wrapper training · Gaming | BD-rate MS-SSIM: -20.8 vs 0.01 without stop-grad | Helped |
| Replacing DPP preprocessor with proposed wrapper on AV1 | SVT-AV1 · Gaming | BD-rate AVQT: -19.6 vs -17.6 for DPP | Helped |
Surprising findings
- Jointly training pre- and post-processors is needed: preprocessor-only or postprocessor-only variants lose BD-rate on SSIM/MS-SSIM (e.g., pre-only +17.9% SSIM BD-rate).
- Iterative codec-model updates without stop-gradient cause BD-rate losses on SSIM and MS-SSIM due to domain shift.
Limitations
- PSNR regresses with the wrapper, as the method optimizes perceptual rather than fidelity metrics.
- Gains on VMAF may partly reflect sharpening effects that VMAF is susceptible to.
- Codec-model alignment to VVenC was found to give slight extra gains only at high computational training overhead, so the main results use SVT-AV1 alignment.
Experimental setup
Datasets: Vimeo-90k
Benchmarks: Gaming (16 sequences) XIPH UVG HEVC-B
Metrics: BD-rate SSIM MS-SSIM VMAF AVQT MOS (ITU-T P.910 ACR)
Baselines: AV1+SSIM tuning DPP Hu et al. codec-model sandwich DCVC-DC DCVC-FM VVC+QPA
Builds on: scale-space flow hyperprior compression (Ballé et al.) DPP Hu et al. sandwich codec training
Closest papers by meaning
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.85)
- Controllable Distortion-Perception Tradeoff Through Latent Diffusion for Neural Image Compression (AAAI 2025 · similarity 0.85)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.84)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.84)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.83)
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.