Research map / Video / Video segmentation and tracking
DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeNC, a plug-and-play neural codec for neural-enhanced video streaming that downscales resolution and quantizes color bit-depth at the encoder, and uses a diffusion-based decoder conditioned on the compression to restore perceptual quality.
Problem
Neural-enhanced video streaming (NeVS) restores low-quality compressed video at the media server, but existing neural codecs rely on post-streaming decoder restoration, optimize rate-distortion rather than perception, and are domain-specific, limiting the rate-distortion-perception trade-off.
Approach
Jointly compress frames at the encoder by patch-wise resolution downscaling and K-means color bit-depth quantization, and train a diffusion-based decoder whose conditioning signal is the upscaled compressed frame so restoration is aware of the encoder's compression.
Key results
- DeNC achieves compression ratio of 31.8x on 720p YouTube video versus 2.1x-9.2x for baselines (Gemino 9.2x) and 26.6x on 1080p Netflix video versus up to 15.9x for Gemino.
- DeNC attains VMAF 95.63 and FID 3.17 on 720p YouTube, and MOS 4.1 versus best baseline 3.4 (Gemino).
- DeNC per-frame time is 21.6 ms (720p) and 42.9 ms (1080p) with sampling steps reduced below 10 via progressive distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeNC instead of Gemino | Neural video codec · YouTube 720p 30 fps | compression ratio: 9.2× -> 31.8× | Helped |
| Use DeNC instead of Gemino | Neural video codec · YouTube 720p 30 fps | VMAF: 95.11 -> 95.63 | Helped |
| Use DeNC instead of Gemino | Neural video codec · YouTube 720p 30 fps | MOS: 3.4 -> 4.1 | Helped |
| Use DeNC instead of Gemino | Neural video codec · Netflix 1080p 60 fps | compression ratio: 15.9× -> 26.6× | Helped |
| Use DeNC instead of Gemino | Neural video codec · Netflix 1080p 60 fps | VMAF: 93.16 -> 93.41 | Helped |
| Use DeNC instead of Gemino | Neural video codec · Netflix 1080p 60 fps | MOS: 3.1 -> 3.9 | Helped |
| Increase the resolution downscaling factor (ablation) | DeNC · 720p and 1080p videos | FID | Hurt |
| Decrease the color bit-depth (ablation) | DeNC · 720p and 1080p videos | FID | Hurt |
Surprising findings
- DeNC does not always achieve the highest PSNR, since generative diffusion detail synthesis penalizes PSNR, while it leads in perceptual metrics.
- Gemino achieves better PSNR and some FID values than DeNC in the 1080p Netflix setting, and its per-frame time is the highest among baselines (156.8 ms at 1080p).
Limitations
- Not designed for 4K resolution or HDR color space, as stated by the authors.
- PSNR is not highest; generative details reduce pixel-level fidelity metrics.
- Relies on fixed codebook and patch-based downscaling; generalization across codecs claimed but not extensively tested in the shown experiments.
Experimental setup
Datasets: YouTube videos Netflix videos
Benchmarks: YouTube test videos (720p/30fps) Netflix test videos (1080p/60fps)
Metrics: compression ratio PSNR SSIM VMAF FID UIQI LPIPS IS MOS per-frame time
Baselines: Gemino Grace LiFteR NeuroScaler NEMO
Builds on: DDPM SR3 progressive distillation H.265
Closest papers by meaning
- Towards Practical Real-Time Neural Video Compression (CVPR 2025 · similarity 0.88)
- Neural Video Compression with Feature Modulation (CVPR 2024 · similarity 0.88)
- Controllable Distortion-Perception Tradeoff Through Latent Diffusion for Neural Image Compression (AAAI 2025 · similarity 0.87)
- NVRC: Neural Video Representation Compression (NeurIPS 2024 · similarity 0.87)
- Neural Video Compression with Context Modulation (CVPR 2025 · similarity 0.87)
- PNVC: Towards Practical INR-based Video Compression (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.