Research map / Image generation / Image restoration
Controllable Distortion-Perception Tradeoff Through Latent Diffusion for Neural Image Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Plug-and-play latent diffusion decoder-side module that transforms decoded latents of a fixed pretrained codec, with a tunable tau that fuses original and perception-oriented features for controllable distortion-perception tradeoff.
Problem
Neural image compression faces a rate-distortion-perception tradeoff; distortion-optimized codecs blur at low bitrate while perception-oriented codecs lose pixel fidelity, and existing controllable methods require training whole codecs from scratch.
Approach
Train a latent diffusion module on top of a frozen pretrained codec's decoded latents, guided by an auxiliary encoder trained with LPIPS, to produce perception-oriented latents; at inference, interpolate original and diffused latents with a user-set tau to traverse distortion and perception without changing the bitstream or decoder.
Key results
- Ours(Cheng2020, tau=0) improves LPIPS-BDRate by 158.75% over Cheng2020 with about 1.08 dB average PSNR loss.
- Ours(HiFiC, tau=0) saves 22.59% bitrate in PSNR on Kodak relative to HiFiC.
- Ablation: removing the diffusion process and auxiliary encoder (Variant-1) yields +6.91% PSNR-BDRate and +4.69% LPIPS-BDRate versus the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive latent fusion to Cheng2020 | Cheng2020 · Kodak | LPIPS-BDRate: 158.75% | Helped |
| Add adaptive latent fusion to Cheng2020 | Cheng2020 · Kodak | PSNR: 1.08 dB | Hurt |
| Add adaptive latent fusion to Cheng2020 | Cheng2020 · Kodak | PSNR versus HiFiC: 1.48 dB | Helped |
| Add adaptive latent fusion to HiFiC | HiFiC · Kodak | bitrate at PSNR performance: 22.59% | Helped |
| Add adaptive latent fusion to MSHyper | MSHyper · Kodak | LPIPS-BDRate: 170% | Helped |
| Remove diffusion process and auxiliary encoder/loss (ablation) | Ours · Kodak | BDRate (PSNR): 0 -> +6.91% | Hurt |
| Remove diffusion process and auxiliary encoder/loss (ablation) | Ours · Kodak | BDRate (LPIPS): 0 -> +4.69% | Hurt |
| Unfreeze and jointly train the decoder while omitting diffusion (ablation) | Ours · Kodak | BDRate (PSNR): 0 -> -2.36% | Mixed |
| Use more than 10 sampling timesteps (ablation) | Ours · Reconstruction | LPIPS | Hurt |
Surprising findings
- Naively interpolating untreated features (Variant-1 without diffusion/auxiliary encoder) makes PSNR fail to correlate positively with tau.
- Running more than 10 diffusion timesteps degrades LPIPS because the diffusion module introduces excessive high-frequency components.
Limitations
- Perception quality is limited by the fixed decoder and the information carried by encoder-extracted features, so it does not reach the best perceptual models.
- Further LPIPS reductions can introduce high-frequency noise at low bitrates.
- Diffusion steps beyond 10 degrade LPIPS through excessive high-frequency content.
Experimental setup
Datasets: Flickr2W
Benchmarks: Kodak CLIC2020
Metrics: PSNR SSIM DISTS LPIPS FID CLIP-IQA NIQE BD-Rate PIEAPP BRISQUE
Baselines: Cheng2020 HiFiC CDC MRIC MSHyper DIRAC DiffBIR
Builds on: DDIM Cheng2020 learned image compression HiFiC Latent Diffusion Models CDC
Closest papers by meaning
- Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression (CVPR 2025 · similarity 0.92)
- Correcting Diffusion-Based Perceptual Image Compression with Privileged End-to-End Decoder (ICML 2024 · similarity 0.90)
- Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis (CVPR 2024 · similarity 0.88)
- Towards image compression with perfect realism at ultra-low bitrates (ICLR 2024 · similarity 0.88)
- DeNC: Unleash Neural Codecs in Video Streaming with Diffusion Enhancement (AAAI 2025 · similarity 0.87)
- OSCAR: One-Step Diffusion Codec Across Multiple Bit-rates (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.