Research map / Image generation / Image restoration
Towards image compression with perfect realism at ultra-low bitrates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerCo, a VQ-VAE-style image codec whose decoder is a text-and-spatially-conditioned latent diffusion model, targeting perfect-realism reconstructions at ultra-low bitrates (down to 0.003 bpp).
Problem
Rate-distortion-optimized and adversarially trained codecs produce blurry or artefact-laden reconstructions at very low bitrates, and prior diffusion codecs were limited to tiny images or used weak conditioning signals.
Approach
Use a pretrained text-to-image latent diffusion model as the decoder of a VQ-VAE-like codec, conditioned on a quantized hyper-latent (local spatial information) and a losslessly compressed BLIP-2 caption (global semantics), with the diffusion loss replacing pixel distortion losses and classifier-free guidance applied to text only.
Key results
- Achieves realistic reconstructions at bitrates as low as 0.003 bpp on MS-COCO and Kodak, with FID and KID much flatter across bitrates than baselines.
- Reports best FID on MS-COCO 30k at 256x256 for both 0.107 and 0.175 bpp versus HFD/DDPM, MS-ILLM and Multi-Realism.
- At 0.0112 bpp, text-only classifier-free guidance gives FID 4.42 versus 7.89 with no guidance and 13.84 with text and spatial guidance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply classifier-free guidance to text conditioning only (ablation) | PerCo · MS-COCO 30k | FID: 7.89 -> 4.42; 13.84 -> 4.42 | Helped |
| Apply classifier-free guidance to text conditioning only (ablation) | PerCo · MS-COCO 30k | mIoU: 30.84 -> 46.64; 32.26 -> 46.64 | Helped |
| Add text conditioning alongside spatial features (ablation) | PerCo · MS-COCO 30k | FID | Helped |
| Add spatial conditioning alongside text (ablation) | PerCo · MS-COCO 30k | LPIPS | Helped |
| Use more detailed IDEFICS captions instead of BLIP captions, with spatial conditioning (ablation) | PerCo · MS-COCO 30k | CLIP | Helped |
| Use IDEFICS captions instead of BLIP captions, with spatial conditioning (ablation) | PerCo · MS-COCO 30k | LPIPS | Hurt |
| Use ground-truth captions instead of BLIP captions (ablation) | PerCo · MS-COCO 30k | FID | No clear effect |
| Remove quantization (ablation) | PerCo · MS-COCO 30k | FID: 4.42 -> 0.68 | No clear effect |
| Remove quantization (ablation) | PerCo · MS-COCO 30k | LPIPS: 0.39 -> 0.07 | No clear effect |
| Add LPIPS loss at higher bitrates (ablation) | PerCo · Higher bitrates | LPIPS | Helped |
| Add LPIPS loss at higher bitrates (ablation) | PerCo · Higher bitrates | FID | Helped |
| Increase denoising steps at lower bitrates (ablation) | PerCo · Lower bitrates | FID | Mixed |
Surprising findings
- Applying classifier-free guidance to both text and spatial conditioning worsened FID (13.84) versus no guidance (7.89), while text-only guidance was best.
- PerCo has lower PSNR than MS-ILLM and somewhat worse LPIPS above 0.01 bpp, because the LDM autoencoder itself limits fidelity.
Limitations
- Worse MS-SSIM, PSNR and LPIPS (at rates above 0.01) than existing approaches, partly due to the LDM autoencoder.
- Limited to medium-size images up to 768x512 resolution.
- Decoding requires 20 denoising steps (about 2.5 s per Kodak image), roughly 8 to 32 times slower than adversarial codecs.
Experimental setup
Datasets: OpenImages MS-COCO 2014 validation MS-COCO 2017 training Kodak
Benchmarks: Kodak MS-COCO 30k
Metrics: FID KID LPIPS MS-SSIM PSNR CLIP score mIoU
Baselines: VTM MS-ILLM HiFiC MS-PatchGAN Text-Sketch (PIC) Text-Sketch (PICS) DIRAC HFD/DDPM Multi-Realism
Builds on: latent diffusion model VQ-VAE classifier-free guidance Text-Sketch (PICS) BLIP-2
Closest papers by meaning
- Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression (CVPR 2025 · similarity 0.92)
- Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion (ICML 2025 · similarity 0.89)
- Good, Cheap, and Fast: Overfitted Image Compression with Wasserstein Distortion (CVPR 2025 · similarity 0.89)
- Bridging the Gap between Gaussian Diffusion Models and Universal Quantization for Image Compression (CVPR 2025 · similarity 0.88)
- Correcting Diffusion-Based Perceptual Image Compression with Privileged End-to-End Decoder (ICML 2024 · similarity 0.88)
- PICD: Versatile Perceptual Image Compression with Diffusion Rendering (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.