Research map / Image generation / Image restoration
Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TACO, a text-adaptive neural image compression codec that uses text only at the encoder (CLIP text features injected via cross-attention adapter) plus a joint image-text loss, achieving high PSNR and LPIPS.
Problem
Text-guided image compression improves perceptual quality but severely degrades pixel-wise fidelity (PSNR) when text is used for decoding via generative models.
Approach
Use text only in the encoder: a text adapter injects frozen CLIP text features into an ELIC encoder via cross-attention, and the codec is trained from scratch with a joint image-text loss (CLIP contrastive plus CLIP embedding distance), keeping a PSNR-oriented decoder without adversarial loss.
Key results
- On MS-COCO 30k, TACO reaches 0.025 LPIPS at 0.226 bpp versus 0.053 at 0.233 bpp for the best baseline (HiFiC).
- TACO stays within about 1 dB PSNR of LIC-TCM and ELIC while outperforming all baselines on LPIPS across datasets.
- Removing the text adapter raises LPIPS more than 4x on Kodak, showing the adapter drives perceptual gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the text adapter to the ELIC encoder (ablation) | TACO · Kodak | LPIPS: more than 4× | Helped |
| add the text adapter to the ELIC encoder (ablation) | TACO · Kodak | CMMD | Helped |
| add joint image-text loss (ablation) | TACO · Kodak | LPIPS | Helped |
| add joint image-text loss (ablation) | TACO · Kodak | PSNR: tiny degradation | Hurt |
| add joint image-text loss (ablation) | TACO · Kodak | CMMD | Helped |
| unfreeze the ELIC base model and train it with the text adapter (ablation) | TACO · compression evaluation | performance | Helped |
| use the text adapter instead of the TGIC image-text attention module | ELIC encoder on CLIC · CLIC | compression quality | Helped |
| use text-guided encoding instead of text-guided decoding | TACO vs. Qin 2023 · MS-COCO 40k | LPIPS | Helped |
| use human, OFA, BLIP-2, or GPT-4 captions (ablation) | TACO · MS-COCO, 0.0148 bpp | LPIPS: 0.0435 | No clear effect |
| use OFA or BLIP-2 captions instead of human captions (ablation) | TACO · MS-COCO, 0.0148 bpp | PSNR: 27.42 -> 27.41 | No clear effect |
| add the text adapter to ELIC (ablation) | TACO · compression codecs | PSNR: tiny gain | Hurt |
| add joint image-text loss (ablation) | TACO · compression codecs | CMMD: tiny gain | Helped |
Surprising findings
- Text-guided encoding outperformed the prior text-guided decoding baseline (Qin 2023) on PSNR, LPIPS and FID, contrary to expectation.
- Captions from OFA, BLIP-2 and GPT-4 gave nearly identical LPIPS and PSNR on 100 MS-COCO samples, suggesting low dependence on caption style.
Limitations
- Text injection cost scales quadratically with text sequence length due to cross-attention.
- Training relies on image-text pairs, which are scarcer than image-only data.
- CLIP adds about 63M parameters, making the model larger than vanilla ELIC.
Experimental setup
Datasets: MS-COCO CLIC 2020 Kodak OFA-generated captions
Benchmarks: MS-COCO 30k CLIC 2020 Kodak
Metrics: LPIPS PSNR FID CMMD MS-SSIM PieAPP BLEU-4 CIDEr-D SPICE
Baselines: VTM BPG ELIC LIC-TCM HiFiC MS-ILLM Qin 2023 TGIC
Builds on: ELIC CLIP TGIC
Closest papers by meaning
- PICD: Versatile Perceptual Image Compression with Diffusion Rendering (CVPR 2025 · similarity 0.87)
- Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning (NeurIPS 2024 · similarity 0.86)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.86)
- Bridging Compressed Image Latents and Multimodal Large Language Models (ICLR 2025 · similarity 0.86)
- Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis (ICLR 2025 · similarity 0.86)
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.