Research map / Image generation / Text-to-image diffusion
The CLIP Model is Secretly an Image-to-Prompt Converter
Yuxuan Ding, Chunna Tian, Haoxuan Ding, Lingqiao Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| map CLIP image embedding to T-space with inverse projection matrix instead of C-space | CLIP ViT-L/14 · ImageNet val | Acc@1: 71.41 -> 69.48 | No clear effect |
| same inverse matrix transfer | CLIP · image retrieval | IR@1: 55.54 -> 54.82 | No clear effect |
| training-free SD-IPC vs text-based SD generation | Stable Diffusion · MSCOCO val | FID: 23.65 -> 24.78 | No clear effect |
| SD-IPC vs text generation on CLIP-Score | Stable Diffusion · MSCOCO val | CLIP-Score: 70.15 -> 73.57 | Improved |
| fine-tune both CLIP prompts and U-Net instead of SD-IPC alone (ablation) | SD-IPC · image variation | DINO: 44.60 -> 52.03 | Improved |
| train only CLIP prompts (C) (ablation) | SD-IPC-FT · text-edited variation | DINO: 31.09 -> 29.10 | Hurt |
| train only U-Net cross-attention (U) (ablation) | SD-IPC-FT · text-edited variation | DINO: 31.09 -> 35.21 | Improved |
| fine-tune both components rather than one (ablation) | SD-IPC-FT · text-edited variation | DINO: 35.21 -> 40.28 | Improved |
| fine-tune both rather than U only (ablation) | SD-IPC-FT · image variation | CLIP-T: 26.17 -> 25.90 | Hurt |
| fine-tuning for editing | SD-IPC · DreamBooth editing | CLIP-T: 26.84 -> 28.69 | Improved |
| SD-IPC vs SD-R editing | SD-IPC · DreamBooth editing | CLIP-T: 26.01 -> 26.84 | Improved |
| SD-IPC-CT vs DreamBooth on editing | SD-IPC-CT · DreamBooth benchmark | CLIP-T: 25.81 -> 28.14 | Improved |
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.89)
- Dynamic Prompt Optimizing for Text-to-Image Generation (CVPR 2024 · similarity 0.89)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
