Research map / Image generation / Text-to-image diffusion
AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGAPO, a task of training automatic prompt optimization on known Stable Diffusion checkpoints to generalize to unseen ones, and AP-Adapter, a two-stage method (GPT-3.5 in-context keyword rewriting plus a prototype-anchored adapter over CLIP text embeddings) evaluated on a newly collected CivitAI dataset.
Problem
Automatic prompt optimization methods for text-to-image generation are typically trained for a fixed diffusion model and generalize poorly to newly emerging fine-tuned checkpoints with different parameters.
Approach
Treat each SD checkpoint as a domain, build domain prototypes from channel-dropped CLIP image features that capture inter-model differences, and use similarity-weighted prototypes as anchors to adapt prompt text embeddings toward manual-prompt representations so they transfer to unseen checkpoints.
Key results
- On unseen target checkpoints, AP-Adapter reaches BLIPScore 0.332, aesthetic score 6.384, and ImageReward 0.427, versus 0.286 ImageReward for SUR-adapter and 0.124 for PromptPerfect.
- Color attribute binding score improves to 0.477 over SUR-adapter's 0.472 and other baselines.
- Ablation shows removing the prototype alignment loss La drops ImageReward from 0.427 to 0.384 (w/ all losses 0.427, w/o prototypes 0.258).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | Color: 0.472 -> 0.477 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | Shape: 0.413 -> 0.422 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | Texture: 0.449 -> 0.452 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | Blipscore: 0.325 -> 0.332 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | Aesthetic Score: 6.249 -> 6.384 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | ImageReward: 0.286 -> 0.427 | Helped |
| Use AP-Adapter instead of the strongest reported baseline | Automatic prompt optimization on unseen diffusion models · Target domain data | HPS: +0.005 relative to AKP (0.213 -> 0.218) | Helped |
| Add denoising loss Ld to Lr (ablation) | AP-Adapter · Target domain data | BlipScore: 0.303 -> 0.313 | Helped |
| Add denoising loss Ld to Lr (ablation) | AP-Adapter · Target domain data | AesScore: 6.110 -> 6.139 | Helped |
| Add domain style distillation loss La to Lr and Ld (ablation) | AP-Adapter · Target domain data | ImageReward: 0.258 -> 0.384 | Helped |
| Add domain discrimination loss Lc to Lr, Ld, and La (ablation) | AP-Adapter · Target domain data | ImageReward: 0.384 -> 0.427 | Helped |
| Add learned domain prototypes (ablation) | AP-Adapter · Target domain data | ImageReward: 0.258 -> 0.427 | Helped |
Surprising findings
- Text-optimization methods like PromptPerfect and Promptist improve aesthetic metrics but hurt semantic consistency on unseen models.
- Domain prototypes learned from images sometimes do not have clear semantic meaning, reflecting mixed factors such as detail, lighting, and contrast.
Limitations
- Dataset is relatively small and focuses on single-character images, limiting complex scene handling.
- Number of domain prototypes grows with the number of source checkpoints, increasing model parameters and scalability concerns.
- Evaluation relies on automatic metrics and a limited human study without error bars.
Experimental setup
Datasets: CivitAI community images and SD checkpoints (47,695 collected, 25,395 filtered) LLaVA-generated natural language descriptions
Benchmarks: Custom CivitAI-based target domain (3,064 samples, checkpoints ranked 41-100) T2I-CompBench attribute binding prompts (color, shape, texture)
Metrics: BLIPScore color/shape/texture attribute binding score LAION aesthetic score ImageReward HPS human win-rate for aesthetics and text-image alignment
Baselines: MagicPrompt PromptPerfect Promptist SUR-adapter AKP + SUR-adapter Manual prompts
Builds on: SUR-adapter DomainDrop Promptist Stable Diffusion 1.5
Closest papers by meaning
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.93)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.92)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation (CVPR 2025 · similarity 0.91)
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.91)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.