Research map / Image generation / Text-to-image diffusion
Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free image-prompting method for Stable Diffusion that removes the image prompt from negative classifier-free guidance (conflict-free guidance) and replaces KV concatenation with a weighted, separately computed Stratified Attention to better reflect fine image details.
Problem
Text prompts cannot capture fine visual details such as textures, so image-prompted generation via self-attention modification fails to faithfully reflect the image prompt, due to conflicting classifier-free guidance and attention bias/trade-offs between realism and alignment.
Approach
Use the image prompt only as positive guidance in classifier-free guidance (conflict-free guidance) and compute attention for generated and image-prompt keys/values separately, combining them via a weighted sum (Stratified Attention) so both sources are balanced at all time steps.
Key results
- On CelebAHQ image variation, the method reaches CLIP-I 0.878 versus 0.845 for IP-Adapter and 0.739 for RIVAL.
- On COCO validation image variation, CLIP-I 0.873 vs 0.858 (IP-Adapter), with FID 14.557 vs 14.016 for RIVAL (best FID).
- Training-free method outperforms IP-Adapter on CLIP-I across SD1.5 (0.883 vs 0.870), SD2.1 and SDXL (0.865 vs 0.859).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use image prompts only as positive guidance, excluding them from negative guidance (ablation) | self-attention image-prompting methods · Image prompt alignment | CLIP score | Helped |
| Use Stratified Attention instead of KV concatenation (ablation) | Ours · Image prompt alignment | attention-score difference | Helped |
| Use longer time steps with Stratified Attention (ablation) | Ours · Image prompt alignment | attention-score difference | Helped |
| Use Stratified Attention to balance generated-image and image-prompt attention | Ours · Cross-prompt image generation | CLIP alignment with image and text prompts | Helped |
| Use Stratified Attention across time steps rather than relying on KV replacement | Ours · Image variation | CLIP-I | Helped |
Surprising findings
- Using the image prompt as negative guidance in self-attention modification actively pushes generations away from the image prompt, as shown by generating with negative prompts without guidance.
Limitations
- Output quality depends on the underlying diffusion model; poor hand generation persists.
- RIVAL has lower CLIP-I scores and some baselines achieve better FID or DINO-G than the proposed method on certain datasets.
Experimental setup
Datasets: COCO 2017 ImageNet-R-TI2I CelebAHQ
Benchmarks: COCO 2017 validation CelebAHQ ImageNet-R-TI2I
Metrics: CLIP-I CLIP-T DINO-G DINO-C FID
Baselines: IP-Adapter SSR-Encoder RIVAL StyleAligned
Builds on: RIVAL classifier-free guidance StyleID DDIM inversion ControlNet
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.89)
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.89)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.