Research map / Image generation / Text-to-image diffusion
Towards Transformer-Based Aligned Generation with Self-Coherence Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Coherence Guidance, a training-free method that refines cross-attention maps in PixArt-α during denoising using masks from the next-step attention, improving attribute and style binding.
Problem
Transformer-based text-guided diffusion models (e.g., DiT/PixArt-α) often misalign generated images with complex prompts, including coarse-grained attribute binding, fine-grained part-level attribute binding, and style binding; U-Net-based training-free latent-optimization fixes transfer poorly.
Approach
Instead of optimizing latents with losses computed from cross-attention (U-Net methods), directly re-weight the cross-attention maps of Transformer blocks using concept masks extracted from the averaged attention maps of the next denoising step (K-means clustering, or LLM-planned proportions for fine-grained binding).
Key results
- On BLIP-VQA, the method scores 0.623 on fine-grained binding versus 0.398 for PIXART-α, a roughly 56% improvement.
- In a 50-participant user study, participants chose the method's images 81%, 90.6%, and 92.6% of the time for coarse, fine, and style binding.
- Directly transferring U-Net methods D&B and CONFORM to PIXART-α gives limited gains, with CONFORM's image-text similarity at 0.36 versus 0.36 for the original PIXART-α.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Coarse-grained attribute binding | text-text similarity: 0.808 -> 0.852 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Fine-grained attribute binding | text-text similarity: 0.804 -> 0.827 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Style binding | text-text similarity: 0.688 -> 0.689 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Coarse-grained attribute binding | BLIP-VQA score: 0.293 -> 0.679 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Fine-grained attribute binding | BLIP-VQA score: 0.398 -> 0.623 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Style binding | BLIP-VQA score: 0.676 -> 0.799 | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Coarse-grained attribute binding | user preference: 2.6% -> 81% | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Fine-grained attribute binding | user preference: 4.2% -> 90.6% | Helped |
| apply Self-Coherence Guidance by directly optimizing cross-attention maps | PIXART-α-512 · Style binding | user preference: 5.6% -> 92.6% | Helped |
| directly transfer CONFORM to the Transformer-based model | PIXART-α · Original objects binding benchmark | image-text similarity: 0.36 -> 0.36 | No clear effect |
| directly transfer D&B to the Transformer-based model | PIXART-α · Original objects binding benchmark | image-text similarity: 0.36 -> 0.35 | Hurt |
Surprising findings
- On BLIP-VQA, the original PIXART-α scores lower than U-Net-based SOTA methods D&B and CONFORM on coarse and fine binding.
- Directly transferring U-Net-based training-free methods to Transformer models yields little or no improvement in image-text similarity.
Limitations
- Fine-grained binding relies on LLM-based proportion planning, which adds an external dependency.
- Style-binding gain over PIXART-α on text-text similarity is marginal, attributed to BLIP captioning limits.
- Benchmarks are manually constructed and relatively small (prompt-based evaluation with 64 seeds per prompt).
Experimental setup
Datasets: prompts from prior attribute-binding benchmark (Chefer et al.)
Benchmarks: coarse-grained attribute binding prompts fine-grained attribute binding prompts style binding prompts
Metrics: text-text similarity (BLIP caption vs prompt, CLIP) BLIP-VQA image-text similarity user study preference
Baselines: PIXART-α D&B CONFORM
Builds on: Prompt-to-Prompt Divide & Bind CONFORM PixArt-α
Closest papers by meaning
- Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation (ICML 2025 · similarity 0.88)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.88)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.88)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.