Research map / Image generation / Text-to-image diffusion
Improving Long-Text Alignment for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongAlign, combining segment-level CLIP/T5 text encoding with decomposed CLIP-based preference optimization (orthogonal split into text-relevant and text-irrelevant parts with gradient reweighting) to fine-tune Stable Diffusion v1.5 for long-text alignment.
Problem
Text-to-image diffusion models struggle to encode and align with long, multi-sentence prompts because CLIP text encoders have a 77-token limit and existing human preference models blend text alignment with aesthetics.
Approach
Encode long text segment-by-segment and concatenate embeddings; decompose CLIP-based preference scores into a text-relevant part (orthogonal to a common text-irrelevant direction) and a text-irrelevant part, then reweight the latter (factor omega=0.3) in reward fine-tuning to reduce overfitting.
Key results
- longSD (S+R) reaches FID-5k 19.63 and Denscore-O 35.26 (with reweighting), outperforming PixArt-alpha (22.36 FID, 33.48 Denscore-O) and Kandinsky v2.2 (20.04 FID, 33.30 Denscore-O) on the 5k evaluation set.
- Denscore segment-level model achieves R@1 of 83.96 single vs 91.86 average with text-irrelevant-free embedding, improving over CLIP-H's 86.10/80.40 in the average setting.
- On DPG-Bench, longSD achieves an average of 77.58, above PlayGroundv2 (74.54) and ELLA 1.5 (74.91).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LongAlign reward fine-tuning after supervised fine-tuning (ablation) | Stable Diffusion v1.5 · 5k-image evaluation dataset | Denscore-O: 31.29 -> 32.83/35.26 | Helped |
| Add LongAlign reward fine-tuning after supervised fine-tuning (ablation) | Stable Diffusion v1.5 · 5k-image evaluation dataset | FID-5k: 20.09 -> 19.63/24.28 | Helped |
| Use gradient reweighting with ω=0.3 (ablation) | Reward fine-tuning of the diffusion model · Reward fine-tuning experiments | FID | Helped |
| Use gradient reweighting with ω=0.3 rather than the other tested weighting options (ablation) | Reward fine-tuning of the diffusion model · Reward fine-tuning experiments | Denscore-O | Helped |
| Remove the text-irrelevant component from the text embedding score (ablation) | CLIP-based preference models · 5k text-to-image retrieval, Average | R@1: 75.90 -> 91.86 | Helped |
| Use segment-level preference training (ablation) | Denscore · 5k text-to-image retrieval, 8-sentence maximum | R@1: 84.12 -> 89.94 | Helped |
| Use segment-level preference training (ablation) | Denscore · 5k text-to-image retrieval, 1-sentence maximum | R@1: 53.06 -> 52.72 | Hurt |
| Keep all <sot> token embeddings instead of removing all of them (ablation) | Segment-level CLIP text encoding for diffusion · Image generation | generation quality | Helped |
| Retain all <sot> tokens instead of retaining only the first <sot> token (ablation) | Segment-level CLIP text encoding for diffusion · Image generation | generation quality | No clear effect |
| Group consecutive sentences into a segment instead of using each sentence as a segment (ablation) | Segment-level CLIP text encoding for diffusion · Image generation | generation quality | No clear effect |
| Use CLIP+T5 encoding instead of CLIP-cat or T5-mlp alone (ablation) | Stable Diffusion text encoding · Long-text image generation | FID | Helped |
| Reweight the text-irrelevant reward component instead of using the original loss at ratio 1 (ablation) | LongAlign reward fine-tuning · Image generation | image quality | Helped |
Surprising findings
- Original CLIP's text-irrelevant direction is poorly trained, so CLIP reward fine-tuning is ineffective, while preference models' text-irrelevant component drives overfitting.
- Removing all <sot> token embeddings in concatenated segment encodings destroys generation, while keeping all <sot> tokens works best.
Limitations
- Does not fully address long prompts with complex contextual dependencies or requiring strong semantic understanding, partly due to CLIP constraints.
- Reweighting factor omega's optimal value varies by model and training strategy.
- Relies on CLIP-based preference models rather than stronger reward models.
Experimental setup
Datasets: SAM COCO2017 LAION/CC/SBU (LLaVA subset) JourneyDB Pickscore training data
Benchmarks: 5k evaluation set (FID, Denscore, Denscore-O, VQAscore) DPG-Bench GPT-4o pairwise evaluation on 1k images
Metrics: FID Denscore Denscore-O VQAscore R@1 retrieval accuracy win rate
Baselines: SD-1.5 SD-2.1 PlayGroundv2 PixArt-alpha Kandinsky v2.2 ELLA
Builds on: DRTune LaVi-Bridge Pickscore LoRA LCM-LoRA
Closest papers by meaning
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.92)
- Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.91)
- A Dense Reward View on Aligning Text-to-Image Diffusion with Preference (ICML 2024 · similarity 0.90)
- Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences Through f-Divergence Minimization (AAAI 2025 · similarity 0.90)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.90)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.