Research map / Image generation / Text-to-image diffusion
Aligning Text to Image in Diffusion Models is Easier Than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoftREPA, a contrastive fine-tuning method that trains fewer than 1M learnable soft text tokens on frozen SD1.5/SDXL/SD3 to improve text-image alignment in generation and editing.
Problem
Residual misalignment between text and image representations in text-to-image diffusion and flow models, where standard denoising or flow matching only uses positive pairs and existing fixes need preference datasets or full fine-tuning.
Approach
Learn layer- and timestep-indexed soft text tokens with a contrastive score-matching loss that uses in-batch mismatched captions as negatives, with the logit set to the exponentiated negative denoising error; the authors argue this increases a mutual-information lower bound between image and text.
Key results
- SD3 + SoftREPA reaches 0.70 mean on GenEval versus 0.68 for SD3 and 0.74 for RankDPO, with the best Two-object, Colors, Position and Color Attribution scores but a drop on Counting (0.29 vs 0.56).
- On COCO val5K, SD1.5 ImageReward rises from 17.72 to 32.89 and CLIP from 26.4 to 27.33 with SoftREPA, with peak memory unchanged at 2.621 GB.
- With soft tokens added to FlowEdit on SD3, PIEBench ImageReward improves from 87.70 to 102.24 and CLIP/Edited from 23.19 to 23.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SoftREPA soft tokens | SD1.5 · COCO val5K | CLIP: 26.4 -> 27.33 | Helped |
| Add SoftREPA soft tokens | SDXL · COCO val5K | ImageReward: 75.06 -> 85.29 | Helped |
| Add SoftREPA soft tokens | SD3 · GenEval | Counting: 0.56 -> 0.29 | Hurt |
| Add SoftREPA soft tokens | SD3 · GenEval | Mean: 0.68 -> 0.70 | Helped |
| Add SoftREPA soft tokens to FlowEdit | FlowEdit on PIEBench · PIEBench | ImageReward: 87.70 -> 102.24 | Helped |
| Integrate pretrained SoftREPA tokens at inference | Diffusion-DPO · COCO-val 1K | ImageReward: 22.26 -> 33.10 | Helped |
| Replace soft tokens with LoRA rank 4 on layers 1-5 (ablation) | SD3 with contrastive loss · COCO-val 1K | ImageReward: 106.32 -> 97.72 | Hurt |
| Limit soft tokens from layers 1-5 to layers 1-2 (ablation) | SD3 + SoftREPA · GenEval | Counting: 0.29 -> 0.50 | Helped |
| Add object-counting loss (ablation) | SD3 + SoftREPA, layers 1-5 · GenEval | Counting: 0.29 -> 0.59 | Helped |
| Use 7 rather than 4 soft-token layers, with 8 tokens (ablation) | SD3 + SoftREPA · COCO val 1K | ImageReward: 1.009 -> 0.944 | Hurt |
| Increase token count from 4 to 16 across 5 layers (ablation) | SD3 + SoftREPA · COCO val 1K | ImageReward: 1.063 -> 0.801 | Hurt |
| Remove denoising score matching loss (ablation) | SD1.5 + SoftREPA · COCO-val 5K | ImageReward: 32.89 -> 32.63 | No clear effect |
Surprising findings
- Applying soft tokens across more layers degrades object counting on GenEval (counting 0.29 with layers 1-5 versus 0.56 for vanilla SD3), which the authors attribute to overemphasis on text cues.
- Soft tokens outperform a LoRA baseline with a comparable parameter count when both use the same contrastive loss.
Limitations
- May overemphasize textual guidance at the expense of faithfully preserving user intent.
- Counting accuracy drops on GenEval when soft tokens are applied to many layers.
- Reported results are mostly single runs without error bars or statistical significance tests.
Experimental setup
Datasets: COCO DIV2K PIEBench Cat2Dog GenEval
Benchmarks: COCO val5K COCO val1K GenEval PIEBench DIV2K Cat2Dog
Metrics: ImageReward PickScore CLIP score HPS FID LPIPS GenEval score PSNR SSIM structure distance
Baselines: SD1.5 SDXL SD3 CaPO RankDPO LoRA + contrastive loss Diffusion-DPO DDPO PnP MasaCtrl Direct Inversion FlowEdit
Builds on: REPA CLIP InfoNCE Diffusion-DPO FlowEdit VPT
Closest papers by meaning
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.92)
- Information Theoretic Text-to-Image Alignment (ICLR 2025 · similarity 0.92)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.91)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.90)
- CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.89)
- Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.