Research map / Image generation / Text-to-image diffusion
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically compares 12 text encoders (T5, CLIP, decoder-only LLMs, and LLM-based embedding models) in a controlled SD2-style U-Net pipeline, finding that layer-normalized averaging across all LLM layers beats last-layer embeddings and T5.
Problem
Most text-to-image diffusion models use outdated encoders like T5 and CLIP, and it is unclear whether decoder-only LLMs, and how to extract embeddings from them, improve text-image alignment, since prior works trained on different proprietary data and setups.
Approach
Replace only the text encoder in a fixed Stable Diffusion v2-style U-Net training pipeline and compare embedding extraction strategies; the key finding is that averaging layer-normalized hidden states across all LLM layers, rather than using the last layer, yields better compositional alignment than T5.
Key results
- Mistral-7B with layer-normalized average embeddings reaches 0.769 VQAScore overall vs 0.675 for its last layer and 0.741 for T5-XXL.
- bge-Gemma2 with layer-normalized average embeddings is the best model at 0.789 average VQAScore, outperforming T5-XXL (0.741).
- Using only the 15th layer of Mistral-7B (0.725) beats the last layer (0.675) but remains below T5-XXL (0.741).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use layer-normalized average of all layers instead of last-layer embeddings | Mistral-7B · GenAI-Bench | VQAScore Overall Average: 0.675 -> 0.769 | Helped |
| use layer-normalized average of all layers instead of last-layer embeddings | bge-Gemma2 · GenAI-Bench | VQAScore Overall Average: 0.737 -> 0.789 | Helped |
| use last-layer embeddings from decoder-only LLMs instead of T5-XXL | Mistral-7B, Llama3, Qwen2, Gemma2 and embedding models · GenAI-Bench | VQAScore Overall Average: e.g. Mistral-7B 0.675 vs T5-XXL 0.741 | Hurt |
| use a single middle layer (15th) instead of last layer (ablation) | Mistral-7B · GenAI-Bench | VQAScore Overall Average: 0.675 -> 0.725 | Helped |
| use first layer (layer 0) embeddings (ablation) | Mistral-7B · GenAI-Bench | VQAScore Overall Average: 0.375 vs 0.675 last layer | Hurt |
| plain average across layers without layer normalization (ablation) | Mistral-7B · GenAI-Bench | VQAScore Overall Average: 0.675 -> 0.731 (norm avg 0.769 is better still) | Helped |
| apply layer normalization before averaging vs plain averaging (ablation) | Mistral-7B and bge-Gemma2 · GenAI-Bench | VQAScore Overall Average: Mistral 0.731 -> 0.769; bge-Gemma2 0.774 -> 0.789 | Helped |
| layer-normalized averaging applied to T5 embeddings (ablation) | T5-XXL · GenAI-Bench | VQAScore Overall Average: 0.741 -> 0.747 | No clear effect |
| add pooled global embedding conditioning (SDXL-style) to last-layer embeddings (ablation) | bge-Gemma2 and sfr-Mistral · GenAI-Bench | VQAScore Overall Average: bge-Gemma2 0.737 -> 0.737; sfr-Mistral 0.710 -> 0.698 | No clear effect |
| fine-tune LLM into embedding model (bge-Gemma2, sfr-Mistral vs gte-Qwen2) | Gemma2-9B, Mistral-7B, Qwen2-7B · GenAI-Bench | VQAScore Overall Average: bge-Gemma2 0.737 vs Gemma2 0.710 (up); sfr-Mistral 0.710 vs Mistral-Instruct/base 0.675 (up); gte-Qwen2 0.482 vs Qwen2 0 | Mixed |
| scale LLM size (Qwen2 1.5B to 7B, last layer) (ablation) | Qwen2 · GenAI-Bench | VQAScore Overall Average: 0.655 -> 0.683 | Helped |
| scale LLM size (Gemma2 2B to 9B, last layer) (ablation) | Gemma2 · GenAI-Bench | VQAScore Overall Average: 0.640 -> 0.710 | Helped |
Surprising findings
- Last-layer embeddings from decoder-only LLMs underperform T5 mainly on visio-linguistic reasoning skills, while layer-normalized averaging applied to T5 gives almost no change.
- Guidance weight shifts VQAScore considerably, potentially more than the choice of text encoder.
Limitations
- Ablations use 256x256 resolution and a single U-Net architecture trained on 46M pairs, so results may not transfer to larger models or resolutions.
- Comparison and size-related reasoning remain weak; generated images do not correctly render relative size.
- VQAScore relies on GPT-4o as judge, with up to ±0.003 random variation.
Experimental setup
Datasets: LAION-Aesthetics GenAI-Bench
Benchmarks: GenAI-Bench
Metrics: VQAScore
Baselines: CLIP-ViT-H/14 T5-XXL Playground-v3 Lumina-T2X Sana
Builds on: Stable Diffusion VQAScore SDXL pooled conditioning
Closest papers by meaning
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.93)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.92)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.91)
- Scaling Down Text Encoders of Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.90)
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.