Research map / Large language models / Language, safety and interpretability
Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that decoder-only LLMs degrade diffusion prompt following and proposes LLM-infused Diffuser (instruction insertion, bidirectional linguistic token refiner, collaborative refiner over multiple LLMs), instantiated as LI-DiT-1B/10B.
Problem
Decoder-only LLMs used directly as text encoders for diffusion models degrade prompt following, due to next-token-prediction misalignment and causal-attention positional bias that hurts later prompt tokens.
Approach
Insert an instruction before the prompt so the LLM focuses on image-relevant attributes, then pass LLM hidden states through a full-attention (non-causal) timestep-gated linguistic token refiner, and fuse multiple LLMs' outputs with a collaborative refiner using cross-attention across branches.
Key results
- LI-DiT-10B reaches 0.76 GenEval overall and 84.60 DPG-Bench average, above SD3-8B (0.74) and DALL-E 3 (0.67 GenEval, 83.50 DPG).
- Ablation: removing the causal mask via a full-attention token refiner raises LLaMA3-8B T2I-avg from 38.72 to 54.02 and DPG-avg from 66.15 to 77.08.
- Human evaluation on 200 prompts shows LI-DiT-10B preferred over SD3 and DALL-E 3 for both alignment and quality, and leading alignment vs Midjourney V6 with similar quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LI-DiT-1B instead of open-source comparison models | text-to-image generation · T2I-CompBench and DPG-Bench | alignment score | Helped |
| Scale the proposed model from LI-DiT-1B to LI-DiT-10B | LI-DiT · T2I-CompBench, GenEval, and DPG-Bench | alignment score | Mixed |
| Add an image-description instruction (ablation) | base LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 38.72 -> 38.47 | No clear effect |
| Use multimodal instruction fine-tuning (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 38.72 -> 44.22 | Helped |
| Add an image-description instruction to multimodal instruction-tuned LLaMA3 (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 44.22 -> 48.41 | Helped |
| Add a linguistic token refiner (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 38.72 -> 54.02 | Helped |
| Add a linguistic token refiner with full attention (ablation) | T5 encoder · T2I-CompBench | T2I-avg: 48.82 -> 49.52 | No clear effect |
| Use causal attention in the token refiner instead of full attention (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 54.02 -> 45.84 | Hurt |
| Increase token-refiner blocks from 1 to 2 (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 48.25 -> 54.02 | Helped |
| Increase token-refiner blocks from 2 to 3 (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 54.02 -> 55.13 | Helped |
| Remove gating from the token refiner (ablation) | LLaMA3-8B prompt encoder · T2I-CompBench | T2I-avg: 54.02 -> 53.47 | Hurt |
| Fuse LLaMA and Qwen representations with a collaborative refiner instead of concatenation (ablation) | LI-DiT text encoder · T2I-CompBench | T2I-avg: 58.32 -> 60.31 | Helped |
Surprising findings
- Directly using a pretrained LLaMA3-8B as text encoder performs worse than the smaller T5-XL encoder on image-text alignment.
- Removing the causal mask in the refiner hurts the T5 encoder's gains, suggesting the benefit is specific to decoder-only LLMs.
Limitations
- Experiments only use 7-8B LLMs; 13B or 70B LLMs not validated due to compute limits.
- Training data for LI-DiT-10B is internal and billion-scale, limiting reproducibility.
- Potential misleading or false image content is noted as a societal risk; no statistical significance reported.
Experimental setup
Datasets: ImageNet CC12M LAION-5B internal bilingual datasets T2I-CompBench
Benchmarks: T2I-CompBench GenEval DPG-Bench
Metrics: T2I-CompBench attribute scores GenEval overall DPG-Bench average human preference for alignment and quality
Baselines: SD v1.5 SD v2 SD XL SD3-1B DALL-E 2 PixArt-alpha DALL-E 3 SD3-8B Midjourney V6
Builds on: DiT PixArt-alpha Stable Diffusion 3 T5 latent diffusion models
Closest papers by meaning
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.93)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.91)
- Scaling Down Text Encoders of Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.91)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.