Research map / Image generation / Text-to-image diffusion
LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM4GEN, a plug-and-play Cross-Adapter Module that fuses LLM (T5-XL) hidden states with CLIP text embeddings for Stable Diffusion, plus an entity-guided attention regularization loss and a 7,000-prompt DensePrompts benchmark.
Problem
Diffusion text-to-image models struggle with complex, dense prompts involving multiple objects, attribute binding, and long descriptions, because CLIP text encoders have limited semantic representation.
Approach
Implicitly inject LLM semantic representations into the original CLIP text embedding through a cross-attention adapter (LLM features as queries, CLIP features as keys/values) whose output is concatenated residually with the CLIP embedding, avoiding large-scale alignment training; add an entity-attribute attention regularization loss.
Key results
- On T2I-CompBench, LLM4GEN_SDXL reaches 76.59 color vs 63.69 for SDXL (+12.90), and LLM4GEN_SD1.5 reaches 47.34 color vs 37.65 for SD1.5 (+9.69).
- On MSCOCO subset, LLM4GEN_SD1.5 improves FID from 26.89 to 25.20 and LLM4GEN_SDXL improves FID from 24.75 to 24.21.
- User study: LLM4GEN_SDXL receives 60.3% and 66.5% higher voting preference than SDXL for aesthetics and semantics, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LLM4GEN | SD1.5 · T2I-CompBench | Color: 37.65 -> 47.34 | Helped |
| Add LLM4GEN | SDXL · T2I-CompBench | Color: 63.69 -> 76.59 | Helped |
| Add LLM4GEN | SD1.5 · MSCOCO subset | FID: 26.89 -> 25.20 | Helped |
| Add LLM4GEN | SDXL · MSCOCO subset | CLIP Score: 30.10 -> 30.91 | Helped |
| Use Llama-2/7B as the LLM (ablation) | LLM4GEN on SD1.5 · T2I-CompBench | Color: 37.65 -> 43.21 | Helped |
| Use Llama-2/13B rather than Llama-2/7B (ablation) | LLM4GEN on SD1.5 · T2I-CompBench | Color: 43.21 -> 44.98 | Helped |
| Use T5-XL rather than Llama-2/13B (ablation) | LLM4GEN on SD1.5 · T2I-CompBench | Color: 44.98 -> 47.34 | Helped |
| Fine-tune the original CLIP text encoder (ablation) | SD1.5 · T2I-CompBench | Color: 37.65 -> 38.76 | Helped |
| Transform LLM embeddings with MLP or CrossAttention (ablation) | SD1.5 · T2I-CompBench | Color: 37.65 -> 39.25 | Helped |
| Project LLM embeddings and concatenate them with text embeddings (ablation) | SD1.5 · T2I-CompBench | Color: 37.65 -> 40.24 | Helped |
| Use CrossAttention and concatenate embeddings (ablation) | SD1.5 · T2I-CompBench | Color: 37.65 -> 42.31 | Helped |
| Swap CAM roles so CLIP is Q and LLM is K and V (ablation) | SD1.5 · T2I-CompBench | Color: 37.65 -> 43.69 | Helped |
Surprising findings
- Simply fine-tuning the SD1.5 CLIP text encoder yields only modest gains, while concatenating LLM-derived embeddings gives a larger boost, suggesting direct alignment of LLM representations to the latent space is harder than enhancing text embeddings.
Limitations
- Relies on training data collected from LAION-2B and internet, with a 10M-pair training set whose scaling saturates beyond 5M-10M.
- DensePrompts is built with GPT-4V and GPT-4 generated text with manual refinement, and evaluated with CLIP and aesthetic scores that have known limitations for compositional alignment.
Experimental setup
Datasets: LAION-2B MSCOCO T2I-CompBench PartiPrompts
Benchmarks: T2I-CompBench MSCOCO DensePrompts PartiPrompts
Metrics: Color Shape Texture Spatial Non-Spatial Complex FID IS CLIP Score Aesthetic Score
Baselines: Composable Diffusion Structured Diffusion Attn-Exct v2 GORS DALL-E 2 PixArt-α SUR-Adapter ELLA SD1.5 SDXL Playground v2 ParaDiffusion
Builds on: Stable Diffusion CLIP T5 T2I-CompBench DDIM
Closest papers by meaning
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.92)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.91)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.90)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.90)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.