Research map / Image generation / Text-to-image diffusion
Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
An empirical study of deep fusion between a frozen decoder-only LLM (Gemma) and a trainable DiT for text-to-image generation, comparing it to shallow-fusion baselines and ablating design choices.
Problem
Deep fusion of LLMs with diffusion transformers for text-to-image synthesis has been reported with system-level results but lacks controlled comparisons with established conditioning baselines, design-space exploration, and reproducible training recipes.
Approach
Systematically compares a deep-fusion architecture (layer-wise shared self-attention between a frozen LLM and a trainable DiT mirroring its config) against shallow-fusion DiT baselines (self-attention and cross-attention on last-layer text features), and ablates timestep conditioning, positional encoding, base LLM choice, and architecture alignment.
Key results
- Deep fusion reaches GenEval 0.51 and DPG 76.6 vs. self-attention DiT 0.42/73.9 and cross-attention DiT 0.49/76.3, while shallow fusion has better FID.
- Removing timestep conditioning cuts parameters ~20% and improves FID from 27.33 to 21.27 with comparable GenEval (0.49).
- Final recipe with Gemma 2 2B reaches GenEval 0.60, DPG 81.6, FID 7.54 on 26M training pairs, trained for 800K steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deep fusion instead of shallow self-attention fusion | Gemma 2B + DiT text-to-image model · GenEval | score: 0.42 -> 0.51 | Helped |
| Use deep fusion instead of shallow cross-attention fusion | Gemma 2B + DiT text-to-image model · MJHQ-30K | FID: 24.00 -> 27.33 | Hurt |
| Remove timestep conditioning (ablation) | Deep Fusion · MJHQ-30K | FID: 27.33 -> 21.27 | Helped |
| Remove timestep conditioning (ablation) | Deep Fusion · DPG-Bench | score: 76.6 -> 76.7 | No clear effect |
| Add pooled CLIP L/14 text conditioning to adaLN-Zero (ablation) | Deep Fusion · GenEval | score: 0.51 -> 0.50 | Hurt |
| Replace adaLN-Zero with adaLN-Single (ablation) | Deep Fusion · GenEval | score: 0.51 -> 0.47 | Hurt |
| Replace adaLN-Zero with additive timestep conditioning (ablation) | Deep Fusion · MJHQ-30K | FID: 27.33 -> 26.40 | Helped |
| Use separate text 1D-RoPE and image 2D-RoPE instead of 1D-RoPE + APE (ablation) | Deep Fusion · MJHQ-30K | FID: 27.33 -> 25.42 | Helped |
| Use 1D-RoPE for both text and image sequences (ablation) | Deep Fusion · GenEval | score: 0.51 -> 0.46 | Hurt |
| Replace separate text 1D-RoPE and image 2D-RoPE with M-RoPE (ablation) | Deep Fusion · DPG-Bench | score: 76.4 -> 74.9 | Hurt |
| Use an instruction-tuned LLM (ablation) | Deep Fusion · GenEval | score: 0.51 -> 0.49 | Hurt |
| Add the instruction prompt “Imagine: ” (ablation) | Instruction-tuned Deep Fusion · GenEval | score: 0.49 -> 0.50 | Helped |
Surprising findings
- Removing timestep conditioning entirely gives comparable alignment and much better FID than adaLN-Zero.
- Instruction-tuned Gemma 2B IT slightly hurts text-to-image performance compared with the base model.
Limitations
- Final model trained with limited compute and data (26M pairs), below industry-scale datasets.
- Mixed-modal positional encodings and architecture alignment remain unresolved; M-RoPE underperforms.
Experimental setup
Datasets: CC12M SA-1B JourneyDB MJHQ-30K
Benchmarks: GenEval DPG-Bench MJHQ-30K FID
Metrics: GenEval score DPG score FID inference latency
Baselines: self-attention DiT cross-attention DiT SD 3 M FLUX.1 [dev] PixArt-Σ SDXL Sana 1.0 1.6B Transfusion PG 3
Builds on: diffusion transformer (DiT) rectified flow MM-DiT PixArt-alpha
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.92)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.91)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.91)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.91)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.91)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.