Research map / Image generation / Text-to-image diffusion
Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiT-ST, which uses an LLM to parse captions into a hierarchical split-text caption and injects object, relation, and attribute tokens into MM-DiT at SNR- and cross-attention-derived timesteps.
Problem
Diffusion transformers suffer a "complete-text comprehension defect": long, syntactically complex captions cause overlooked details and semantic entanglement because all semantic primitives are modeled simultaneously from the start.
Approach
Convert a complete-text caption into a hierarchical split-text caption (simplified sentences grouped by object, relation, attribute) and inject each primitive type into MM-DiT via cross-attention at different denoising stages, with injection timesteps chosen from cross-attention convergence and the SNR curve's maximum-curvature point.
Key results
- DiT-ST Medium (2B) reaches 0.69 GenEval overall, matching DreamEngine and approaching SDv3.5 Large (0.71).
- On COCO-5K, DiT-ST Medium achieves CLIPScore 34.09 vs 32.74 for SDv3.5 Large and lowest FID 22.11 among caption-form comparisons.
- On COCO-30K, DiT-ST Large obtains FID 17.16 vs 17.31 for SDv3.5 Large; DiT-ST Medium 18.78 vs 18.82 for SDv3 Medium.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use split-text captions instead of original or LLM-enhanced captions (ablation) | MM-DiT 2B-E · COCO-5K | CLIPScore: 31.68 -> 34.09; 32.13 -> 34.09 | Helped |
| Use split-text captions instead of original or LLM-enhanced captions (ablation) | MM-DiT 2B-E · COCO-5K | FID: 24.61 -> 22.11; 24.46 -> 22.11 | Helped |
| Apply hierarchical caption input rather than original or LLM-enhanced captions (ablation) | SDv3 Medium · COCO-5K | CLIPScore: 31.31 -> 32.42; 31.95 -> 32.42 | Helped |
| Apply hierarchical caption input rather than original or LLM-enhanced captions (ablation) | DiT-ST Medium · COCO-5K | FID: 24.61 -> 23.85; 24.43 -> 23.85 | Helped |
| Add incremental primitive injection (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 32.81 -> 34.09 | Helped |
| Add incremental primitive injection (ablation) | DiT-ST Medium · COCO-5K | FID: 23.85 -> 22.11 | Helped |
| Use O-R-A rather than A-R-O, A-O-R, R-O-A, R-A-O, or O-A-R injection order (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 29.41; 31.10; 32.52; 30.16; 30.79 -> 34.09 | Helped |
| Change object injection from default timing to timestep 10 or 20 (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 32.71; 32.24 vs 34.09 | Hurt |
| Change relation injection to timestep 30, 40, 60, or 70 (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 32.57; 32.73; 32.76; 32.45 vs 34.09 | Hurt |
| Change attribute injection to timestep 200, 300, 500, or 600 (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 31.26; 31.75; 32.96; 32.83 vs 34.09 | Hurt |
| Set the sliding-window size to 1, 2, 4, or 5 instead of 3 (ablation) | DiT-ST Medium · COCO-5K | CLIPScore: 33.16; 33.55; 33.72; 33.43 vs 34.09 | Hurt |
| Use DiT-ST Medium instead of SDv3 Medium | Text-to-image generation · GenEval | overall accuracy: 0.62 -> 0.69 | Helped |
Surprising findings
- Hierarchical split-text input alone, with no fine-tuning, raises SDv3 Medium's CLIPScore from 31.31 to 32.42.
- Injecting attribute tokens early (t=200) causes the largest CLIPScore drop (about 8%) in the step-selection ablation.
Limitations
- Gains on COCO-30K FID over SDv3 and SDv3.5 are small (about 0.2% and 0.9%).
- Timestep selection is statistical across a batch rather than per-sample, and the paper acknowledges per-sample adaptation is costly.
Experimental setup
Datasets: SAM-LLaVA COCO CC12M ImageNet
Benchmarks: GenEval COCO-5K COCO-30K
Metrics: GenEval overall accuracy CLIPScore FID gFID
Baselines: MM-DiT 2B-E MM-DiT 8B-E (SDv3.5 Large) SDv3 Medium SDv3.5 Large Flux.1 Dev PixArt-alpha DreamEngine DALL-E 3
Builds on: MM-DiT Stable Diffusion 3 PixArt-alpha CLIP T5
Closest papers by meaning
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.89)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.88)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.