Research map / Image generation / Text-to-image diffusion
TokenCompose: Text-to-Image Diffusion with Token-level Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes Stable Diffusion with token-level cross-attention losses supervised by Grounded-SAM segmentation masks of noun tokens to improve multi-category instance composition, and introduces the MultiGen benchmark.
Problem
Text-to-image latent diffusion models often fail to compose multiple object categories from a text prompt, especially categories that rarely co-occur in real scenes, because the denoising objective gives no token-level constraint between prompt nouns and image regions.
Approach
Adds token-level (aggregated attention inside masks) and pixel-level (binary cross-entropy on attention maps) losses over cross-attention maps for each noun token, with masks automatically obtained from a POS tagger and Grounded SAM, applied as a training-only finetuning objective with no inference cost.
Key results
- Object Accuracy on VISOR rises from 29.86 (Stable Diffusion 1.4) to 52.15 with TokenCompose.
- MG5 on COCO instances rises from 0.88 to 3.28 and MG4 from 11.68 to 28.81; COCO FID stays about 20.19 vs 20.88 baseline.
- Mean IoU on COCO-Gen segmentation improves from 0.5371 (frozen SD 1.4) to 0.5876.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune Stable Diffusion 1.4 with token- and pixel-level grounding losses | Stable Diffusion 1.4 · VISOR | Object Accuracy: 52.15 vs 29.86 | Helped |
| finetune Stable Diffusion 1.4 with token- and pixel-level grounding losses | Stable Diffusion 1.4 · MULTIGEN COCO Instances MG3 | success rate: 76.161.04 vs 50.740.89 | Helped |
| finetune Stable Diffusion 1.4 with token- and pixel-level grounding losses | Stable Diffusion 1.4 · COCO validation | FID: 20.19 vs 20.88 | Helped |
| finetune Stable Diffusion 1.4 with token- and pixel-level grounding losses | Stable Diffusion 1.4 · Image generation | latency: 7.560.14 vs 7.540.17 | No clear effect |
| add token and pixel grounding losses (ablation) | Stable Diffusion 1.4 finetuned with denoising loss · MULTIGEN COCO Instances MG3 | success rate: 63.211.73 -> 76.161.04 | Helped |
| add token grounding loss (ablation) | Stable Diffusion 1.4 finetuned with denoising loss · MULTIGEN COCO Instances MG3 | success rate: 71.611.06 -> 76.161.04 | Helped |
| add pixel grounding loss (ablation) | Stable Diffusion 1.4 finetuned with denoising loss · MULTIGEN COCO Instances MG3 | success rate: 61.951.05 vs 63.211.73 | No clear effect |
| remove pixel grounding loss (ablation) | TokenCompose · MULTIGEN COCO Instances MG3 | success rate: 73.801.33 -> 76.161.04 | Hurt |
| remove token grounding loss (ablation) | TokenCompose · MULTIGEN COCO Instances MG3 | success rate: 71.611.06 -> 76.161.04 | Hurt |
| add grounding objectives to middle block and decoder layers (ablation) | Stable Diffusion 1.4 · MULTIGEN COCO Instances MG3 | success rate: 73.801.33 -> 76.161.04 | Helped |
| add grounding objectives to encoder layers (ablation) | TokenCompose · VISOR | Object Accuracy: 52.15 vs 42.92 | Hurt |
| add grounding objectives to Stable Diffusion 2.1 | Stable Diffusion 2.1 · MULTIGEN COCO Instances MG3 | success rate: 70.14 -> 80.48 | Helped |
Surprising findings
- Training with only the denoising loss and the token loss leads to unstable inference and over-concentrated attention, so the pixel-level loss is needed to counteract it.
- Attend-and-Excite reaches a high MG5 success rate despite lower MG2-4 rates, which the authors attribute to training distributions lacking 5-category captions.
Limitations
- Supervision is applied only to noun tokens; adjectives, verbs and determiners are not covered.
- Training uses a small dataset of about 4.5k image-caption pairs.
Experimental setup
Datasets: COCO Visual Spatial Reasoning Flickr30K Entities ADE20K COCO-Gen (DAAM)
Benchmarks: VISOR MultiGen T2I-CompBench COCO FID Flickr30K FID COCO-Gen mIoU (DAAM)
Metrics: object accuracy MG2-5 success rate FID latency mIoU attribute binding score
Baselines: Stable Diffusion 1.4 Composable Diffusion Layout Guidance Diffusion Structured Diffusion Attend-and-Excite
Builds on: Latent Diffusion Model Layout Guidance Diffusion Grounded SAM Attend-and-Excite
Closest papers by meaning
- RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.89)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.89)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.89)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.