Research map / Image generation / Text-to-image diffusion
Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Merging (ToMe), a training-free method that sums CLIP token embeddings of each object with its attributes/sub-objects into a composite token, plus end token substitution and two auxiliary losses applied early in SDXL denoising, to improve semantic binding in text-to-image generation.
Problem
Text-to-image diffusion models often fail to bind objects to their attributes and related sub-objects (semantic binding), and prior fixes require fine-tuning or LLM/layout specification.
Approach
Merge each object token with its related attribute/sub-object tokens by element-wise summation of CLIP text embeddings into one composite token sharing a single cross-attention map, replace the EOT tokens with those from a subject-only prompt (ETS), and iteratively update the composite token in the first 20% of denoising using an entropy loss and a semantic binding loss.
Key results
- On T2I-CompBench with SDXL, ToMe reaches BLIP-VQA of 0.7656 (color), 0.6894 (texture), 0.6051 (shape), outperforming SDXL base (0.6369, 0.5637, 0.5408) and ELLA/CoMat on shape.
- On the proposed 50-prompt GPT-4o object binding benchmark, ToMe scores 0.9549 versus 0.6481 for ELLA and 0.4907 for SDXL.
- Inference with 50 steps takes 83s versus 42s for SDXL on a single A40 GPU, a roughly two-fold overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ToMe to SDXL | SDXL · T2I-CompBench color, texture, and shape | BLIP-VQA: Color: 0.7774 vs 0.7656; texture: 0.6686 vs 0.6894; shape: 0.5634 vs 0.6051 | Mixed |
| Apply ToMe to SDXL | SDXL · GPT-4o object binding benchmark | GPT-4o score: 0.6481 -> 0.9549 | Helped |
| Add token merging and ETS (ablation) | SDXL baseline · T2I-CompBench color, texture, and shape | BLIP-VQA: Color: 0.6369 -> 0.6577; texture: 0.5637 -> 0.5828; shape: 0.5408 -> 0.5437 | Helped |
| Add entropy loss to token merging and ETS (ablation) | Config B · T2I-CompBench color, texture, and shape | BLIP-VQA: Color: 0.6577 -> 0.7525; texture: 0.5828 -> 0.6775; shape: 0.5437 -> 0.5797 | Helped |
| Add semantic binding loss to Config C (ablation) | Config C · T2I-CompBench color, texture, and shape | BLIP-VQA: Color: 0.7525 -> 0.7656; texture: 0.6775 -> 0.6894; shape: 0.5797 -> 0.6051 | Helped |
| Remove entropy loss from full ToMe (ablation) | ToMe full · T2I-CompBench color, texture, and shape | BLIP-VQA: Color: 0.7656 -> 0.6804; texture: 0.6894 -> 0.6263; shape: 0.6051 -> 0.5645 | Hurt |
| Use optimization losses without token merging (ablation) | SDXL · T2I-CompBench color, texture, and shape | BLIP-VQA: Config D: 0.5881, 0.6194, 0.5386; Config E: 0.5983, 0.5798, 0.5125; baseline: 0.6369, 0.5637, 0.5408 | Hurt |
| Replace original end tokens with ETS (ablation) | ToMe · Object binding prompts | semantic binding | Helped |
| Splice token embeddings from different prompts instead of the same prompt (ablation) | ToMe · Text-to-image generation | object presence | Hurt |
Surprising findings
- Naively adding the CLIP embeddings of a dog and a hat yields images containing hats (68.61% DetScore), close to the full prompt 'a dog wearing a hat' (73.12%), suggesting embedding addition is semantically meaningful.
- Splicing token embeddings taken from different prompts often causes the object to disappear from the generated image.
Limitations
- Inherits SDXL limitations such as artifacts and inability to produce complex layouts.
- Relies on the CLIP text encoder, which may not capture subtle semantic nuances.
- Evaluation on the GPT-4o benchmark relies on a judge model and a self-constructed 50-prompt set.
Experimental setup
Datasets: T2I-CompBench
Benchmarks: T2I-CompBench GPT-4o Benchmark
Metrics: BLIP-VQA ImageReward GPT-4o score DetScore
Baselines: SDXL Playground-v2 Ranni ELLA SynGen CoMat
Builds on: SDXL Attend-and-Excite SynGen Divide-and-Bind
Closest papers by meaning
- Token-to-Token Alignment of Text Embeddings for Semantic Blending (NeurIPS 2026 · similarity 0.88)
- Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective (ICLR 2025 · similarity 0.87)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.87)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.