Research map / Image generation / Autoregressive and latent generation
Multimodal Promptable Token Merging for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Promptable Token Merging (MPTM), a training-free token merging method that measures token similarity in a prompt-induced space so diffusion U-Nets and diffusion detectors can merge tokens while keeping prompt-relevant information.
Problem
Token merging for attention-based diffusion models relies on token-to-token feature similarity, which ignores prompt semantics and can merge informative tokens (e.g., cat and snow) together, degrading dense prediction and generation quality.
Approach
Projects input tokens onto a multimodal prompt space (a fusion of prompt-encoder tokens from several modalities) and measures cosine similarity there for bipartite soft matching, plus a threshold-based token unmerging scheme that reduces error propagation at high merge ratios.
Key results
- On ImageNet-1k text-to-image with SD v1.5, MPTM at 60% merge ratio gives FID 33.14 and CLIP-R 57.46 versus ToMeSD at 50% (FID 33.15, CLIP-R 49.33), with higher throughput-quality tradeoff.
- DiffusionDet with Swin-B and MPTM (50%) plus negative prompts reaches 41.8 AP on LVIS versus 39.5 for the original DiffusionDet and 34.2 for ToMe (50%).
- On AudioCaps text+image to video+audio, MPTM (60%) achieves SIM-VA 0.241 at 10.23 s/sample versus ToMe (50%) at 0.194 and 12.65 s/sample.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use prompt-induced similarity instead of ToMe similarity for 50% token merging | Stable Diffusion v1.5 · ImageNet-1k | CLIP-R: 49.33±1.78 -> 58.51±0.72 | Helped |
| use prompt-induced similarity instead of ToMe similarity for 50% token merging | Stable Diffusion v1.5 · ImageNet-1k | FID: 33.15±0.07 -> 33.07±0.06 | Helped |
| use prompt-induced similarity instead of ToMe similarity for 50% token merging | DiffusionDet with ResNet-50 · LVIS V1.0 val | AP: 25.1 ± 0.3 -> 29.5 ± 0.2 | Helped |
| add negative prompts (ablation) | DiffusionDet with ResNet-50 and MPTM (50%) · LVIS V1.0 val | AP: 29.5 ± 0.2 -> 30.3 ± 0.3 | Helped |
| use prompt-induced similarity instead of ToMe similarity for 50% token merging | DiffusionDet with Swin-B · LVIS V1.0 val | AP: 34.2 ± 0.8 -> 39.9 ± 0.6 | Helped |
| add negative prompts (ablation) | DiffusionDet with Swin-B and MPTM (50%) · LVIS V1.0 val | AP: 39.9 ± 0.6 -> 41.8 ± 0.7 | Helped |
| use MPTM (60%) rather than ToMe (50%) | CoDi · AudioCaps val | FIDt+a: 16.92±0.52 -> 16.79±0.47 | No clear effect |
| use MPTM (60%) rather than ToMe (50%) | CoDi · AudioCaps val (Text + Image → Video + Audio) | SIM-VA: 0.194±0.035 -> 0.241±0.022 | Helped |
Surprising findings
- At 60% merge ratio, MPTM's FID and CLIP-R on ImageNet-1k are comparable to or better than the unmerged SD v1.5 baseline.
- Plain ToMe degrades DiffusionDet AP (29.4 to 25.1 with ResNet-50), which the authors attribute to the lack of semantic consideration in merging.
Limitations
- Object detection variant requires retraining, and the training-free claim applies mainly to pretrained U-Net diffusion models.
- Results on multimodal generation are reported only on AudioCaps and limited to small-scale comparisons with single-run standard deviations over three runs.
Experimental setup
Datasets: ImageNet-1k LVIS V1.0 COCO AudioCaps
Benchmarks: ImageNet-1k LVIS V1.0 COCO AudioCaps
Metrics: FID CLIP-R AP AP50 AP75 FPS s/image GB/image FIDt+a SIM-VA
Baselines: ToMe ToMeSD DiffusionDet CoDi
Builds on: ToMe ToMeSD DiffusionDet CoDi Latent diffusion models
Closest papers by meaning
- Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation (NeurIPS 2026 · similarity 0.90)
- ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.90)
- Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration (CVPR 2025 · similarity 0.88)
- TokenCompose: Text-to-Image Diffusion with Token-level Supervision (CVPR 2024 · similarity 0.88)
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.86)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.