Research map / Image generation / Autoregressive and latent generation
Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDTM, a training-free, post-training token merging method for diffusion transformers that merges similar low-structure tokens early and inattentive low-detail tokens late, achieving about 1.55x speedup on SD3 Medium with negligible FID change.
Problem
Diffusion transformers are computationally expensive at inference; existing token reduction methods ignore the denoising priors (structure-then-detail) and either need fine-tuning or give suboptimal acceleration and quality.
Approach
Targets token redundancy in regions not attended by the denoising process: similarity-prioritized structure merging early and attention-based inattentive-prioritized detail merging late, with timestep-varying compression ratios and ChatGPT-guided prompt token reweighting, all without fine-tuning.
Key results
- SDTM reaches 1.55x speedup on SD3 Medium (config b) with FID 28.97 for SDTM* vs 29.02 baseline on COCO2017.
- At ~1.3x speed, SDTM* achieves FID 28.57 vs ToMeSD 45.28 and AT-EDM 34.72.
- Against fine-tuned TokenCache and DyDiT at ~1.5x, SDTM (FID 29.60) is comparable without fine-tuning, while achieving the highest CLIP (0.3224-0.3237).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SDTM* token merging to SD3 Medium | SD3 Medium · COCO2017 validation | Latency: 10.67 -> 6.90 | Helped |
| Add SDTM* token merging to SD3 Medium | SD3 Medium · COCO2017 validation | FID: 29.02 -> 28.97 | No clear effect |
| Use SDTM* with 20-DPM sampling | SD3 Medium · COCO2017 validation, 20-DPM | FID: 29.04 -> 29.08 | Hurt |
| Use SDTM* with 15-RF sampling | SD3 Medium · COCO2017 validation, 15-RF | FID: 29.36 -> 29.51 | Hurt |
| Use SDTM* with 28-RF sampling | SD3.5 Large · COCO2017 validation, 28-RF | FID: 25.91 -> 25.72 | Helped |
| Use SSM early and IDM later rather than the reverse (ablation) | SDTM · COCO2017 validation | FID: 34.52 -> 29.60 | Helped |
| Use IDM early and SSM later rather than another merging order (ablation) | SDTM · COCO2017 validation | FID: 29.60 -> 34.52 | Hurt |
| Set ratio deviation to 0.2 with cosine decline rather than linear decline (ablation) | SDTM · COCO2017 validation | FID: 31.64 -> 29.60 | Helped |
| Increase ratio deviation to 0.3 with cosine decline (ablation) | SDTM · COCO2017 validation | FID: 29.60 -> 31.81 | Hurt |
| Increase ratio deviation to 0.4 with cosine decline (ablation) | SDTM · COCO2017 validation | FID: 29.60 -> 36.03 | Hurt |
| Add prompt token reweighting (ablation) | SDTM · COCO2017 validation | CLIP: 0.3186 -> 0.3224 | Helped |
| Add prompt token reweighting (ablation) | SDTM · COCO2017 validation | FID: 29.94 -> 29.60 | Helped |
Surprising findings
- Token merging with SDTM at ~1.3x slightly improves FID over the unaccelerated SD3 Medium baseline.
Limitations
- Larger compression demands cause merging errors and quality degradation, requiring other acceleration methods such as distillation.
- Prompt token categorization relies on ChatGPT queries.
Experimental setup
Datasets: MS-COCO2017 validation PartiPrompts
Benchmarks: MS-COCO2017 PartiPrompts
Metrics: FID CLIP score MACs latency speed
Baselines: ToMeSD AT-EDM TokenCache DyDiT
Builds on: ToMeSD AT-EDM ALGM Rectified Flow
Closest papers by meaning
- Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation (NeurIPS 2026 · similarity 0.90)
- ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.90)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.90)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
- Multimodal Promptable Token Merging for Diffusion Models (AAAI 2025 · similarity 0.88)
- AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.