Research map / Image generation / Autoregressive and latent generation

Which Tokens to Merge? Diffusion Dynamics for Efficient Image Generation

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DaTMe, a training-free token merging method for diffusion transformers that chooses which tokens to merge using a Tweedie-estimate-based temporal and spatial importance map, evaluated on PixArt-α and FLUX.

Problem

Training-free token merging for accelerating diffusion models often merges structurally important or prompt-unmentioned tokens (e.g., background details) because existing importance signals such as CFG discrepancy are biased toward prompt-aligned foreground, degrading image structure.

Approach

Defines token importance from diffusion dynamics: a temporal signal (inter-step change of the Tweedie clean-image estimate) combined with a spatial signal (high-pass residual of the Tweedie estimate), used to assign anchor (constraint) tokens and merge-source tokens with grid-based source completion and locality-constrained matching.

Key results

• On PixArt-α at compression ratio 0.5, DaTMe reaches CLIP-T 31.01 and FID 43.42 versus CFG-based merging 30.46 / 52.29 and ToMeSD 29.52 / 94.89.
• On PixArt-α at compression ratio 0.3, DaTMe improves SSIM to 0.8112 and LPIPS to 0.1501 versus CFG-based 0.7137 and 0.2369.
• On FLUX.1-dev at compression ratio 0.5, DaTMe reaches SSIM 0.7087 and PSNR 20.20 versus ToMA 0.7009 and 20.04, with higher HPSv2 (28.93 vs 27.76).

Surprising findings

• Replacing the raw score slope with the Tweedie difference as the temporal signal avoids late-step scale-term domination that the authors show makes the score slope unreliable.

Limitations

• Quality degrades under aggressive compression in highly complex scenes with many objects.
• Layer-wise roles of DiT diffusion layers remain underexplored.
• DaTMe is slightly slower in latency than prior methods such as ToMeSD and CFG-based merging on PixArt-α.

Experimental setup

Datasets: COCO 2014 validation set

Benchmarks: COCO 2014 validation (10k prompts)

Metrics: CLIP-T FID HPSv2 SSIM PSNR LPIPS Aesthetic score latency TFLOPs

Baselines: ToMeSD CFG-based merging ToMA PixArt-α (unmerged) FLUX.1-dev (unmerged)

Builds on: ToMeSD CFG-based token merging Token Merging (ToMe) Tweedie's formula

Closest papers by meaning

• Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration (CVPR 2025 · similarity 0.90)
• ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.90)
• Multimodal Promptable Token Merging for Diffusion Models (AAAI 2025 · similarity 0.90)
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (NeurIPS 2024 · similarity 0.87)
• Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution (NeurIPS 2025 · similarity 0.87)
• DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking (NeurIPS 2026 · similarity 0.87)