Research map / Image generation / Autoregressive and latent generation
Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Re-ttention, a training-free sparse attention method for DiT visual generation that reshapes sparse softmax scores using cached denominator ratios from prior denoising steps plus residual caching, reaching ~96.9% sparsity.
Problem
Training-free sparse attention for diffusion transformers typically degrades visual quality beyond ~70% sparsity because omitting tokens shrinks the softmax denominator, distorting attention statistics.
Approach
Cache the ratio between sparse and full softmax denominators from a full-attention step (which is stable across neighboring denoising steps), rescale sparse attention probabilities by this ratio (with a ramp-up), and add a cached residual equal to the masked tokens' attention output to restore normalization.
Key results
- On CogVideoX-2B at 96.9% sparsity, Re-ttention achieves PSNR 27.96, SSIM 0.894, LPIPS 0.059, outperforming DiTFastAttn (27.93, 0.865, 0.098) and SVG/MInference.
- On PixArt-Σ at 96.9% sparsity, Re-ttention reaches GenEval 0.512 vs. full attention 0.544 and DiTFastAttn 0.233.
- On Hunyuan-DiT at 96.9% sparsity, Re-ttention keeps GenEval 0.590 (full 0.610) while DiTFastAttn drops to 0.002.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Re-ttention instead of DiTFastAttn at 96.9% sparsity | CogVideoX-2B · T2V | SSIM: 0.865 -> 0.894 | Helped |
| use Re-ttention instead of DiTFastAttn at 96.9% sparsity | CogVideoX-2B · T2V | PSNR: 27.93 -> 27.96 | Helped |
| use Re-ttention instead of SVG at 96.9% sparsity | CogVideoX-2B · T2V | SubConsist: 96.73% -> 94.80% | Hurt |
| use Re-ttention instead of DiTFastAttn at 93.8% sparsity | PixArt-α · T2I | GenEval: 0.431 -> 0.456 | Helped |
| use Re-ttention instead of DiTFastAttn at 96.9% sparsity | PixArt-Σ · T2I | GenEval: 0.233 -> 0.512 | Helped |
| use Re-ttention instead of DiTFastAttn at 96.9% sparsity | Hunyuan-DiT · T2I | HPSv2: 12.28 -> 28.89 | Helped |
| set ramp-up hyperparameter λ to 0.04 rather than 0 (ablation) | Re-ttention on PixArt-Σ · HPSv2 | HPSv2: 27.46 -> 27.72 | Helped |
| set ramp-up hyperparameter λ to 0.04 rather than 0.01 (ablation) | Re-ttention on PixArt-Σ · HPSv2 | HPSv2: 26.83 -> 27.72 | Helped |
| set ramp-up hyperparameter λ to 0.04 rather than 0.02 (ablation) | Re-ttention on PixArt-Σ · HPSv2 | HPSv2: 26.82 -> 27.72 | Helped |
Surprising findings
- Post-softmax masking that preserves the full softmax denominator yields visibly better videos than pre-softmax sparse attention at the same sparsity, supporting the denominator-shift hypothesis.
- SVG at 96.9% sparsity attains the highest SubConsist score, exceeding full attention, though it underperforms on all other metrics.
Limitations
- Designed for static masks; dynamic mask extension is future work.
- No custom GPU kernel implemented; latency comparison only on typical GPUs.
- Not directly applicable to autoregressive or causally masked attention where sequential cross-step caching is unavailable.
Experimental setup
Datasets: VBench GenEval HPSv2 MS-COCO 2014
Benchmarks: VBench GenEval HPSv2 MS-COCO 2014
Metrics: VBench Image Quality VBench Subject Consistency PSNR SSIM LPIPS GenEval HPSv2 ImageReward CLIP score
Baselines: Sparse VideoGen MInference DiTFastAttn
Builds on: DiTFastAttn Sparse VideoGen MInference LongFormer
Closest papers by meaning
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.90)
- PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models (NeurIPS 2025 · similarity 0.90)
- Bidirectional Sparse Attention for Faster Video Diffusion Training (NeurIPS 2026 · similarity 0.89)
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity (ICML 2025 · similarity 0.89)
- PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers (NeurIPS 2026 · similarity 0.89)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.