Research map / Architectures and efficiency / Attention and transformers
Can We Achieve Efficient Diffusion Without Self-Attention? Distilling Self-Attention into Convolutions
Ziyi Dong, Chengxing Zhou, Weijian Deng, Pengxu Wei, Xiangyang Ji, Liang Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace self-attention with ΔConvBlock | SD1.5 · LAION 10k 512x512 | DINOv2 Score: 42.74 -> 44.72 | Improved |
| replace self-attention with ΔConvBlock | SDXL · LAION 10k 1024x1024 | DINOv2 Score: 42.65 -> 45.14 | Improved |
| replace self-attention with ΔConvBlock | PixArt 512x512 · LAION 10k | DINOv2 Score: 41.31 -> 42.56 | Improved |
| replace self-attention with ΔConvBlock | SD1.5 · LAION 10k 512x512 | FDD: 210.86 -> 200.15 | Hurt |
| replace self-attention with ΔConvBlock | PixArt 512x512 · LAION 10k | FDD: 173.88 -> 181.06 | Hurt |
| replace self-attention with ΔConvBlock | PixArt 1024x1024 · LAION 10k | CLIP Score: 30.60 -> 30.55 | No clear effect |
| replace self-attention with ΔConvBlock | SDXL · LAION 10k 1024x1024 | CLIP Score: 30.92 -> 30.87 | No clear effect |
| replace self-attention with ΔConvBlock (K=13) | PixArt · 1024x1024 latency | latency (ms): 5.91 -> 3.53 | Improved |
| use ΔConvBlock instead of LinFusion | PixArt · 1024x1024 latency | latency (ms): 5.82 -> 3.53 | Improved |
| increase kernel size K from 13 to 25 (ablation) | ΔConvFusion · 4K latency | latency (ms): 54.84 -> 131.71 | Hurt |
| replace self-attention with ΔConvBlock | SD1.5 · 16K FLOPs | Mean FLOPs (G): 11,010,934.11 -> 1,589.10 | Improved |
| replace self-attention with ΔConvBlock | PixArt · 512x512 FLOPs | Mean FLOPs (G): 31.42 -> 8.43 | Improved |
Closest papers by meaning
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.91)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.89)
- DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · similarity 0.89)
- Attention Distillation: A Unified Approach to Visual Characteristics Transfer (CVPR 2025 · similarity 0.88)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.88)
- DiC: Rethinking Conv3x3 Designs in Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
