Research map / Image generation / Fast diffusion sampling
Diffusion Models Without Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiffuSSM, an attention-free latent diffusion architecture built on gated bidirectional state space models (S4D) with hourglass MLPs, avoiding patchification and matching or beating DiT on ImageNet with fewer Gflops.
Problem
Scaling diffusion models to high resolution is computationally costly because self-attention is quadratic in sequence length, and existing fixes (patchification, multi-scale U-Nets) lose spatial detail.
Approach
Replace attention in diffusion transformers with a gated bidirectional SSM (S4D) backbone operating on the full-length flattened latent sequence without patchification, and use hourglass feed-forward layers that shorten the sequence in MLPs to reduce FLOPs.
Key results
- Without classifier-free guidance, DiffuSSM-XL reaches FID 9.07 and sFID 5.52 on ImageNet 256x256, beating DiT-XL/2 (9.62 FID) with about 20% fewer total Gflops.
- With classifier-free guidance, DiffuSSM-XL-G achieves the best sFID among DDPM-based models at 256x256 (4.49) and FID 2.28 vs DiT-XL/2-G 2.27.
- On ImageNet 512x512 with guidance, DiffuSSM-XL-G gets FID 3.41 vs DiT-XL/2-G 3.04 while using about 25% fewer Gflops and 40% fewer training images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace attention with DIFFUSSM | ImageNet 256×256 class-conditional generation · ImageNet 256×256 | FID: 9.62 -> 9.07 | Helped |
| replace attention with DIFFUSSM | ImageNet 256×256 class-conditional generation · ImageNet 256×256 | sFID: 6.85 -> 5.52 | Helped |
| replace attention with DIFFUSSM | ImageNet 256×256 class-conditional generation · ImageNet 256×256 | Total Gflops: 2.13 × 1011 -> 1.85 × 1011 | Helped |
| use classifier-free guidance with DIFFUSSM | ImageNet 256×256 class-conditional generation · ImageNet 256×256 | FID: 2.27 -> 2.28 | Mixed |
| use classifier-free guidance with DIFFUSSM | ImageNet 256×256 class-conditional generation · ImageNet 256×256 | sFID: 4.60 -> 4.49 | Helped |
| replace attention with DIFFUSSM | ImageNet 512×512 class-conditional generation · ImageNet 512×512 | FID: 3.04 -> 3.41 | Hurt |
| replace attention with DIFFUSSM | ImageNet 512×512 class-conditional generation · ImageNet 512×512 | sFID: 5.02 -> 5.84 | Hurt |
| replace attention with DIFFUSSM | LSUN-Church unconditional generation · LSUN-Church 256×256 | FID: 4.02 -> 3.94 | Helped |
| replace attention with DIFFUSSM | LSUN-Bedroom unconditional generation · LSUN-Bedroom 256×256 | FID: 2.95 -> 3.02 | Hurt |
| use bidirectional SSMs instead of unidirectional SSMs (ablation) | DIFFUSSM · diffusion training loss | loss | Helped |
| use the DIFFUSSM architecture instead of the UViSSM structure (ablation) | DIFFUSSM · diffusion training loss | loss | Helped |
| remove patchification (ablation) | DIFFUSSM · ImageNet qualitative image generation | visual quality | Helped |
Surprising findings
- Removing patchification in the attention-free model keeps quality comparable to uncompressed models at lower compute, while the non-hourglass unidirectional SSM variant performs notably worse than bidirectional.
Limitations
- Only evaluated on (un)conditional image generation, not text-to-image.
- Does not incorporate recent improvements such as masked image training.
- Does not adopt U-ViT long skip connections, leaving possible gains unexplored.
Experimental setup
Datasets: ImageNet-1k LSUN Church LSUN Bedroom
Benchmarks: ImageNet 256x256 ImageNet 512x512 LSUN Church 256x256 LSUN Bedroom 256x256
Metrics: FID sFID Inception Score Precision Recall Gflops
Baselines: DiT-XL/2 U-ViT-H/2 ADM ADM-U LDM-4 LDM-8 BigGAN-deep StyleGAN-XL MaskGIT CDM
Builds on: DiT S4D Latent Diffusion Models ADM classifier-free guidance Gated SSM (Mega-style gating)
Closest papers by meaning
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.91)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.91)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- PFDiff: Training-Free Acceleration of Diffusion Models Combining Past and Future Scores (ICLR 2025 · similarity 0.90)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.90)
- Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.