Research map / Architectures and efficiency / Attention and transformers
DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiG, a diffusion backbone built on gated linear attention (GLA) with a spatial reorient & enhancement module for block-wise scan directions and local depth-wise conv awareness, in plain and U-shape variants.
Problem
Diffusion transformers (DiT) have quadratic attention cost, which becomes prohibitive for high-resolution and long-sequence visual generation; existing linear/SSM alternatives suffer from unidirectional scanning and lack of local awareness.
Approach
Adapts gated linear attention to 2D diffusion by alternating transpose and flip operations block-by-block (SREM) to get four-directional global context at minimal cost, plus a 3x3 depth-wise conv with identity initialization for local detail; a U-shape variant adds hierarchy and shortcuts.
Key results
- DiG-XL/2 with classifier-free guidance reaches FID 2.07 on ImageNet 256x256 vs DiT-XL/2-G 2.27 (7M iterations for DiT).
- DiG-S/2 is 2.5x faster and uses 75.7% less GPU memory than DiT-S/2 at 1792x1792 resolution.
- DiG-XL/2 is 4.2x faster than the Mamba-based DiS at 1024 resolution and 1.8x faster than Flash-DiT at 2048 resolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DiT with DiG at the same model scale | ImageNet 256×256 generation · ImageNet 256×256, 400K | FID-50K: 68.40 -> 62.06; 43.47 -> 39.50; 23.33 -> 22.90; 19.47 -> 18.53 | Helped |
| use DiG-XL/2 instead of DiT-XL/2, with fewer training iterations | ImageNet 256×256 generation · ImageNet 256×256 | FID-50K: 9.62 -> 8.60 | Helped |
| use DiG-XL/2-G instead of DiT-XL/2-G, with fewer training iterations | ImageNet 256×256 generation with classifier-free guidance · ImageNet 256×256 | FID-50K: 2.27 -> 2.07 | Helped |
| add global bidirectional scanning to causal DiG (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 175.84 -> 69.28 | Helped |
| add identity-initialized DWConv2d for local awareness (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 69.28 -> 63.84 | Helped |
| use DWConv2d without identity initialization (ablation) | DiG-S/2 with bidirectional scanning on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 69.28 -> 96.83 | Hurt |
| add four-directional scanning to the identity-initialized DWConv2d setup (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 63.84 -> 62.06 | Helped |
| move SREM before attention instead of after FFN (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 62.06 -> 62.34 | Hurt |
| move SREM between attention and FFN instead of after FFN (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 62.06 -> 62.93 | Hurt |
| add a shortcut (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 62.06 -> 57.01 | Helped |
| add a shortcut (ablation) | DiG-S/2 on ImageNet 512×512 · ImageNet 512×512 | FID-50K: 99.04 -> 91.42 | Helped |
| add the hierarchical U-shape structure and shortcut (ablation) | DiG-S/2 on ImageNet 256×256 · ImageNet 256×256 | FID-50K: 62.06 -> 52.20 | Helped |
Surprising findings
- Multi-path (bidirectional or four-directional) scanning leads to numerical NaN during training, while block-by-block scanning is stable.
- Plain DiG without bidirectional scanning reaches FID 175.84 at DiT-S/2 scale, far worse than DiT-S/2's 68.40.
Limitations
- Building a large foundation model like Sora upon DiG is left unexplored.
- Main evaluation limited to class-conditional ImageNet; no text-to-image or video experiments.
- Plain DiG with causal scanning alone performs poorly without the SREM module.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID-50K sFID Inception Score Precision Recall FPS GPU memory
Baselines: DiT DiS Flash-DiT DiM DiMSUM ADM LDM
Builds on: Gated Linear Attention Transformer DiT DDPM Mamba-based DiS
Closest papers by meaning
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.90)
- Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · similarity 0.90)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.90)
- ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
- Diffusion Models Without Attention (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.