Research map / Architectures and efficiency / Attention and transformers
On Inductive Biases That Enable Generalization in Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pixel-space DiTs lack geometry-adaptive harmonic bases unlike UNets, identifies attention-map locality as the inductive bias linked to DiT generalization, and injects local attention windows to control generalization and FID under limited data.
Problem
Prior analyses of diffusion generalization via harmonic bases cover only UNets; it is unknown what inductive bias drives generalization in transformer-based diffusion models (DiTs).
Approach
Attention-map locality is the inductive bias linked to DiT generalization; restricting self-attention to local windows in early layers improves the PSNR gap and FID when training data is limited, with placement and window size giving fine-grained control.
Key results
- Local attention in the first six layers of DiT-XS/1 reduces PSNR gap on CelebA at N=10^4 by 12.74% and FID from 9.69 to 8.46.
- On ImageNet with N=10^4, DiT-XS/1 FID drops from 52.57 to 43.87 with local attention.
- At N=10^5, local attention yields comparable or slightly worse FID (e.g., ImageNet DiT-XS/1 17.31 to 18.07), so gains are concentrated in the low-data regime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add local attention to the first six layers | DiT-XS/1 and DiT-S/1 · CelebA, ImageNet, LSUN Church, LSUN Bedroom, LSUN Bridge, and LSUN Tower; N=10^4 | FID: 9.6932 -> 8.4580; 52.5650 -> 43.8687; 12.8842 -> 10.4794; 14.8354 -> 11.9566; 23.1771 -> 18.1470; 12.5532 -> 10.5644 | Helped |
| Add local attention to the first six layers | DiT-XS/1 and DiT-S/1 · CelebA, ImageNet, LSUN Church, LSUN Bedroom, LSUN Bridge, and LSUN Tower; N=10^5 | FID: 2.6303 -> 2.5469; 17.3114 -> 18.0671; 5.2927 -> 5.2672; 5.4066 -> 5.3542; 8.0791 -> 8.3546; 4.6619 -> 4.8041 | Mixed |
| Add local attention to the first six layers | DiT-XS/1 and DiT-S/1 · Six datasets; N=10^3, 10^4, and 10^5 | PSNR gap | Helped |
| Place local attention in early rather than interleaved or tail layers (ablation) | DiT · CelebA and ImageNet | PSNR gap | Helped |
| Interleave local attention with global attention (ablation) | DiT · CelebA and ImageNet | PSNR gap | Hurt |
| Place local attention in tail layers (ablation) | DiT · CelebA and ImageNet | PSNR gap | Hurt |
| Keep total window size fixed but redistribute sizes across six local attention layers: (5,5,5,5,5,5) versus (3,3,5,5,7,7) (ablation) | DiT · CelebA and ImageNet | PSNR gap | No clear effect |
| Reduce the attention window size (ablation) | DiT-XS/1 · CelebA and ImageNet | PSNR gap | Helped |
| Increase the attention window size (ablation) | DiT-XS/1 · CelebA and ImageNet | PSNR gap | Hurt |
| Share attention-block parameters between blocks 1 and 2, and blocks 3 and 4 (ablation) | DiT-XS/1 · CelebA | FID: 9.6932 -> 8.7819 | Helped |
| Replace the first three attention maps with PCA-composed maps (ablation) | DiT-XS/1 · CelebA, N=10^4 | FID: 9.6932 -> 11.3872 | Hurt |
Surprising findings
- Pixel-space DiTs do not show geometry-adaptive harmonic bases in Jacobian eigenvectors, unlike UNets, despite generalizing better than UNets at equal FLOPs.
- Placing local attention in tail layers or interleaving with global attention tends to hurt generalization, the opposite of early-layer placement.
Limitations
- Experiments are limited to small pixel-space DiTs (XS/1, S/1) at 32x32 resolution.
- Local attention slightly compromises FID with abundant data (N=10^5).
Experimental setup
Datasets: CelebA ImageNet LSUN Church LSUN Bedroom LSUN Bridge LSUN Tower
Benchmarks: CelebA ImageNet LSUN Church LSUN Bedroom LSUN Bridge LSUN Tower
Metrics: PSNR gap FID PSNR
Baselines: DiT-XS/1 DiT-S/1 UNet (improved diffusion) weight sharing PCA attention reduction
Builds on: geometry-adaptive harmonic bases DiT improved diffusion PSNR gap
Closest papers by meaning
- Generalization in diffusion models arises from geometry-adaptive harmonic representations (ICLR 2024 · similarity 0.92)
- Towards a Mechanistic Explanation of Diffusion Model Generalization (ICML 2025 · similarity 0.91)
- Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure (NeurIPS 2024 · similarity 0.91)
- Generalization through variance: how noise shapes inductive biases in diffusion models (ICLR 2025 · similarity 0.90)
- An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models (NeurIPS 2025 · similarity 0.89)
- DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised $h$-transform (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.