Research map / Image generation / Autoregressive and latent generation
LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEDiT, a diffusion transformer without explicit positional encodings that uses causal attention plus a convolutional locality module to extrapolate to up to 4x higher resolutions.
Problem
Diffusion transformers degrade when generating images at resolutions beyond training, largely because explicit positional encodings such as RoPE must extrapolate to unseen positions.
Approach
Remove explicit PEs and alternate self-attention with causal attention, which implicitly encodes global position via token-variance decay (Var ~ 1/(i+1)); add a convolution after patchification (with multi-dilation training) to restore local fine-grained detail.
Key results
- On ImageNet 256 trained model at 384x384, LEDiT reaches FID 9.34 vs. 13.51 for the best prior method (DiT-VisionYaRN).
- At 512x512 (4x pixels), LEDiT achieves FID 33.25 vs. 49.86 for the best prior baseline.
- Trained on ImageNet 256 with 4x extrapolation, LEMMDiT on COCO reports FID 29.89 and sFID 39.98, improving over RoPE-based variants; in-distribution 256x256 FID is slightly worse than DiT (2.38 vs 2.27).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace explicit positional encoding with LEDiT's causal attention and convolution | DiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 49.86 -> 33.25 | Helped |
| add causal attention to a NoPE model (ablation) | DiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 378.95 -> 286.01 | Helped |
| add convolution to a NoPE model (ablation) | DiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 378.95 -> 130.91 | Helped |
| combine causal attention and convolution (ablation) | DiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 378.95 -> 35.86 | Helped |
| use causal scan variant (d) rather than variant (a) (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 62.49 -> 35.86 | Helped |
| use multi-dilation training (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 39.20 -> 35.86 | Helped |
| alternate self-attention then causal-attention blocks instead of sequential self-then-causal blocks (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 48.65 -> 35.86 | Helped |
| increase multi-dilation probability from 0.1 to 0.2 (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 35.86 -> 37.99 | Hurt |
| increase multi-dilation probability from 0.1 to 0.5 (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 35.86 -> 37.56 | Hurt |
| use dilation rates (2,3) rather than 2 (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID: 35.86 -> 37.24 | Hurt |
| fine-tune LEDiT for additional steps past 50K (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | FID | Hurt |
| apply attention scaling (ablation) | LEDiT-XL/2 · ImageNet 256×256 → 512×512 | image quality | Helped |
Surprising findings
- Pure NoPE DiT generates noise-like images at higher resolution (FID 378.95), and causal attention alone or convolution alone is insufficient; only their combination works well.
- Switching from causal to self-attention in the late denoising stage (T<100) has minor effect on extrapolation, suggesting positional information is acquired early.
Limitations
- Only trained on ImageNet and COCO; not tested on large-scale datasets such as LAION-5B.
- Slight in-distribution performance gap relative to DiT, attributed to causal attention limiting long-range dependencies.
- Higher-resolution training has not been explored.
Experimental setup
Datasets: ImageNet COCO
Benchmarks: ImageNet 256x256 (extrapolated to 384, 448, 512) ImageNet 512x512 (extrapolated to 768, 896, 1024) COCO (512x512 extrapolation)
Metrics: FID sFID IS Precision Recall
Baselines: DiT-Sin/Cos PE DiT-RoPE DiT-VisionNTK DiT-VisionYaRN DiT-RIFLEx FiTv2-VisionNTK FiTv2-VisionYaRN MMDiT-RoPE
Builds on: DiT FiT RIFLEx NoPE REPA
Closest papers by meaning
- SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers (NeurIPS 2026 · similarity 0.89)
- DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · similarity 0.88)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.87)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.87)
- Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers (ICML 2024 · similarity 0.86)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.