Research map / Image generation / Autoregressive and latent generation
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REPA, a regularization that aligns noisy-input diffusion transformer hidden states with clean DINOv2 features, speeding up SiT/DiT training by over 17.5x and reaching FID 1.42 on ImageNet 256x256 with guidance interval.
Problem
Diffusion transformers learn poor internal representations during generative training, making large-scale training slow; their representations lag far behind self-supervised encoders.
Approach
Add a regularization loss that maximizes patch-wise similarity between a projected intermediate diffusion transformer hidden state (from noisy latent input, early layers) and clean-image features from a frozen pretrained self-supervised encoder such as DINOv2, so the generative model learns semantic representations more easily.
Key results
- SiT-XL/2 + REPA reaches FID 7.9 at 400K iterations without CFG, exceeding vanilla SiT-XL at 7M iterations (FID 8.3), a >17.5x speedup.
- With classifier-free guidance, SiT-XL/2 + REPA improves FID from 2.06 to 1.80 at 4M iterations; with guidance interval it reaches FID 1.42 (state of the art reported).
- Aligning only the first 8 layers works best; cosine similarity matches or beats NT-Xent at convergence, and the method works across DINOv2, MoCov3, MAE, CLIP and SigLIP targets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add REPA alignment to the SiT-L/2 model | SiT-L/2 · ImageNet 256×256, 400K iterations | FID: 18.8 -> 10.0 | Helped |
| Align to higher-quality pretrained visual representations (ablation) | SiT-L/2 + REPA · ImageNet 256×256, 400K iterations | FID: 12.5 -> 10.0 | Helped |
| Replace DINOv2-B with DINOv2-L or DINOv2-g as the target encoder (ablation) | SiT-L/2 + REPA · ImageNet 256×256, 400K iterations | FID: 9.7, 10.0, 9.8 | No clear effect |
| Set alignment depth to 6 instead of 8 (ablation) | SiT-L/2 + REPA · ImageNet 256×256, 400K iterations | FID: 10.0 -> 10.3 | No clear effect |
| Extend alignment beyond depth 8 (ablation) | SiT-L/2 + REPA · ImageNet 256×256, 400K iterations | FID: 10.0 -> 10.5; 11.2; 11.6; 12.1 | Hurt |
| Use cosine similarity instead of NT-Xent for alignment (ablation) | SiT-L/2 + REPA · ImageNet 256×256, 400K iterations | FID: 10.0 -> 9.9 | No clear effect |
| Increase the REPA coefficient beyond λ=0.5 (ablation) | SiT-XL/2 + REPA · ImageNet 256×256, 400K iterations | FID | No clear effect |
| Add REPA alignment to DINOv2-B | SiT-XL/2 · ImageNet 256×256, no guidance | FID: 8.3 -> 7.9 | Helped |
| Add REPA alignment to DINOv2-B | SiT-XL/2 · ImageNet 256×256, classifier-free guidance | FID: 2.06 -> 1.80 | Helped |
Surprising findings
- Aligning only the first few layers (e.g., layer 6 or 8) gives better generation than aligning deeper layers, with later layers free to model high-frequency details.
- Vanilla diffusion transformer representations are only weakly aligned with DINOv2 even after 7M iterations, and linear probing peaks mid-network then declines.
Limitations
- Requires a pretrained external visual encoder (e.g., DINOv2) and its features, adding dependence on that encoder.
- Evaluation focuses on ImageNet class-conditional generation at 256x256; text-to-image and 512x512 results are only in the appendix.
- Target encoder size differences (DINOv2-B/L/g) are marginal, and the analysis is limited to SiT/DiT architectures.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256
Metrics: FID sFID IS precision recall linear probing accuracy CKNNA
Baselines: ADM LDM U-ViT-H/2 MDTv2-XL/2 DiT-XL/2 SiT-XL/2 MaskDiT SD-DiT
Builds on: SiT DiT DINOv2 stochastic interpolants classifier-free guidance
Closest papers by meaning
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.94)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.93)
- REPA Works Until It Doesn’t: Early-Stopped, Holistic Alignment Supercharges Diffusion Training (NeurIPS 2025 · similarity 0.92)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.92)
- AHPA: Adaptive Hierarchical Prior Alignment for Diffusion Transformers (NeurIPS 2026 · similarity 0.92)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.