Research map / Image generation / Fast diffusion sampling
Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REG, which entangles a single noised DINOv2 class token with SiT image latents during training (plus alignment and velocity losses), accelerating diffusion transformer training on ImageNet 256 and 512 with negligible overhead.
Problem
Diffusion transformers converge slowly; existing external representation alignment (REPA) is absent at inference and does not let the model generate discriminative semantics itself during denoising.
Approach
Concatenate one noised class token from a pretrained vision foundation model (DINOv2) with the noised image latent tokens and jointly denoise both with velocity prediction plus REPA-style alignment, so the model generates image-class semantics from pure noise at inference and uses them to guide generation.
Key results
- SiT-XL/2 + REG reaches FID 1.8 at 4M iterations without CFG and matches SiT-XL/2 (7M steps) FID 8.2 at 110K steps, reported as 63x faster convergence.
- SiT-L/2 + REG at 400K iterations achieves FID 4.6, beating SiT-XL/2 + REPA at 4M iterations (FID 5.9) per the paper's comparison.
- REG adds +0.30% parameters, +0.38% FLOPs and +0.49% latency over SiT-XL/2 + REPA while reducing FID from 7.90 to 3.44 in the cost table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add REG class-token entanglement and representation alignment | SiT-B/2 · ImageNet 256×256, 400K iterations, no CFG | FID: 24.4 -> 15.2 | Helped |
| Add REG class-token entanglement and representation alignment | SiT-L/2 · ImageNet 256×256, 400K iterations, no CFG | FID: 10.0 -> 4.6 | Helped |
| Add REG class-token entanglement and representation alignment | SiT-XL/2 · ImageNet 256×256, 4M iterations, no CFG | FID: 5.9 -> 1.8 | Helped |
| Add REG | SiT-XL/2 · ImageNet 512×512, CFG | FID: 2.08 -> 1.68 | Helped |
| Use CLIP-L instead of DINOv2-B as the target representation (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 21.30 | Hurt |
| Use DINOv2-L instead of DINOv2-B as the target representation (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 17.36 | Hurt |
| Set alignment depth to 2 instead of 4 (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 18.19 | Hurt |
| Set alignment depth to 6 or 8 instead of 4 (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 16.31; 15.22 -> 17.31 | Hurt |
| Increase β from 0.03 to 0.05 or 0.10 (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 16.28; 15.22 -> 18.41 | Hurt |
| Reduce β from 0.03 to 0.01 or 0.02 (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 15.76; 15.22 -> 15.64 | Hurt |
| Replace DINOv2 class token with one learnable token (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 23.31 | Hurt |
| Replace DINOv2 class token with average latent features (ablation) | SiT-B/2 + REG · ImageNet 256×256, 400K iterations | FID: 15.22 -> 24.12 | Hurt |
Surprising findings
- Entangling the DINOv2 class token without any representation alignment already lowers FID from 33.0 to 26.67 on SiT-B/2.
Limitations
- Convergence speedups are measured against the reported baselines and on ImageNet only
- Gains from CFG-based results depend on guidance interval tuning
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID sFID IS Precision Recall CKNNA GPU hours FLOPs latency
Baselines: SiT-XL/2 REPA REPA-E DiT-XL/2 ADM-U LDM-4 MaskDiT MDTv2 U-ViT RCG
Builds on: SiT REPA DiT DINOv2 classifier-free guidance
Closest papers by meaning
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.94)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.90)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.90)
- End-to-End Training for Unified Tokenization and Latent Denoising (NeurIPS 2026 · similarity 0.90)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.