Research map / Image generation / Fast diffusion sampling

Why DiT Models Underperform as Representation Learners without Long Skip Connections

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via from-scratch DiT training that long skip connections improve diffusion feature quality for discriminative tasks, and proposes a "diffusion noise bypassing" hypothesis explaining the effect.

Problem

DiT diffusion backbones yield weaker discriminative features than U-Net backbones despite stronger generation, and the cause is only partly understood.

Approach

Long skip connections let diffusion noise bypass the target-reconstruction section of the backbone, so the middle-layer features are less polluted by noise; this links LSCs to the diffusion objective rather than only to high-frequency shortcutting.

Key results

• From-scratch DiT-XL/2 with LSCs reaches FID 14.39 vs 20.87 without, and SPair (a) PCK 41.10 vs 34.68.
• Even with FID aligned (w/ LSC 100k, FID 21.67), LSC features still beat the non-LSC model on SPair (a) (38.85 vs 34.68) and Horse-21 (35.24 vs 30.35).
• Injecting high-frequency latent plus noise into skip connections causes representation collapse (NaN) or failed convergence, supporting the noise-carrying role of LSCs.

Surprising findings

• Fine-tuning the whole DiT on SPair (c) can narrow or reverse the gap between non-LSC and LSC models.
• Adding LSCs to a pre-trained PixArt-Sigma via zero-init fine-tuning only slightly improves feature quality and does not change the quality-layer curve.

Limitations

• Experiments are on conventional isotropic DiT rather than MMDiT architectures.
• Effectively augmenting pre-trained DiTs with LSCs is difficult; the single-stage fix gives only slight gains.
• The LSC explanation is not claimed to be exhaustive as the cause of DiT feature degradation.

Experimental setup

Datasets: ImageNet SPair-71k Horse-21 CityScapes NYU Depth v2 DiffusionDB

Benchmarks: SPair-71k Horse-21 CityScapes NYU Depth v2

Metrics: FID PCK@0.1 mIoU d1 mutual information (MINE) Correlation Decay Slope

Baselines: DiT-XL/2 U-ViT Skip-DiT SD1.5 SDXL Flux

Builds on: DiT U-ViT Skip-DiT REPA MINE

Closest papers by meaning

• Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.91)
• DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing (NeurIPS 2026 · similarity 0.90)
• Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features (NeurIPS 2024 · similarity 0.90)
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
• Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling (NeurIPS 2025 · similarity 0.89)
• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)