Research map / Image generation / Autoregressive and latent generation

Rethinking Cross-Layer Information Routing in Diffusion Transformer

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Diagnoses residual-stream routing in DiTs (magnitude inflation, gradient decay, block redundancy) and proposes DAR, a timestep-adaptive softmax attention over preceding sublayer outputs that replaces the fixed residual sum, improving SiT-XL/2 on ImageNet 256 and stacking with REPA.

Problem

Standard pre-norm residual routing in Diffusion Transformers is fixed, time-agnostic and uniformly weighted, leading to PreNorm dilution-like symptoms across depth, and it may be suboptimal for the time-varying denoising process.

Approach

Replace the unit-weight residual sum with a learnable softmax attention over all preceding sublayer outputs, with queries (static, dynamic, or timestep-injected) that make routing timestep-adaptive, using chunked summaries to bound memory.

Key results

• DAR static c4 reaches FID 6.92 (SDE, no CFG) on ImageNet 256 at 600K iterations versus 9.67 ODE for SiT baseline at 1.75M iterations.
• DAR matches the SiT baseline's quality (FID 9.67) in roughly 8.75x fewer iterations, and DAR+REPA reaches FID 7.09 at 100K vs 9.89 for REPA.
• Under SDE with CFG, DAR static c4 outperforms U-DiT-L by 0.77 FID with 0.83x parameters.

Surprising findings

• Dynamic query variant gives a sharp departure from the LLM AttnRes finding, where dynamic queries improve only marginally over static ones.
• Counterfactual gate gradients on the unmodified SiT baseline already vary with timestep, suggesting latent timestep-dependent source preferences.

Limitations

• Experiments mostly on SiT-XL/2 at 256x256 ImageNet; scaling to large MM-DiT/video DiT is left to future work.
• Large-scale T2I evaluation is limited to a preliminary DMD experiment with qualitative and small quantitative comparison.
• Chunk size analysis relies on a simplified rate-distortion model with an alpha parameter fit only qualitatively.

Experimental setup

Datasets: ImageNet-1K GenEval2 prompts

Benchmarks: ImageNet 256x256

Metrics: FID sFID IS Precision Recall LPIPS RAPSD high-frequency ratio deviation

Baselines: DiT SiT SiT-Plus U-ViT U-DiT REPA Attention Residuals

Builds on: Attention Residuals SiT REPA DiT U-ViT

Closest papers by meaning

• ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.90)
• Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.89)
• Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts (ICML 2025 · similarity 0.89)
• BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers (CVPR 2025 · similarity 0.89)
• UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction (NeurIPS 2026 · similarity 0.89)
• Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.89)