Research map / Image generation / Autoregressive and latent generation
Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiMR, a multi-branch diffusion denoiser that refines features from low to high resolution (Transformer at lowest resolution, ConvNeXt at higher ones) plus TD-LN, a parameter-efficient time conditioning that makes layer-norm affine parameters functions of time, reaching SOTA FID on ImageNet 256 and 512.
Problem
Transformer diffusion models face a trade-off between visual fidelity and compute from patchification: large patches cause image distortion, while small patches are expensive due to quadratic self-attention. Existing time conditioning (adaLN) also adds many parameters and cannot be applied directly
Approach
A feature cascade: a multi-branch network where each branch processes one resolution, with lower-resolution features upsampled and added to the next branch, so the final branch operates at input resolution; combined with TD-LN, which models layer-norm gamma and beta as sigmoid-interpolations between learnable parameters over time, avoiding a parameter-heavy MLP.
Key results
- DiMR-XL/2R achieves FID 1.70 with CFG on ImageNet 256x256 (800 epochs) and DiMR-G/2R reaches 1.63, state of the art among diffusion models in the paper's comparison.
- DiMR-XL/3R achieves FID 2.89 with CFG on ImageNet 512x512, outperforming DiT-XL/2 (3.04) with about 39% of the GFLOPs.
- On ImageNet 64x64 without CFG, DiMR-M/3R (133M) reaches FID 3.65 vs. U-ViT-M/4's 5.85; TD-LN reduces parameters versus adaLN-Zero (154.0M vs 217.9M in the ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AdaLN-Zero to U-ViT-M/4 (ablation) | U-ViT-M/4 · ImageNet 64×64 | FID: 5.85 to 5.44 | Helped |
| add a multi-branch network to the AdaLN-Zero configuration (ablation) | U-ViT-M/4 with AdaLN-Zero · ImageNet 64×64 | FID: 5.44 to 7.91 | Hurt |
| replace AdaLN-Zero with TD-LN in the multi-branch network (ablation) | AdaLN-Zero multi-branch model · ImageNet 64×64 | FID: 7.91 to 5.21 | Helped |
| replace the vanilla MLP block with GLU (ablation) | TD-LN multi-branch model · ImageNet 64×64 | FID: 5.21 to 4.86 | Helped |
| add multi-scale loss (ablation) | TD-LN, multi-branch, GLU model · ImageNet 64×64 | FID: 4.86 to 3.65 | Helped |
| add a multi-branch design with TD-LN, GLU, and multi-scale loss (ablation) | U-ViT-M/4 baseline · ImageNet 64×64 | FID: 5.85 to 3.65 | Helped |
| use Transformer rather than ConvNeXt blocks in the lowest-resolution branch (ablation) | DiMR-XL/2R · ImageNet 256×256, CFG, 400 epochs | FID: 2.09 to 1.77 | Helped |
| use addition rather than concatenation for feature cascading (ablation) | DiMR-XL/2R · ImageNet 256×256, CFG, 400 epochs | FID: 2.06 to 1.77 | Helped |
| add skip connections between branches (ablation) | DiMR-XL/2R · ImageNet 256×256, CFG, 400 epochs | FID: 1.77 to 1.96 | Hurt |
| use DiMR-M/3R instead of U-ViT-M/4 | class-conditional ImageNet generation · ImageNet 64×64 | FID: 5.85 to 3.65 | Helped |
| use DiMR-XL/2R instead of DiT-XL/2 | class-conditional ImageNet generation · ImageNet 256×256, CFG | FID: 2.27 to 1.70 | Helped |
| use DiMR-XL/3R instead of DiT-XL/2 | class-conditional ImageNet generation · ImageNet 512×512, CFG | FID: 3.04 to 2.89 | Helped |
Surprising findings
- AdaLN-Zero, which helps the U-ViT baseline slightly, degrades the multi-branch network from 5.44 to 7.91 FID, while TD-LN improves it to 5.21.
- Human evaluation found only 29.2% of low-fidelity DiMR images were truly distorted, versus 63.5% and 71.0% for baselines.
Limitations
- Only class-conditional generation is studied, not text-to-image.
- Scaling exploration stops at about 525M parameters for 512x512 due to compute limits.
- Hyperparameters were inherited from U-ViT without tuning.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet 64x64 ImageNet 256x256 ImageNet 512x512
Metrics: FID-50K Inception Score Precision/Recall
Baselines: DiT-XL/2 U-ViT-H/2 U-ViT-M/4 ADM-U LDM-4 MDTv2-G VAR-d30 DiffiT
Builds on: DiT U-ViT adaLN-Zero classifier-free guidance ConvNeXt
Closest papers by meaning
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.90)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.89)
- ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models (ICLR 2024 · similarity 0.89)
- ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion (NeurIPS 2025 · similarity 0.89)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.88)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.