Research map / Image generation / Autoregressive and latent generation

Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

ICML 2024 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Hourglass Diffusion Transformer (HDiT), a hierarchical pure-transformer diffusion backbone with neighborhood attention that scales linearly with pixel count and trains directly in pixel space up to 1024x1024.

Problem

Transformer diffusion backbones scale quadratically with token count, so high-resolution image synthesis has mostly relied on latent autoencoders or multiscale/self-conditioned tricks; pixel-space transformer diffusion at megapixel resolution is impractical.

Approach

Combine the Hourglass hierarchical transformer (tokens merged/split across levels) with diffusion-transformer blocks, using global attention only at low-resolution levels and neighborhood (local) attention at higher levels, plus axial RoPE, GEGLU, learnable lerp skips and soft-min-SNR loss weighting, so that compute scales O(n) with pixel count.

Key results

• On ImageNet-256^2 (557M params, no CFG) HDiT reaches FID 6.92 and IS 135.2, substantially better than DiT-XL/2 (FID 9.62).
• On FFHQ-1024^2 with an 85M model, HDiT achieves FID 8.48 (5k samples) and sets state-of-the-art DINOv2 Frechet/Kernel distances, with 50k-sample FID 5.23.
• At 256^2 HDiT is more than 10x more efficient in FLOPs than a parameter-matched pixel-space DiT, rising to over 100x at 1024^2.

Measured effects reported in the paper

Surprising findings

• The largest ImageNet-256 patch-size-4 model underperforms smaller models in the scaling sweep, attributed to a fixed learning rate that is too high for large models.
• Scaled cosine similarity attention with the standard tau parametrization was unstable, requiring learnable tau directly.

Limitations

• Does not reach FID of state-of-the-art GANs such as StyleGAN-XL on FFHQ-1024.
• Only unconditional and class-conditional generation evaluated; no text-to-image or super-resolution experiments.
• Large-scale ImageNet model not hyperparameter-tuned; larger models underperform with fixed learning rate.

Experimental setup

Datasets: ImageNet FFHQ

Benchmarks: ImageNet 128^2 ablation FID FFHQ-1024^2 FID ImageNet-256^2 FID/IS

Metrics: FID IS FD_DINOv2 KD_DINOv2 GFLOPs

Baselines: DiT-B/4 NCSN++ StyleGAN2 StyleGAN3 StyleGAN-XL HiT-B StyleSwin DiT-XL/2 ADM RIN simple diffusion VDM++

Builds on: Diffusion Transformer (DiT) Hourglass Transformer Neighborhood Attention Min-SNR loss weighting RoPE

Closest papers by meaning

• Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.90)
• Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.89)
• DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.89)
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models (ICLR 2024 · similarity 0.89)
• Effective Diffusion Transformer Architecture for Image Super-Resolution (AAAI 2025 · similarity 0.89)
• UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction (NeurIPS 2026 · similarity 0.89)