Research map / Image generation / Autoregressive and latent generation
Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoNAT, which automatically optimizes the training (mask ratio distribution) and generation (re-masking ratio, sampling/re-masking temperatures, guidance scale) strategies of non-autoregressive Transformers via alternating optimization, matching latest diffusion models at ~5x lower inference cost.
Problem
Non-autoregressive Transformers (NATs) for image synthesis lag behind diffusion models in quality, partly because their heuristic-designed training and generation strategies (scheduling functions and mask ratio distribution) are likely suboptimal.
Approach
Formulates the design of NAT training and generation strategies as a unified optimization problem solved by an alternating algorithm: finite-difference gradient descent on generation hyperparameters (re-masking ratios, temperatures, guidance scales) and a greedy line search over a Beta-distributed mask ratio distribution.
Key results
- AutoNAT-S reaches FID 4.30 on ImageNet-256 with 4 steps (0.2 TFLOPs), versus 8.40 for the heuristic baseline.
- AutoNAT-L achieves FID 2.68 on ImageNet-256 with 8 steps and 3.74 on ImageNet-512, exceeding compared diffusion models with DPM-Solver at similar or lower compute.
- On CC3M, applying optimized generation strategy to Muse improves FID from 7.67 to 6.90 at the same 8-step cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize training and generation strategies instead of using heuristic configurations | AutoNAT-S · ImageNet-256 | FID-50K: 8.40 -> 4.30 | Helped |
| use a fixed training mask ratio r ≡ 0.8 instead of the heuristic mask-ratio distribution | NAT · ImageNet-256 | FID-50K: 8.40 -> 8.31 | Helped |
| add the optimized training strategy (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 5.21 -> 4.30 | Helped |
| add the optimized generation strategy (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 7.88 -> 4.30 | Helped |
| alternate training and generation strategy optimization rather than optimize concurrently | AutoNAT · ImageNet-256 | FID-50K: 8.01 -> 4.30 | Helped |
| transfer the smaller model's optimized strategy to the base model | Base model · ImageNet-256, 4 steps | FID-50K: 6.56 -> 3.87 | Helped |
| transfer the smaller model's optimized strategy to the large model | Large model · ImageNet-256, 4 steps | FID-50K: 5.42 -> 3.36 | Helped |
| add optimized re-masking ratios r (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 4.58 -> 4.30 | Helped |
| add optimized sampling temperatures τ1 (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 6.57 -> 4.30 | Helped |
| add optimized re-masking temperatures τ2 (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 5.68 -> 4.30 | Helped |
| add optimized guidance scales s (ablation) | AutoNAT-S · ImageNet-256 | FID-50K: 5.43 -> 4.30 | Helped |
Surprising findings
- A fixed mask ratio (r=0.8) for training gives better FID (8.31) than the heuristic mask ratio distribution (8.40).
- Optimized generation strategies transfer from smaller to larger models with only about 0.1 FID loss.
Limitations
- Alternating optimization requires repeatedly training the model for each candidate mask ratio distribution, which is expensive (about 1.9 days on one A100 node).
- Generation strategies are searched per dataset and architecture, with one additional search needed for each new scenario.
Experimental setup
Datasets: ImageNet MS-COCO CC3M
Benchmarks: ImageNet-256 ImageNet-512 MS-COCO CC3M
Metrics: FID IS TFLOPs latency
Baselines: MaskGIT Token-Critic MAGE ADM-G LDM U-ViT DiT Draft-and-revise Muse VQ-Diffusion Frido VQGAN
Builds on: MaskGIT Muse U-ViT VQGAN DPM-Solver
Closest papers by meaning
- Autoregressive Distillation of Diffusion Transformers (CVPR 2025 · similarity 0.90)
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.90)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.89)
- HART: Efficient Visual Generation with Hybrid Autoregressive Transformer (ICLR 2025 · similarity 0.88)
- Neural Diffusion Models (ICML 2024 · similarity 0.88)
- Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.