Research map / Image generation / Autoregressive and latent generation
ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes spatial and temporal token interactions in non-autoregressive transformers (NATs) for image synthesis and proposes ENAT, which disentangles visible/[MASK] token computation and reuses previous-step features to cut compute while improving FID.
Problem
NATs for token-based image generation spend equal computation on visible and [MASK] tokens at every step and recompute all tokens each step, which is inefficient; the paper asks what interaction patterns make NATs work and how to exploit them.
Approach
Empirical analysis shows [MASK] tokens mainly gather information from visible tokens and only newly decoded tokens change much across steps; ENAT encodes visible tokens independently, decodes [MASK] tokens with SC-Attention over them, and reuses projected previous-step visible features so only new tokens are recomputed.
Key results
- ENAT-L reaches FID 2.79 on ImageNet 256x256 at 8 steps with 0.3 TFLOPs, versus U-ViT-H at FID 3.37 with 2.4 TFLOPs.
- ENAT-L achieves FID 4.00 on ImageNet 512x512 with 1.3 TFLOPs at 8 steps.
- Ablation: disentangled architecture lowers FID from 6.54 to 4.78 at similar GFLOPs; computation reuse cuts GFLOPs from 39.8 to 22.6 (about 1.8x) at FID 4.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ENAT-L instead of U-ViT-H | ImageNet class-conditional generation · ImageNet 256×256 | FID: 3.37 -> 2.79 | Helped |
| use ENAT-L instead of U-ViT-H | ImageNet class-conditional generation · ImageNet 256×256 | TFLOPs: 2.4 TFLOPs vs 0.3 TFLOPs | Helped |
| use ENAT-L | ImageNet class-conditional generation · ImageNet 512×512 | FID: 4.00 vs 4.03 | Helped |
| use ENAT-B instead of U-ViT | Text-to-image generation · MS-COCO | FID: 16.20 -> 6.82 | Helped |
| disentangle visible-token encoding from mask-token decoding and prioritize visible-token computation (ablation) | Baseline NAT · ImageNet 256×256 | FID: 6.54 -> 4.78 | Helped |
| add computation reuse to the disentangled architecture (ablation) | ENAT-S · ImageNet 256×256 | GFLOPs: 39.8 -> 22.6 | Helped |
| add computation reuse to the disentangled architecture (ablation) | ENAT-S · ImageNet 256×256 | FID: 4.78 -> 4.97 | Hurt |
| use SC-Attention instead of alternating self-attention and cross-attention (ablation) | ENAT-S with reuse · ImageNet 256×256 | FID: 5.85 -> 4.97 | Helped |
| use SC-Attention instead of alternating self-attention and cross-attention (ablation) | ENAT-S with reuse · ImageNet 256×256 | GFLOPs: 25.0 -> 22.6 | Helped |
| add the reuse projection module (ablation) | ENAT-S with reuse · ImageNet 256×256 | FID: 5.96 -> 4.97 | Helped |
| reuse only visible-token features from the previous step (ablation) | ENAT-S with reuse · ImageNet 256×256 | GFLOPs: 37.5 -> 22.6 | Helped |
| reuse only visible-token features from the previous step (ablation) | ENAT-S with reuse · ImageNet 256×256 | FID: 4.95 -> 4.97 | No clear effect |
Surprising findings
- Removing visible-to-[MASK] attention only marginally hurts performance, while [MASK]-to-visible attention is essential for convergence.
- Allocating more layers to visible-token encoding and only one decoder layer for [MASK] tokens improves FID (4.78) at similar cost.
Limitations
- Evaluated only on ImageNet and MS-COCO; not on CelebA, LSUN, or unconditional generation.
- Largest model is about 0.6B parameters and dataset is at most 1.2M images; scalability to larger models and data is untested.
- Reuse requires a training-time reuse mode and fixed step schedule; generalization to other adaptive inference settings is left to future work.
Experimental setup
Datasets: ImageNet MS-COCO
Benchmarks: ImageNet 256x256 ImageNet 512x512 MS-COCO
Metrics: FID IS TFLOPs GFLOPs GPU/CPU latency
Baselines: MaskGIT Token-Critic Draft-and-revise MAGE MaskGIT-FSQ AdaNAT DiT-XL U-ViT-H LDM MDT-XL VQ-Diffusion Frido
Builds on: MaskGIT VQGAN MAE CrossMAE U-ViT
Closest papers by meaning
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.87)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.87)
- OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (NeurIPS 2024 · similarity 0.87)
- Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.