Research map / Image generation / Autoregressive and latent generation
Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hi-MAR, a hierarchical masked autoregressive model that first predicts low-resolution continuous image tokens as pivots, then predicts dense tokens with a Diffusion Transformer head, improving on MAR for class-conditional ImageNet and text-to-image MS-COCO generation.
Problem
Single-scale next-token and masked autoregressive image generators lack global context for early token prediction, and MAR's MLP diffusion head models each masked token independently, ignoring inter-token dependencies.
Approach
A two-phase hierarchy in which low-resolution continuous tokens are generated first and their conditional outputs (not ground-truth tokens) guide dense-token prediction, combined with a scale-aware adaLN-Zero Transformer and a Diffusion Transformer head that attends across all tokens.
Key results
- Hi-MAR-B achieves FID 1.93 with CFG on ImageNet 256, improving over MAR-B (2.31) by 0.38; Hi-MAR-H reaches FID 1.52 with CFG.
- On MS-COCO text-to-image, Hi-MAR-S achieves FID 4.77, 0.59 better than AutoNAT-S (5.36).
- Ablation: conditional tokens instead of visual tokens for pivots and Transformer-based second diffusion head reduce FID from 2.31 to 1.98; scale vector gives 1.93.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical low-resolution pivots with conditional-token guidance rather than single-scale MAR | Hi-MAR-B · ImageNet 256×256 class-conditional, w/ CFG | FID: 2.31 -> 1.93 | Helped |
| Use hierarchical low-resolution pivots and conditional-token guidance | Hi-MAR-S · MS-COCO 256×256 validation text-to-image | FID: 5.36 -> 4.77 | Helped |
| Use Hi-MAR-S instead of U-ViT-S/2 (Deep) | Text-to-image generation · T2I-CompBench Object Relationship Spatial | score: 0.0353 -> 0.0409 | Hurt |
| Use Hi-MAR-S instead of U-ViT-S/2 (Deep) | Text-to-image generation · T2I-CompBench Object Relationship Non-Spatial | score: 0.2693 -> 0.2690 | Hurt |
| Add hierarchical modeling with visual-token pivots (ablation) | MAR · ImageNet 256×256 class-conditional ablation | FID: 2.31 -> 2.28 | Helped |
| Use conditional tokens rather than visual tokens as pivots (ablation) | Hierarchical MAR · ImageNet 256×256 class-conditional ablation | FID: 2.28 -> 2.07 | Helped |
| Replace the second-phase MLP diffusion head with a Diffusion Transformer head (ablation) | Hi-MAR with conditional-token pivots · ImageNet 256×256 class-conditional ablation | FID: 2.07 -> 1.98 | Helped |
| Replace the first-phase MLP diffusion head with a Transformer head (ablation) | Hi-MAR with a Transformer second-phase head · ImageNet 256×256 class-conditional ablation | FID: 1.98 -> 1.98 | No clear effect |
| Add scale vectors to the scale-aware Transformer (ablation) | Hi-MAR with conditional-token pivots and a second-phase Transformer head · ImageNet 256×256 class-conditional ablation | FID: 1.98 -> 1.93 | Helped |
Surprising findings
- Using low-resolution visual tokens directly as guidance gave almost no FID gain (2.31 to 2.28), attributed to training-inference discrepancy; conditional tokens were needed.
Limitations
- Quality of the second phase depends strongly on first-phase low-resolution output; CFG is disabled in the first phase for the w/o CFG setting.
- Text-to-image evaluation is limited to MS-COCO at 256x256 with small models (Hi-MAR-S).
Experimental setup
Datasets: ImageNet MS-COCO
Benchmarks: ImageNet 256x256 MS-COCO 256x256 T2I-CompBench
Metrics: FID IS Precision Recall inference speed
Baselines: MAR DiT-XL/2 LlamaGen VAR AutoNAT-S U-ViT-S/2 (Deep) MaskGIT GIVT
Builds on: MAR DiT MaskGIT AutoNAT
Closest papers by meaning
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.91)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.90)
- ControlAR: Controllable Image Generation with Autoregressive Models (ICLR 2025 · similarity 0.90)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.90)
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.90)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.