Research map / Image generation / Autoregressive and latent generation
HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HART, an autoregressive text-to-image model that combines discrete VAR-style tokens with a lightweight 37M-parameter residual diffusion head to directly generate 1024px images at diffusion-comparable quality with much higher throughput.
Problem
Autoregressive visual generation lags diffusion models because discrete tokenizers reconstruct poorly and generating 1024px images with AR models is prohibitively expensive to train and run.
Approach
A hybrid tokenizer splits continuous autoencoder latents into multi-scale discrete VAR tokens plus continuous residual tokens; the discrete part is modeled by a scalable-resolution AR transformer (relative 1D/2D RoPE, text concatenation) and the residual by a small diffusion MLP conditioned on AR hidden states, trained with alternating discrete/continuous decoder training.
Key results
- Hybrid tokenizer reduces 1024px MJHQ-30K rFID from 2.11 (VAR) to 0.30, and generation FID from 7.85 to 5.38 (31% improvement).
- At 1024x1024, HART achieves 4.5-7.7x higher throughput and 3.1-5.9x lower latency than diffusion baselines, with 6.9-13.4x lower MACs.
- On ImageNet class-conditional generation, HART-d30 (2.0B) reaches FID 1.77 vs MAR-L 1.78 with 10.7x fewer MACs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add residual diffusion to model residual tokens (ablation) | HART-d20 · ImageNet class-conditioned generation, 256px | FID: 2.67 -> 2.39 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART-d20 · ImageNet class-conditioned generation, 256px | IS: 297.3 -> 316.4 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART-d24 · ImageNet class-conditioned generation, 256px | FID: 2.23 -> 2.00 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART-d30 · ImageNet class-conditioned generation, 256px | FID: 2.00 -> 1.77 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART · MJHQ-30K 256px text-to-image | FID: 6.11 -> 5.52 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART · MJHQ-30K 512px text-to-image | FID: 6.29 -> 5.22 | Helped |
| Add residual diffusion to model residual tokens (ablation) | HART · MJHQ-30K 1024px text-to-image | FID: 5.73 -> 5.38 | Helped |
| Use the hybrid tokenizer instead of the discrete-only VAR tokenizer | HART tokenizer · MJHQ-30K 1024px reconstruction | rFID: 2.11 -> 0.30 | Helped |
| Use hybrid tokenization and residual diffusion rather than the discrete-only baseline | HART · MJHQ-30K 1024px text-to-image | FID: 7.85 -> 5.38 | Helped |
| Train one decoder with alternating discrete/continuous inputs instead of separate decoders (ablation) | HART hybrid tokenizer · Class-conditioned image generation | generation convergence | Helped |
| Replace absolute position embeddings with relative position embeddings (ablation) | HART scalable-resolution transformer · Higher-resolution finetuning | convergence | Helped |
| Discard 80% of final-step tokens and supervise subsampled tokens (ablation) | HART residual diffusion training · 512px and 1024px training | training speed: 1.4× at 512px; 1.9× at 1024px | Helped |
Surprising findings
- Low reconstruction FID does not necessarily imply better generation FID; separate decoders for continuous and discrete tokens matched rFID but hurt generation FID.
Limitations
- Relies on large synthetic recaptioned and internal MidJourney-style data for text-to-image training.
- Token subsampling during training assumes mostly local attention patterns in the last stage, which is an empirical observation.
Experimental setup
Datasets: ImageNet JourneyDB OpenImages MJHQ-30K GenEval DPG-Bench
Benchmarks: MJHQ-30K GenEval DPG-Bench ImageNet
Metrics: FID CLIP score rFID IS throughput latency MACs
Baselines: SDXL PixArt-Sigma Playground v2.5 SD3-medium SD v2.1 PixArt-alpha LlamaGen Show-o VAR MAR DiT-XL/2
Builds on: VAR MAR VQGAN DiT Llama
Closest papers by meaning
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.89)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.88)
- Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis (CVPR 2024 · similarity 0.88)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.88)
- RandAR: Decoder-only Autoregressive Visual Generation in Random Orders (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.