Research map / Image generation / Autoregressive and latent generation
Continuous Visual Autoregressive Generation via Score Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Energy-based AutoRegression (EAR), a continuous visual autoregressive framework trained with the energy score (a strictly proper scoring rule) that avoids vector quantization and generates each continuous token in a single forward pass.
Problem
Visual autoregressive models typically rely on vector quantization, which causes information loss; existing continuous alternatives (GIVT, diffusion loss) are limited in expressiveness or slow due to multi-step denoising.
Approach
Frame continuous autoregressive generation as score maximization over strictly proper scoring rules, and use the likelihood-free energy score as the training loss with an implicit MLP generator that takes noise as input, so each next token is sampled in one forward pass.
Key results
- EAR-B reaches FID 2.83 with guidance and 205M parameters on ImageNet 256x256; EAR-H reaches FID 1.97 with guidance (937M).
- EAR is roughly 10x faster than MAR at comparable FID, producing an image in about 1s on a single A100.
- Energy loss with alpha=1 gives FID 3.55 for EAR-B, while alpha=2 (non-strictly proper) collapses to FID 188.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use continuous energy-based autoregression instead of MAR | ImageNet 256×256 conditional generation · ImageNet 256×256, w/ guidance | FID: 1.55 -> 1.97 | Hurt |
| use continuous energy-based autoregression instead of MAR | ImageNet 256×256 conditional generation · ImageNet 256×256, w/o guidance | FID: 2.35 -> 3.16 | Hurt |
| increase EAR model size from EAR-B to EAR-H | EAR · ImageNet 256×256, w/ guidance | FID: 2.83 -> 1.97 | Helped |
| set energy-loss α to 2.0 instead of 1.0 (ablation) | EAR-B · ImageNet 256×256, 400 epochs, guidance scale 3.0 | FID: 3.55 -> 188.1 | Hurt |
| increase energy-loss α from 1.0 to 1.25 (ablation) | EAR-B · ImageNet 256×256, 400 epochs, guidance scale 3.0 | FID: 3.55 -> 3.73 | Hurt |
| increase energy-loss α from 1.0 to 1.5 (ablation) | EAR-B · ImageNet 256×256, 400 epochs, guidance scale 3.0 | FID: 3.55 -> 4.10 | Hurt |
| increase energy-loss α from 1.0 to 1.75 (ablation) | EAR-B · ImageNet 256×256, 400 epochs, guidance scale 3.0 | FID: 3.55 -> 4.32 | Hurt |
| use uniform instead of Gaussian noise, with d_noise=64 (ablation) | EAR-B · ImageNet 256×256, w/ guidance | FID: 3.60 -> 3.55 | Helped |
| increase uniform noise dimension from d_noise=32 to 64 (ablation) | EAR-B · ImageNet 256×256, w/ guidance | FID: 3.89 -> 3.55 | Helped |
| increase uniform noise dimension from d_noise=64 to 128 (ablation) | EAR-B · ImageNet 256×256, w/ guidance | FID: 3.55 -> 4.34 | Hurt |
| use bidirectional instead of causal attention (ablation) | EAR-B · ImageNet 256×256, w/ guidance | FID: 8.10 -> 3.55 | Helped |
| lower the MLP generator learning rate to λ=0.25× the global rate (ablation) | EAR-B · ImageNet 256×256 training | training stability | Helped |
Surprising findings
- Energy score with alpha=2 (proper but not strictly proper) fails completely (FID 188.1), while alpha in [1,2) works well.
- Causal autoregressive modeling reaches only FID around 17.8 to 20, far worse than masked bidirectional modeling (FID 7.95 without guidance).
Limitations
- Alpha below 1 causes gradient instability and training collapse.
- Global learning rate causes collapse, requiring a reduced learning rate for the MLP generator.
- Causal autoregressive variant performs poorly (FID around 20).
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID Inception Score Precision Recall
Baselines: MAR GIVT VAR-d30 LlamaGen-3B MAGVIT-v2 DiT-XL/2 VDM++ MaskGIT RQ-Transformer StyleGAN-XL
Builds on: MAR GIVT diffusion loss classifier-free guidance VQGAN
Closest papers by meaning
- Autoregressive Image Generation without Vector Quantization (NeurIPS 2024 · similarity 0.89)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.88)
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.87)
- TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation (NeurIPS 2025 · similarity 0.86)
- Guiding Visual Autoregressive Models through Spectrum Weakening (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.