Research map / Image generation / Autoregressive and latent generation
FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexVAR, a visual autoregressive model that predicts ground-truth multi-scale latents at each step instead of residuals, enabled by a scalable multi-scale VQVAE and scalable 2D positional embeddings, allowing flexible resolutions, aspect ratios, and step counts.
Problem
Residual prediction in scale-wise visual autoregressive modeling (VAR) imposes a rigid step design, limiting generation flexibility across resolutions, aspect ratios, and inference steps.
Approach
Replace residual prediction with ground-truth prediction at each scale so each autoregressive step independently yields a plausible image; this is made possible by a scalable VQVAE trained on randomly downsampled multi-scale latents and by scalable 2D sin-cosine-initialized positional embeddings without step embeddings.
Key results
- FlexVAR-d24 (1.0B) reaches 2.21 FID on ImageNet 256 with 10 steps versus 2.33 for VAR-d24, and 2.08 FID with 13 zero-shot steps.
- Zero-shot 512x512 generation with FlexVAR-d24 (training only at <=256) achieves 4.43 FID vs 2.63 for VAR-d36 (2.3B, trained at 512).
- Replacing residual with GT prediction using the scalable tokenizer and scalable PE gives 3.71 FID in the ablation (d20, 40 epochs), versus 4.00 for residual prediction with the VAR tokenizer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace VAR residual prediction with GT prediction using the VAR tokenizer (ablation) | VAR · ImageNet-1K 40-epoch ablation | FID: 4.00 -> N/A | Hurt |
| Use the LlamaGen tokenizer for GT prediction (ablation) | VAR · ImageNet-1K 40-epoch ablation | FID: 4.00 -> 17.75 | Hurt |
| Use the scalable FlexVAR tokenizer for GT prediction (ablation) | VAR · ImageNet-1K 40-epoch ablation | FID: 17.75 -> 3.82 | Helped |
| Use scalable rather than fixed-length positional embeddings (ablation) | GT prediction with FlexVAR tokenizer · ImageNet-1K 40-epoch ablation | FID: 3.82 -> 3.71 | Helped |
| Replace Transformer with Mamba (ablation) | FlexVAR-d16 · ImageNet-1K 40-epoch ablation | FID: 4.32 -> 4.22 | Helped |
| Replace Transformer with Mamba (ablation) | FlexVAR-d20 · ImageNet-1K 40-epoch ablation | FID: 3.71 -> 3.80 | Hurt |
| Remove step positional embedding (ablation) | FlexVAR with fixed-length learnable coordinate embeddings · ImageNet-1K 40-epoch ablation | FID: 3.82 -> 3.87 | No clear effect |
| Replace fixed-length coordinates with scalable non-learnable coordinates (ablation) | FlexVAR without step PE · ImageNet-1K 40-epoch ablation | FID: 3.87 -> 3.74 | Helped |
| Use scalable learnable coordinate embeddings instead of scalable non-learnable coordinates (ablation) | FlexVAR without step PE · ImageNet-1K 40-epoch ablation | FID: 3.74 -> 3.71 | Helped |
| Increase inference steps from 10 to 13 | FlexVAR-d24 · ImageNet-1K 256×256 | FID: 2.21 -> 2.08 | Helped |
| Use FlexVAR-VAE instead of VAR-VAE | ImageNet tokenizer reconstruction · ImageNet validation | rFID: 1.92 -> 3.79 | Hurt |
Surprising findings
- Using the VAR or LlamaGen tokenizers directly with GT prediction either fails to converge (N/A) or yields FID 17.75, showing tokenizers must be trained for multi-scale latents.
- Mamba variant of FlexVAR achieves competitive results to the transformer, though the authors did not adopt it due to lack of speed advantage.
Limitations
- Generating images at 3x or more the training resolution produces wavy textures and blur.
- Attributed to homogeneous ImageNet-1K training data lacking multi-scale structure.
- FlexVAR-VAE has worse reconstruction (rFID 3.79) than VAR-VAE (1.92) due to fewer epochs and smaller batch size.
Experimental setup
Datasets: ImageNet-1K OpenImages
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID IS precision recall rFID inference time
Baselines: VAR AiM LlamaGen-XL DiT-XL/2 LDM-4-G MaskGIT RCG StyleGAN-XL BigGAN GigaGAN
Builds on: VAR LlamaGen VQ-VAE Mamba
Closest papers by meaning
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.92)
- FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models (ICML 2026 · similarity 0.91)
- DVAR: Dynamic Visual Autoregressive Modeling for Image Super-Resolution (CVPR 2026 · similarity 0.91)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.88)
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.88)
- Visual Autoregressive Modeling for Image Super-Resolution (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.