Research map / Image generation / Autoregressive and latent generation
Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper proves that single-layer, single-head VAR transformers (and FlowAR/HOFAR variants) are universal approximators for Lipschitz word-to-image and image-to-image mappings, via a theoretical construction with no experiments.
Problem
Lack of theoretical understanding of why Visual Autoregressive (VAR) transformers and flow-based autoregressive models (FlowAR) perform well in image generation; the question is whether they are universal approximators.
Approach
Extends universal approximation results for minimal (single-layer, single-head) transformers with prompt/soft-token constructions to the VAR pyramid up-interpolation architecture, composing per-scale approximation errors with Lipschitz perturbation bounds, and transfers the result to FlowAR and HOFAR.
Key results
- Theorem 6.6: a properly constructed single-layer, single-head VAR transformer approximates any Lipschitz word-to-image function with error O(ε) in the norm.
- Corollary 7.2: FlowAR models achieve the same O(ε) universal approximation guarantee.
- Corollary 7.3: HOFAR models achieve the same O(ε) guarantee, since errors of the two blocks add linearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a single-layer, single-head VAR Transformer with up-interpolation | VAR Transformer · Lipschitz image-to-image mappings | approximation error: ≤ O(ϵ) | Helped |
| Extend the autoregressive transformer with flow matching | FlowAR · Lipschitz word-to-image mappings | approximation error: ≤ O(ϵ) | Helped |
| Use first- and second-order flow predictions | HOFAR · Lipschitz word-to-image mappings | approximation error: ≤ O(ϵ) | Helped |
Surprising findings
- Even the minimal single-layer, single-head configuration suffices for universal approximation, with the number of FFN neurons scaling as O(ε^{-d h_r w_r}) in the construction.
Limitations
- Purely theoretical; no experiments validate the construction or its practical trained behavior.
- Construction requires exponentially many MLP neurons in the number of tokens, so the bound is not practically efficient.
- Assumes Lipschitz target functions and that interpolation layers approximate the given Lipschitz maps.
Experimental setup
Builds on: Yun et al. 2020 universal approximation of transformers Hu et al. 2024b single-layer prompt-tuned transformer universality Ke et al. 2025a VAR theoretical analysis FlowAR (Ren et al. 2024) HOFAR (Liang et al. 2025b)
Closest papers by meaning
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.93)
- Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis (CVPR 2024 · similarity 0.90)
- HART: Efficient Visual Generation with Hybrid Autoregressive Transformer (ICLR 2025 · similarity 0.89)
- A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation (ICLR 2025 · similarity 0.89)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.88)
- Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.