Research map / Image generation / Autoregressive and latent generation
Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an image tokenizer on a frozen vision foundation model (DINOv2-L) using region-adaptive deformable-attention sampling into 256 tokens, a shared ViT decoder, and a VFM feature reconstruction loss, enabling faster AR convergence and CFG-free generation (gFID 1.36 with RAR-XL).
Problem
Visual tokenizers for autoregressive image generation are trained from scratch for reconstruction, yielding latent spaces with weak semantics and redundancy that slow AR training and require classifier-free guidance; it is unclear whether frozen vision foundation models can serve as effective tokeni
Approach
Use a frozen VFM as the tokenizer encoder, sample irregular region-level tokens from multi-level VFM features with learnable anchor queries and deformable attention, quantize them, and decode both pixels (via VQGAN decoder) and the VFM's last-layer features through a shared causal ViT.
Key results
- RAR-XL with VFMTok (1.5B) reaches gFID 1.36 with CFG on ImageNet 256x256, reported as state of the art.
- VFMTok with 256 tokens achieves rFID 0.89 and rIS 215.4 versus VQGAN's 576 tokens at rFID 0.95 and rIS 197.3.
- VFMTok enables about 3x faster AR convergence than VQGAN, and VFMTok-XXL (1.4B) achieves gFID 1.95 without CFG versus LlamaGen-3B's 9.38.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a frozen VFM encoder (ablation) | VQGAN tokenizer · ImageNet, image reconstruction | rIS: 197.3 -> 206.3 | Helped |
| add region-adaptive tokenization and reduce tokens from 576 to 256 (ablation) | VFM-based tokenizer · ImageNet, image reconstruction | rFID: 0.99 -> 1.20 | Hurt |
| add multi-level features (ablation) | VFMTok with region-adaptive quantization · ImageNet, image reconstruction | rFID: 1.20 -> 0.92 | Helped |
| add a feature reconstruction objective (ablation) | VFMTok with multi-level features · ImageNet, image reconstruction | rFID: 0.92 -> 0.89 | Helped |
| add a feature reconstruction objective (ablation) | VFMTok with multi-level features · ImageNet, AR generation | gFID: 3.71 -> 3.42 | Helped |
| remove the frozen VFM encoder (ablation) | VFMTok · ImageNet, AR generation | gFID: 3.42 -> 3.73 | Hurt |
| share the ViT between image and feature reconstruction (ablation) | VFMTok · ImageNet, image reconstruction | rFID: 0.91 -> 0.89 | Helped |
| replace deformable attention with cross-attention (ablation) | VFMTok query-feature interaction · ImageNet, AR generation | gFID: 3.42 -> 3.89 | Hurt |
| remove register tokens (ablation) | VFMTok · ImageNet, AR generation | gFID: 3.42 -> 3.50 | Hurt |
| remove classifier-free guidance (ablation) | VFMTok-XXL · ImageNet, class-conditional generation | gFID: 2.19 -> 1.95 | Helped |
| remove classifier-free guidance (ablation) | LlamaGen-3B · ImageNet, class-conditional generation | gFID: 2.19 -> 9.38 | Hurt |
| increase the number of tokens from 36 to 576 (ablation) | VFMTok · ImageNet, image reconstruction | rFID: 2.61 -> 0.60 | Helped |
Surprising findings
- A random-initialized encoder in the same framework drops reconstruction to VQGAN-level, while a frozen VFM encoder gives both better reconstruction and generation.
- Reducing tokens to 144 still performs reasonably, suggesting further redundancy in region-level representations.
Limitations
- Tokenizer and AR training are limited to ImageNet class-conditional generation; text-conditional results are not reported.
- Gains from frozen VFM choice vary, and performance correlates with VFM representation strength, with no guarantee for other VFMs.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 class-conditional generation ImageNet reconstruction
Metrics: gFID rFID gIS rIS sFID precision recall linear probing accuracy codebook usage
Baselines: VQGAN LlamaGen RAR TiTok ImageFolder MaskGiT VAR ViT-VQGAN DiT SiT FasterDiT MaskDiT
Builds on: VQGAN LlamaGen RAR DINOv2 deformable attention TiTok
Closest papers by meaning
- ImageFolder: Autoregressive Image Generation with Folded Tokens (ICLR 2025 · similarity 0.91)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.90)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.90)
- Channel-wise Vector Quantization (NeurIPS 2026 · similarity 0.90)
- LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (ICLR 2025 · similarity 0.90)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.