Research map / Image generation / Autoregressive and latent generation
Image Understanding Makes for A Good Tokenizer for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that image tokenizers trained by feature reconstruction (distilling pretrained image-understanding encoders, VQ-KD) yield better autoregressive and non-autoregressive image generation than pixel-reconstruction tokenizers like VQGAN, and proposes clustering pretrained encoders directly as tokenizers.
Problem
Pixel-reconstruction objectives dominate image tokenizer training for token-based image generation, and the potential of image understanding (IU) encoders to improve image generation (IG) remains unexplored.
Approach
Train the tokenizer to reconstruct features of a frozen pretrained IU encoder (VQ-KD) rather than pixels, and show that stronger IU teachers yield better generation; additionally, cluster the features of pretrained IU encoders directly to form codebooks.
Key results
- VQ-KD CLIP tokenizer with a 1.4B AR proposal network achieves 4.10 FID on ImageNet-1k class-conditional generation, vs. 15.78 for VQGAN.
- On MS-COCO text-to-image, VQ-KD CLIP reaches FID 11.17 vs. 24.11 for VQGAN.
- Clustering pretrained ViT features into a codebook (no tokenizer training) gives 13.40 FID_AR on MS-COCO, outperforming all trained tokenizers in the comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pixel reconstruction with feature distillation from an IU teacher | image tokenizers with AR proposal networks · ImageNet-1k class-conditional | FID: 24.11 -> 11.78 | Helped |
| replace pixel reconstruction with feature distillation from an IU teacher | image tokenizers with NAR proposal networks · ImageNet-1k class-conditional | FID: 20.03 -> 8.45 | Helped |
| use VQ-KD CLIP instead of VQGAN | 1.4B AR generator · ImageNet-1k 256×256 class-conditional | FID: 15.78 -> 4.10 | Helped |
| use VQ-KD DINO instead of VQGAN | AR generator · MS-COCO unconditional | FID: 38.43 -> 17.55 | Helped |
| use VQ-KD CLIP instead of VQGAN | text-to-image generator · MS-COCO text-to-image | FID: 24.11 -> 11.17 | Helped |
| use larger OpenCLIP teacher (ViT-L/14 to ViT-G/14) (ablation) | VQ-KD tokenizer · ImageNet-1k class-conditional | FID AR: 10.31 -> 8.70 | Helped |
| increase codebook size from 2^10 to 2^14 (ablation) | VQ-KD CLIP · ImageNet-1k class-conditional | FID AR: 11.65 -> 11.61 | No clear effect |
| increase code dimension from 16 to 128 (ablation) | VQ-KD CLIP · ImageNet-1k class-conditional | FID AR: 12.19 -> 10.50 | Helped |
| apply strong random-crop augmentation (ablation) | VQ-KD ViT · ImageNet-1k class-conditional | FID AR: 11.40 -> 13.32 | Hurt |
| apply strong random-crop augmentation (ablation) | VQ-KD MAE · ImageNet-1k class-conditional | FID AR: 26.85 -> 36.63 | Hurt |
| scale AR proposal network to GPT-2 XL (ablation) | VQ-KD MAE · ImageNet-1k class-conditional | FID AR: 26.85 -> 17.11 | Helped |
| scale AR proposal network to GPT-2 XL (ablation) | VQ-KD CLIP · ImageNet-1k class-conditional | FID AR: 11.78 -> 11.27 | Helped |
Surprising findings
- FSQ, despite 100% codebook usage, has much higher PPL than VQGAN and worse generation, showing high codebook usage alone does not help AR modeling.
- Stronger IU teachers gain less from a larger AR proposal network, while weaker teachers (MAE) benefit more from scaling.
Limitations
- VQ-KD tokenizers may alter fine visual details during reconstruction, which could hinder image editing.
- Cluster-based tokenizers have worse rFID because they encode little appearance detail.
- No error bars are reported due to computational cost.
Experimental setup
Datasets: ImageNet-1k MS-COCO
Benchmarks: ImageNet-1k MS-COCO
Metrics: FID IS rFID PPL codebook usage CLIP score
Baselines: VQGAN FSQ RQ-VAE ViT-VQGAN MoVQ MaskGIT LDM-8-G CVQ-VAE
Builds on: VQ-KD VQGAN FSQ MaskGIT MAGE
Closest papers by meaning
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.92)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.91)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.90)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.90)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.89)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.