Research map / Image generation / Autoregressive and latent generation
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that better VQ tokenizer reconstruction does not improve generation, and proposes SeQ-GAN, which uses a semantic-enhanced perceptual loss plus decoder-only finetuning to balance semantic compression and detail preservation for AR/NAR image generation.
Problem
VQ tokenizers for VQ-based image generators are usually optimized for reconstruction fidelity, but it is unclear how reconstruction quality affects the generation ability of downstream generative transformers.
Approach
Introduces a visualization pipeline that feeds ground-truth index contexts to an AR transformer to compare tokenizers, finds that better reconstruction does not imply better generation, and proposes a two-phase tokenizer training: semantic-enhanced perceptual loss for the encoder/codebook, then decoder-only finetuning for details.
Key results
- SeQ-GAN + AR-L (364M) reaches FID 6.25 and IS 140.9 on 256x256 ImageNet class-conditional generation, versus VIT-VQGAN (714M) at 11.20 FID and 97.2 IS.
- SeQ-GAN + NAR-L (364M, 12 steps) reaches FID 4.55 and IS 200.4 on ImageNet, compared with MaskGIT at 6.18 FID.
- In the ablation, the attention-enhanced decoder lowered rFID from 3.45 to 2.90 but raised AR generation FID from 16.97 to 17.42, showing reconstruction gains did not translate into generation gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add two attention blocks to the jointly trained decoder (ablation) | baseline VQ tokenizer and generative transformer · VQ tokenizer with AR transformer | generation FID: 16.97 -> 17.42 | Hurt |
| Add two attention blocks to the jointly trained decoder (ablation) | baseline VQ tokenizer and generative transformer · VQ tokenizer with NAR transformer | generation FID: 13.26 -> 14.02 | Hurt |
| Increase semantic ratio in tokenizer perceptual loss (ablation) | VQ tokenizer with AR transformer · ImageNet generation | generation FID | Helped |
| Use factorized and L2-normed codes (ablation) | VQGAN tokenizer · ImageNet generation (AR) | FID: 17.19 -> 24.91 | Hurt |
| Use offline K-means codebook clustering (ablation) | VQGAN tokenizer · ImageNet generation (AR) | FID: 17.19 -> 16.84 | Helped |
| Add entropy regularization to codebook learning (ablation) | VQGAN tokenizer · ImageNet generation (AR) | FID: 17.19 -> 16.97 | Helped |
| Replace low-level perceptual loss with semantic-enhanced perceptual loss (ablation) | VQ tokenizer · ImageNet generation (AR) | FID: 16.97 -> 12.07 | Helped |
| Replace low-level perceptual loss with semantic-enhanced perceptual loss (ablation) | VQ tokenizer · ImageNet generation (NAR) | FID: 13.26 -> 8.84 | Helped |
| Remove shallow VGG perceptual-loss layers (ablation) | VQ tokenizer · ImageNet generation (AR) | FID: 16.97 -> 13.34 | Helped |
| Remove the spatial feature and use only the logit feature (ablation) | VQ tokenizer · ImageNet generation (AR) | FID: 16.97 -> 17.88 | Hurt |
| Remove second-phase decoder finetuning (ablation) | SeQ-GAN · ImageNet generation (AR) | FID: 7.55 -> 7.83 | Hurt |
| Remove second-phase decoder finetuning (ablation) | SeQ-GAN · ImageNet generation (NAR) | FID: 4.99 -> 5.31 | Hurt |
Surprising findings
- Enhancing the decoder improves reconstruction fidelity yet degrades generation quality, so the baseline tokenizer achieves the best generation results in the ablation.
- Removing shallow perceptual layers and adding the logit feature improves generation FID even though it hurts reconstruction in some variants.
Limitations
- Reconstruction fidelity is not the best among compared tokenizers (MoVQ and VIT-VQGAN achieve lower rFID)
- The semantic ratio alpha must be tuned, and the loss-scale balancing requires extensive tuning per the authors' remarks
- Evaluated only on ImageNet, FFHQ and LSUN image datasets
Experimental setup
Datasets: ImageNet FFHQ LSUN
Benchmarks: ImageNet 256x256 FFHQ LSUN-Church LSUN-Cat LSUN-Bedroom
Metrics: FID IS rFID codebook usage
Baselines: VIT-VQGAN RQ-VAE MoVQ MaskGIT VQGAN ADM StyleGAN2 VQ-Diffusion BigGAN-Deep DCT
Builds on: VQGAN MaskGIT MoVQ VIT-VQGAN RQ-VAE
Closest papers by meaning
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.90)
- Efficient Generative Modeling with Residual Vector Quantization-Based Tokens (ICML 2025 · similarity 0.89)
- SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · similarity 0.88)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.88)
- Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99% (NeurIPS 2024 · similarity 0.87)
- Highly Compressed Tokenizer Can Generate Without Training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.