Research map / Image generation / Autoregressive and latent generation
Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQGAN-LC, which freezes a 100K codebook initialized by K-means over CLIP ViT-L/14 features and trains only a linear projector, reaching over 99% utilization and better reconstruction/generation than VQGAN-FC and VQGAN-EMA.
Problem
Standard VQGAN codebooks cannot be scaled beyond about 16K entries: entries are randomly initialized and only a few get updated, so utilization collapses (e.g., 11.2% for VQGAN-FC at 16K) and performance saturates or degrades.
Approach
Instead of optimizing codebook entries, initialize a large static codebook from K-means centers of pretrained vision-encoder (CLIP ViT-L/14) patch features and train only a linear projector that maps the whole codebook into the encoder's latent space, so nearly all entries stay active.
Key results
- VQGAN-LC with a 100K codebook reaches 99.9% utilization and 2.62 rFID on ImageNet reconstruction, versus 4.65 for VQGAN-FC and 3.46 for VQGAN-EMA at 100K.
- With LDM on ImageNet, VQGAN-LC (100K codebook) achieves FID 8.36 versus 9.78 for VQGAN-FC and 9.13 for VQGAN-EMA at their best 16K sizes.
- On ImageNet classification with MAE ViT-B, VQGAN-LC reaches 75.7 top-1, exceeding VQGAN-FC and VQGAN-EMA by 1.6 and 1.9 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the standard quantizer with VQGAN-LC using a 100,000-entry codebook | VQGAN · ImageNet-1K reconstruction, 256 tokens | rFID: 3.41 -> 2.62 | Helped |
| replace the standard quantizer with VQGAN-LC using a 100,000-entry codebook | VQGAN · FFHQ reconstruction, 256 tokens | rFID: 4.79 -> 3.81 | Helped |
| use VQGAN-LC tokens instead of VQGAN-FC tokens | GPT-404M on FFHQ · FFHQ generation | FID: 3.23 -> 2.61 | Helped |
| increase VQGAN-LC codebook size from 1,000 to 100,000 (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 4.98 -> 2.62 | Helped |
| increase codebook size from 100,000 to 200,000 (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 2.62 -> 2.66 | Hurt |
| initialize the frozen codebook with random values instead of extracted and clustered image features (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 2.62 -> 108.7 | Hurt |
| use ViT-L rather than ViT-B features for codebook initialization (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 2.70 -> 2.62 | Helped |
| use ViT-L rather than ResNet-50 features for codebook initialization (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 2.71 -> 2.62 | Helped |
| use K-means clustering rather than random feature selection for initialization (ablation) | VQGAN-LC with ViT-L features · ImageNet reconstruction | rFID: 2.95 -> 2.62 | Helped |
| add a trainable projector for the static codebook (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 18.6 -> 2.62 | Helped |
| make initialized codebook entries trainable instead of keeping them static (ablation) | VQGAN-LC with a projector · ImageNet reconstruction | rFID: 2.62 -> 2.65 | No clear effect |
| increase projected codebook dimension from 8 to 16 (ablation) | VQGAN-LC · ImageNet reconstruction | rFID: 2.66 -> 2.36 | Helped |
Surprising findings
- Random initialization of the frozen codebook is catastrophic (rFID 108.7, utilization 5.4%), showing the codebook must be initialized from encoder features.
- Making codebook entries trainable in addition to the projector has minimal impact on performance.
Limitations
- Trained only on ImageNet and FFHQ, restricting generated images to limited categories.
- Scaling to larger datasets such as LAION-5B would be costlier and is not explored.
Experimental setup
Datasets: ImageNet-1K FFHQ
Benchmarks: ImageNet-1K FFHQ
Metrics: rFID LPIPS PSNR SSIM codebook utilization top-1 accuracy FID
Baselines: VQGAN VQGAN-FC VQGAN-EMA RQVAE DQVAE DF-VQGAN DiVAE ViT-VQGAN RQTransformer DQTransformer Stackformer LDM
Builds on: VQGAN VQGAN-FC (ViT-VQGAN factorized codes) VQGAN-EMA (VQVAE-2 EMA update) latent diffusion models DiT SiT
Closest papers by meaning
- LG-VQ: Language-Guided Codebook Learning (NeurIPS 2024 · similarity 0.89)
- Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling (CVPR 2024 · similarity 0.88)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.88)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.87)
- Learning to Quantize for Training Vector-Quantized Networks (ICML 2025 · similarity 0.87)
- Style Quantization for Data-Efficient GAN Training (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.