Research map / Architectures and efficiency / Quantization
Finite Scalar Quantization: VQ-VAE Made Simple
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes finite scalar quantization (FSQ), which bounds and rounds each of a few latent dimensions to yield an implicit codebook, as a drop-in replacement for VQ in VQ-VAEs, and shows competitive results with MaskGIT and UViM without auxiliary losses.
Problem
Vector quantization in VQ-VAEs suffers from underutilized codebooks and requires complex auxiliary machinery (commitment losses, codebook reseeding, splitting, entropy penalties) to learn expressive discrete representations.
Approach
Project the encoder output to a few dimensions, bound each channel to L_i fixed values via a bounding function, and round to integers with straight-through gradients; the product of per-channel level sets forms an implicit codebook with near-full utilization and no auxiliary losses.
Key results
- On ImageNet 256 MaskGIT, FSQ and VQ achieve very comparable FID, precision and recall, with FSQ using L=[8,5,5,5] (codebook 1024).
- FSQ attains near-100% codebook usage at codebook size 2^14 while VQ usage drops below 50% above 2^11.
- UViM with FSQ obtains depth RMSE 0.473 vs 0.468 for VQ, panoptic PQ 43.2 vs 43.4, and colorization FID-5k 17.55 vs 16.90; without context FSQ panoptic PQ is 40.2 vs 39.0 for VQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace VQ with FSQ | UViM · NYU Depth v2 | RMSE: 0.468±0.012 -> 0.473±0.012 | Hurt |
| Replace VQ with FSQ | UViM · COCO Panoptic | PQ: 43.4±0.0008 -> 43.2±0.0014 | Hurt |
| Replace VQ with FSQ | UViM · ImageNet Colorization | FID-5k: 16.90±0.056 -> 17.55±0.057 | Hurt |
| Remove VAE context input (ablation) | UViM (VQ) · COCO Panoptic | PQ: 43.4±0.0008 -> 39.0±0.0023 | Hurt |
| Remove VAE context input (ablation) | UViM (FSQ) · COCO Panoptic | PQ: 43.2±0.0014 -> 40.2±0.0019 | Hurt |
| Disable codebook splitting (ablation) | UViM (VQ) · NYU Depth v2 | RMSE: 0.468±0.012 -> 0.490±0.0037 | Hurt |
| Increase FSQ codebook size | MaskGIT · 128×128 ImageNet | Reconstruction FID | Helped |
| Increase VQ codebook size beyond 2^11 codes | MaskGIT · 128×128 ImageNet | Reconstruction FID | Hurt |
| Use FSQ instead of VQ at large codebook sizes | MaskGIT · 128×128 ImageNet | Sampling FID | Helped |
| Use FSQ instead of VQ for codebooks larger than 2^11 | MaskGIT · 128×128 ImageNet | Codebook usage | Helped |
| Use FSQ instead of VQ | MaskGIT · 256×256 ImageNet | Sampling FID | No clear effect |
| Use FSQ levels below 5 in any channel (ablation) | FSQ MaskGIT · 128×128 ImageNet | Sampling FID | Hurt |
Surprising findings
- VQ without codebook splitting in UViM drops codebook usage to 0.78% and degrades depth RMSE, while FSQ reaches 99% usage without any auxiliary algorithm.
- Without VAE context in panoptic segmentation, FSQ outperforms VQ (40.2 vs 39.0 PQ).
Limitations
- FSQ is marginally worse than VQ on UViM tasks with context (e.g., depth, colorization, panoptic with context).
- Low-codebook regimes show VQ marginally outperforming FSQ in reconstruction.
- Evaluated only on image tasks (MaskGIT on ImageNet and UViM on three dense tasks).
Experimental setup
Datasets: ImageNet NYU Depth v2 COCO Panoptic
Benchmarks: ImageNet NYU Depth v2 COCO Panoptic
Metrics: FID precision recall reconstruction FID codebook usage compression cost RMSE PQ FID-5k
Baselines: MaskGIT (VQ) ADM UViM (VQ) DenseDepth DETR-R101 ColTran
Builds on: VQ-VAE VQ-GAN MaskGIT UViM straight-through estimator
Closest papers by meaning
- Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization (ICLR 2026 · similarity 0.91)
- VAEVQ: Enhancing Discrete Visual Tokenization Through Variational Modeling (AAAI 2026 · similarity 0.90)
- Restructuring Vector Quantization with the Rotation Trick (ICLR 2025 · similarity 0.89)
- Channel-wise Vector Quantization (NeurIPS 2026 · similarity 0.88)
- Learning to Quantize for Training Vector-Quantized Networks (ICML 2025 · similarity 0.88)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.