Research map / Probabilistic and scientific modeling / Variational and generative sampling
Dimensional Collapse in VQVAEs: Evidence and Remedies
Jiayou Zhang, Yifan Shen, Guangyi Chen, Le Song, Eric P. Xing
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies "dimensional collapse" in VQVAEs, where high-dimensional code embeddings occupy only a small subspace (about 4 to 10 dimensions), and proposes remedies.
Problem
VQVAE codebook embeddings, although high-dimensional, appear to concentrate their representations in a much smaller subspace, limiting codebook utilization and representation capacity.
Approach
Names and characterizes a consistent phenomenon, dimensional collapse, in which VQVAEs compress high-dimensional embeddings into roughly 4 to 10 effective dimensions, and proposes remedies to counter it.
Key results
- VQVAEs consistently compress high-dimensional embeddings into a subspace of typically only 4 to 10 dimensions (dimensional collapse).
Surprising findings
- Despite high-dimensional embeddings, VQVAEs collapse representations into a much smaller subspace, typically 4 to 10 dimensions.
Experimental setup
Builds on: VQVAE
Closest papers by meaning
- Unveiling And Addressing Dimensional Collapse In Vector Quantization Models Via Codebook Regularization (ICML 2026 · similarity 0.91)
- Restructuring Vector Quantization with the Rotation Trick (ICLR 2025 · similarity 0.88)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.88)
- Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders (ICLR 2024 · similarity 0.87)
- Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables (NeurIPS 2025 · similarity 0.87)
- Be a Goldfish: Forgetting Bad Conditioning in Sparse Linear Regression via Variational Autoencoders (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
