Research map / Probabilistic and scientific modeling / Variational and generative sampling

Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables

NeurIPS 2025 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Structured latent VAEs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives decoder-independent, information-theoretic generalization bounds for VQ-VAE reconstruction loss using a supersample setting with a data-dependent prior over latent variables, plus a permutation-symmetric variant that converges asymptotically and a W2 bound on generated data.

Problem

Theoretical understanding of how latent variables and encoder complexity affect generalization of VQ-VAEs in unsupervised learning, where existing supersample IT bounds depend on the decoder and do not vanish with sample size.

Approach

Constructs a data-dependent prior over discrete latent indices via supersample shuffling of LVs, and extends to a permutation-symmetric setting over all 2n points, yielding bounds that depend only on encoder and latent-variable complexity (not the decoder) and that converge to zero with n.

Key results

• Permutation-symmetric bound removes the non-vanishing empirical KL term and yields gen(n,D)=O(sqrt(d_phi d_z log n / n)) via metric entropy, decoder-independent.
• On MNIST (K=128, d_z=64), the second KL term correlates positively with the generalization gap (r≈0.46–0.60), while the first empirical KL term/n does not decrease with n and correlates negatively.
• Adding a single ResBlock (~74,000 parameters) has a negligible effect on the generalization gap despite lower training loss for n≥1000.

Measured effects reported in the paper

Surprising findings

• The empirical KL term over latent variables does not vanish as n grows even under the optimal prior, and it is negatively correlated with the generalization gap.
• Increasing decoder complexity lowers training loss but leaves the generalization gap largely unchanged.

Limitations

• The permutation-symmetric bound is hard to compute numerically, so it is impractical as an evaluation metric.
• Analysis is limited to discrete latent spaces; continuous-latent extensions are non-trivial.
• Bounds assume the decoder can adequately fit the training data.

Experimental setup

Datasets: MNIST

Benchmarks: MNIST

Metrics: generalization gap KL divergence terms training loss test loss

Baselines: basic IT-bound (supersample) PAC-Bayes VAE bound fCMI-based bound

Builds on: supersample information-theoretic generalization bounds VQ-VAE Donsker-Varadhan inequality PAC-Bayes bounds for VAEs VampPrior

Closest papers by meaning

• Information-Theoretic Generalization Bounds for VAEs: A Role of Encoder and Latent Variable (ICML 2026 · similarity 0.96)
• Trading Information between Latents in Hierarchical Variational Autoencoders (ICLR 2023 · similarity 0.90)
• Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis (ICLR 2025 · similarity 0.89)
• Consistency Regularization for Variational Auto-Encoders (NeurIPS 2021 · similarity 0.89)
• Marginalization is not Marginal: No Bad VAE Local Minima when Learning Optimal Sparse Representations (ICML 2023 · similarity 0.89)
• Restructuring Vector Quantization with the Rotation Trick (ICLR 2025 · similarity 0.88)