Research map / Probabilistic and scientific modeling / Variational and generative sampling
Preventing Model Collapse in Gaussian Process Latent Variable Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes advisedRFLVM, a GPLVM that learns projection variance and uses a spectral mixture kernel with a differentiable random Fourier feature approximation trained by variational inference, to prevent latent model collapse.
Problem
GPLVMs often collapse into vague latent representations (distortion or homogeneity) due to improperly chosen projection noise and inadequate kernel flexibility.
Approach
Analyze projection variance in a linear GPLVM (DPPCA) to show how improper sigma^2 causes zero-columns, then combine a spectral mixture kernel with a differentiable RFF approximation so that mixture weights, projection variance, and latents are all learned by auto-differentiated ELBO maximization, avoiding inducing points.
Key results
- advisedRFLVM achieves the highest 1-NN accuracy on all seven datasets in the table, e.g., 0.795 on MNIST vs 0.602 for RFLVM and 0.643 for VAE.
- In missing-data imputation on MNIST with 0% missing, advisedRFLVM reaches test MSE 0.025 and kNN accuracy 0.806, versus 0.035 and 0.715 for VAE.
- On S-shaped manifold learning, advisedRFLVM and RFLVM consistently outperform BGPLVM and GPLVM-SVI in R^2 score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn projection variance σ² instead of fixing it (ablation) | advisedRFLVM · MNIST | KNN accuracy | Helped |
| Use a flexible SM kernel rather than a preliminary RBF kernel | GPLVM · RBF+periodic synthetic S-shaped manifold | R² score | Helped |
| Replace fixed projection variance with learned σ² | linear GPLVM · linear GPLVM | stable maximum | Helped |
| Use advisedRFLVM instead of VAE | latent representation learning · MNIST | KNN accuracy: 0.643 ± 0.021 -> 0.795 ± 0.015 | Helped |
| Use advisedRFLVM instead of RFLVM | latent representation learning · Yale | KNN accuracy: 0.653 ± 0.067 -> 0.765 ± 0.010 | Helped |
| Use advisedRFLVM instead of VAE | missing-data reconstruction · MNIST, 30% missing | test MSE: 0.045 -> 0.039 | Helped |
| Use advisedRFLVM instead of VAE | missing-data reconstruction · Brendan, 30% missing | test MSE: 0.043 -> 0.045 | Hurt |
| Use advisedRFLVM instead of VAE | missing-data reconstruction · Brendan, 60% missing | test MSE: 0.150 -> 0.152 | Hurt |
| Use advisedRFLVM instead of VAE | missing-data reconstruction · MNIST, 60% missing | test MSE: 0.068 -> 0.068 | No clear effect |
Surprising findings
- With a fixed projection variance, the latent variable collapses to zero columns as the initial sigma^2 increases, whereas learning sigma^2 avoids collapse regardless of initialization.
- GPLVM variants with inducing points (BGPLVM, GPLVM-SVI) perform comparably to PCA on latent-space kNN accuracy, despite their flexibility.
Limitations
- Out-of-distribution inference is not supported because there is no encoder mapping observations to latents.
- Practical training time may be too long for massive datasets despite linear scaling in N.
- Theoretical collapse analysis is exact only for the linear GPLVM (DPPCA); generalization to nonlinear GPLVMs is not proven.
Experimental setup
Datasets: MNIST CIFAR-10 Brendan Bridges Montreal Newsgroups Yale synthetic S-shaped manifold
Benchmarks: MNIST CIFAR-10 Brendan Bridges Montreal Newsgroups Yale
Metrics: knn accuracy R^2 score test mse number of zero columns
Baselines: PCA LDA Isomap HPF BGPLVM GPLVM-SVI VAE NBVAE DCA CVQ-VAE RFLVM
Builds on: GPLVM random Fourier features spectral mixture kernel dual probabilistic PCA RFLVM
Closest papers by meaning
- Multi-View Oriented GPLVM: Expressiveness and Efficiency (NeurIPS 2025 · similarity 0.91)
- Decomposing feature-level variation with Covariate Gaussian Process Latent Variable Models (ICML 2019 · similarity 0.89)
- Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes (NeurIPS 2017 · similarity 0.87)
- Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling (ICML 2025 · similarity 0.87)
- Latent variable model for high-dimensional point process with structured missingness (ICML 2024 · similarity 0.86)
- Amortized Structured Stochastic Variational Inference for Gaussian Process Latent Variable Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.