Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a block-diagonal scaling matrix for the conditional posterior covariance in sparse variational GPs (provably tighter bound than diagonal), and extends the Power-EP sparse GP framework with structured posteriors (M = m I) for tractable PEP updates.
Problem
Sparse GP inference with inducing points relies on q(f|u)=p(f|u); recent diagonal-scaling relaxations tighten bounds, but more expressive structures for the scaling matrix and their use in Power-EP inference remain unexplored.
Approach
Replace the diagonal scaling matrix M with a block-diagonal one whose blocks match likelihood partitions (block size ~M), giving a provably tighter collapsed bound with the same O(M^3) stochastic cost; also show that M = mI makes PEP with power alpha analytically tractable and extends the PEP unifying framework.
Key results
- On kin40k (5,000 points, M=256), the block-diagonal bound BT-SGPR (B=10) reaches objective 0.659 vs 0.883 for SGPR and 0.779 for T-SGPR, with lower noise estimates (0.227 vs 0.299 for SGPR).
- The block-diagonal variational approximation (BT-SVGP) tightens the ELBO over the diagonal T-SVGP and gives comparable or better test RMSE/LL on four UCI datasets across M=256 and M=512.
- For Power-EP, M = mI with alpha=0.5 outperforms M=I on all five UCI datasets in RMSE, with log-likelihood degrading as alpha approaches 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use block-diagonal scaling instead of diagonal scaling | Sparse variational GP on kin40k · kin40k, M=256 | RMSE: 0.223 -> 0.200 | Helped |
| Use block-diagonal scaling instead of diagonal scaling | Sparse variational GP on kin40k · kin40k, M=256 | log likelihood: -0.057 -> -0.032 | Helped |
| Use block-diagonal scaling instead of diagonal scaling | Sparse variational GP on kin40k · kin40k, M=512 | RMSE: 0.184 -> 0.173 | Helped |
| Use block-diagonal scaling instead of diagonal scaling | Sparse variational GP on kin40k · kin40k, M=512 | log likelihood: 0.115 -> 0.133 | Helped |
| Use structured posterior in PEP | PEP α=0.5 on kin40k · kin40k, M=256 | RMSE: 0.235 -> 0.200 | Helped |
| Use structured posterior in PEP | PEP α=0.5 on kin40k · kin40k, M=512 | log likelihood: 0.140 -> 0.190 | Helped |
| Increase the block size in block-diagonal variational scaling (ablation) | BT-SGPR on kin40k · kin40k, M=256 | RMSE: 0.217 -> 0.200 | Helped |
| Increase the block size in block-diagonal variational scaling (ablation) | BT-SGPR on kin40k · kin40k, M=512 | RMSE: 0.181 -> 0.173 | Helped |
| Set the PEP scaling to m ≠ 1 rather than m=1 (ablation) | Power-EP on five UCI regression datasets · Five UCI regression datasets | RMSE | Helped |
| Set the PEP scaling to m ≠ 1 rather than m=1 (ablation) | Power-EP on five UCI regression datasets · Five UCI regression datasets, α closer to 1 | log likelihood | Hurt |
Surprising findings
- Estimated observation noise is smaller and kernel variance larger when using the tighter structured bounds, partly mitigating the known large-noise pathology of variational sparse GPs.
Limitations
- Regression only; non-Gaussian likelihood extension deferred to future work.
- PEP approximate marginal likelihood is not guaranteed to be a lower bound and can behave pathologically at alpha=1.
- Block-diagonal and diagonal structures make the predictive covariance expensive, so test-time predictions revert to the standard approximation.
Experimental setup
Datasets: Snelson kin40k (5,000-point subset) UCI regression datasets
Benchmarks: Snelson kin40k UCI regression datasets
Metrics: ELBO marginal likelihood RMSE log likelihood noise standard deviation lengthscales
Baselines: SVGP T-SVGP SGPR T-SGPR PEP T-PEP FITC
Builds on: sparse variational GP (Titsias) tighter sparse GP bound (diagonal scaling, T-SVGP) Power-EP sparse GP framework spherical bound FITC
Closest papers by meaning
- New Bounds for Sparse Variational Gaussian Processes (ICML 2025 · similarity 0.92)
- Input Dependent Sparse Gaussian Processes (ICML 2022 · similarity 0.91)
- Sparse within Sparse Gaussian Processes using Neighbor Information (ICML 2021 · similarity 0.91)
- Sparse Gaussian Processes with Spherical Harmonic Features (ICML 2020 · similarity 0.90)
- Variational nearest neighbor Gaussian process (ICML 2022 · similarity 0.90)
- Robust Gaussian Processes via Relevance Pursuit (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.