Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

New Bounds for Sparse Variational Gaussian Processes

ICML 2025 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Sparse variational Gaussian processes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Relaxes the conditional GP factor in sparse variational GP posteriors with per-training-point variational variances, yielding a tighter collapsed bound (SGPR-new/SVGP-new) at unchanged O(NM^2) cost.

Problem

Sparse variational GP training relies on approximating p(f|u,y) with the conditional GP prior p(f|u), which causes underfitting, biased noise variance estimates and looser ELBOs.

Approach

Replace p(f|u) in the variational distribution by a Gaussian q(f|u) with the same mean but covariance (K_ff-Q_ff)^{1/2} V (K_ff-Q_ff)^{1/2} with N extra variational parameters v_i; optimizing them analytically gives a tighter collapsed bound with a log(1+(k_ii-q_ii)/sigma^2) regularizer that remains O(NM^2) and supports minibatch training.

Key results

• On Pol, Bike, Elevators with M=1024, SGPR-new reaches test log-likelihood 0.920 (Pol) and 2.326 (Bike) versus 0.821 and 2.176 for SGPR.
• On large-scale UCI datasets with M=2048, SVGP-new achieves best test log-likelihood on 6 of 8 datasets, e.g. HouseElectric 1.421 vs 1.419 for SVGP.
• On NYBikes Poisson regression with M=8, SVGP-new gets -9.713 test log-likelihood versus -36.397 for SVGP.

Measured effects reported in the paper

Surprising findings

• On the Elevators dataset, all sparse methods essentially match exact GP training, since k_ii - q_ii is near zero and the v_i concentrate near one.
• For the NYBikes Poisson example the learned scalar v becomes very small (below 0.01), indicating a strongly tightened variational covariance.

Limitations

• Non-Gaussian extension uses a spherical V = vI, so only a single extra parameter per GP function is learned, and the non-conjugate bound is evaluated on single-function models only.
• The predictive distribution uses the standard SVGP form rather than the tighter high-cost q, so predictions are not fully consistent with the new variational distribution.
• Experiments are limited to regression and a single Poisson model; more complex GP models are left for future work.

Experimental setup

Datasets: Snelson 1-D Pol Bike Elevators Kin40k Protein KeggDirected KEGGU 3dRoad Song Buzz HouseElectric

Benchmarks: Pol Bike Elevators Kin40k Protein KeggDirected KEGGU 3dRoad Song Buzz HouseElectric NYBikes

Metrics: test log-likelihood RMSE ELBO noise variance estimate

Baselines: SGPR (standard collapsed bound) SGPR-artemev SVGP (uncollapsed) Exact GP ODVGP SOLVE-GP Full GP

Builds on: sparse variational GP (Titsias 2009) DTC / projected process approximation GPflow Artemev et al. 2019 bound

Closest papers by meaning

• Input Dependent Sparse Gaussian Processes (ICML 2022 · similarity 0.92)
• Variational Gaussian Processes with Decoupled Conditionals (NeurIPS 2023 · similarity 0.92)
• Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited (NeurIPS 2025 · similarity 0.92)
• A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression (AAAI 2017 · similarity 0.92)
• Sparse Variational Student-t Processes (AAAI 2024 · similarity 0.90)
• Robust Gaussian Processes via Relevance Pursuit (NeurIPS 2024 · similarity 0.89)