Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Robust and Conjugate Gaussian Process Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes robust conjugate GP (RCGP) regression by applying generalised Bayesian inference with a weighted score-matching loss, giving closed-form posteriors that down-weight outliers at no extra cost over standard GPs.
Problem
Standard Gaussian process regression with i.i.d. Gaussian noise is highly non-robust to outliers, while existing robust GP methods (Student-t, mixture, outlier removal) break closed-form conjugacy and are computationally expensive.
Approach
Replace the Gaussian log-likelihood with a weighted generalised score-matching loss (with weight w(x,y) depending on the observation y, e.g. an inverse multi-quadratic kernel on residuals), which yields a quadratic loss in f and hence a Gaussian posterior and predictive in closed form; the weight bounds the posterior influence of outliers.
Key results
- On datasets with focused outliers, RCGP achieved the lowest test MAE on all four datasets (e.g. Yacht 0.10 vs GP 0.26), and under asymmetric outliers it was best on Boston, Yacht and Synthetic-uniform settings among GP variants.
- RCGP training clock times were comparable to GP (e.g. Synthetic 1.2s vs 1.5s) while t-GP and m-GP took up to ~34s on Energy.
- RCSVGP gave far more accurate predictions than SVGP at similar runtime on the synthetic dataset with 10% uniform outliers; RCGP matched or outperformed GP in Bayesian optimisation with contamination.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RCGP instead of GP | GP regression · No-outlier benchmarks | mean absolute error: Synthetic: 0.09 (0.00) -> 0.09 (0.00); Boston: 0.19 (0.01) -> 0.19 (0.01) | No clear effect |
| Use RCGP instead of GP | GP regression · Focused outlier benchmarks | mean absolute error: Synthetic: 0.19 (0.00) -> 0.15 (0.00); Boston: 0.23 (0.06) -> 0.22 (0.01); Energy: 0.03 (0.04) -> 0.02 (0.00); Yacht: 0. | Helped |
| Use RCGP instead of GP | GP regression · Asymmetric outlier benchmarks | mean absolute error: Synthetic: 1.14 (0.00) -> 0.63 (0.00); Boston: 0.63 (0.02) -> 0.49 (0.00); Yacht: 0.54 (0.06) -> 0.35 (0.02) | Helped |
| Use RCGP instead of GP | GP regression · Uniform outlier benchmarks | mean absolute error: Synthetic: 0.34 (0.00) -> 0.21 (0.00); Boston: 0.53 (0.13) -> 0.51 (0.12); Energy: 0.25 (0.03) -> 0.24 (0.03); Yacht: 0. | Mixed |
| Use t-GP instead of RCGP | GP regression · Uniform outlier benchmarks | mean absolute error: Boston: 0.51 (0.12) vs 0.30 (0.01); Energy: 0.24 (0.03) vs 0.23 (0.05); Yacht: 0.29 (0.07) vs 0.16 (0.00) | Mixed |
| Remove the shrinkage term (ablation) | RCGP · Synthetic dataset with 10% uniformly generated outliers | predictive posterior mean | Hurt |
| Remove the noise term (ablation) | RCGP · Synthetic dataset with 10% uniformly generated outliers | predictive posterior mean | Hurt |
| Set ε to 0.99 rather than an intermediate value (ablation) | RCGP weighting threshold · Synthetic data with outliers | robustness | Hurt |
| Set ε to 0 rather than an intermediate value (ablation) | RCGP weighting threshold · Synthetic data with outliers | robustness | Hurt |
| Use RCSVGP instead of SVGP | Sparse variational GP · Synthetic dataset with 10% uniform outliers | root mean square error | Helped |
| Use RCGP instead of GP | Bayesian optimization with UCB and 20% asymmetric outliers · Six-hump camel, Branin, McCormick and Rosenbrock | cumulative regret | Helped |
Surprising findings
- Under no outliers RCGP matched or beat the standard GP on Energy (0.02 vs 0.03), suggesting no clear cost to using it as a default.
- Choosing the outlier threshold c by maximising LOO-CV predictive performance was suboptimal, since it tends to increase c and fit outliers.
Limitations
- Performance depends on a well-specified prior mean, since the weight depends on residuals from m(x).
- Cannot place a posterior over kernel hyperparameters because the generalised posterior is not a proper likelihood-based model; hyperparameters are point-estimated via LOO-CV.
- Not directly applicable to noise-free interpolation (sigma=0) without a nugget.
Experimental setup
Datasets: synthetic 1D dataset (n=300) Boston housing (n=506) UCI Energy (n=768) UCI Yacht (n=308) Dow Jones Industrial Average 17/04/2013 (Twitter flash crash) six-hump camel, Branin, McCormick, Rosenbrock functions
Benchmarks: Synthetic Boston Energy Yacht Twitter flash crash DJIA Bayesian optimisation test functions
Metrics: mean absolute error clock time root mean square error cumulative regret posterior influence function
Baselines: standard GP Student-t GP (t-GP) mixture-based outlier GP (m-GP) SVGP t-GP for Bayesian optimisation
Builds on: generalised Bayesian inference weighted score matching sparse variational Gaussian processes leave-one-out cross-validation
Closest papers by meaning
- Robust and Conjugate Spatio-Temporal Gaussian Processes (ICML 2025 · similarity 0.90)
- Robust and Computation-Aware Gaussian Processes (NeurIPS 2025 · similarity 0.90)
- Robust Gaussian Processes via Relevance Pursuit (NeurIPS 2024 · similarity 0.90)
- New Bounds for Sparse Variational Gaussian Processes (ICML 2025 · similarity 0.87)
- Conditioning Gaussian Processes on Almost Anything (NeurIPS 2026 · similarity 0.87)
- Turbocharging Gaussian Process Inference with Approximate Sketch-and-Project (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.