Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Robust Gaussian Processes via Relevance Pursuit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Robust Gaussian Processes via Relevance Pursuit (RRP), which learns data-point-specific noise variances via greedy sequential selection maximizing the GP marginal likelihood, and proves strong concavity under a reparameterization giving approximation guarantees for sparse outlier corruption.
Problem
Standard GP regression assumes homoskedastic Gaussian noise and degrades under sparse, non-Gaussian label corruptions that may lie within the function's range.
Approach
Add learnable data-point-specific noise variances to the GP, select a sparse outlier set greedily using closed-form leave-one-out-based optimal variances, and reparameterize the variances so the negative log marginal likelihood is strongly convex under diagonal dominance, yielding weak-submodularity-based OMP approximation guarantees.
Key results
- On Friedman10 and Hartmann6 regression with corruptions, RRP outperforms baselines in predictive log-likelihood; Student-t GP only overtakes RRP when corruption probability exceeds about 40%.
- In the Twitter flash crash example, RRP is virtually unaffected by outliers while RCGP is affected; with the previous day included, RCGP becomes less robust than a standard GP.
- In Bayesian optimization on real-world problems, RRP is competitive with the oracle on the 3D SVM and performs well on the 20D rover, while a power transform performs best on the 5D CNN problem.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RRP instead of the standard GP | Gaussian process regression on Twitter Flash Crash data · DJIA April 23, 2013 | robustness to outliers | Helped |
| Use RRP instead of RCGP | Gaussian process regression on Twitter Flash Crash data · DJIA April 23, 2013, with previous trading day in training data | robustness to outliers | Helped |
| Use RRP instead of the Student-t likelihood | Robust Bayesian optimization · Hartmann6, U[-3, 3] corruptions, 10% corruption probability | best in-sample objective | Helped |
| Use RRP instead of other compared methods | Robust Bayesian optimization · Hartmann6, constant outliers of value 100, 10% corruption probability | best in-sample objective | Helped |
| Use the power transform instead of RRP | Robust Bayesian optimization · 5D CNN problem | best in-sample objective | Helped |
| Use RRP | Robust Bayesian optimization · 3D SVM problem | best in-sample objective | No clear effect |
| Use RRP | Robust Bayesian optimization · 20D Rover problem | best in-sample objective | Helped |
| Use the Student-t likelihood instead of RRP | Gaussian process regression · Synthetic Student-t corruptions with corruption probability above 40% | predictive log likelihood | Helped |
| Use the forward relevance-pursuit variant instead of the backward variant (ablation) | RRP · Regression with few outliers | performance | Helped |
| Use the backward relevance-pursuit variant instead of the forward variant (ablation) | RRP · Regression with a high outlier frequency | performance | Helped |
Surprising findings
- The reparameterized negative log marginal likelihood is strongly convex in the reparameterized variances under diagonal dominance, despite GP marginal likelihoods generally being non-convex.
- RCGP's a-priori weighting can make it less robust than a standard GP when outliers lie near the target median and previous-day data is included.
Limitations
- Does not explicitly model location-dependent (heteroskedastic) noise, so may be outperformed when noise is input-dependent.
- Theory assumes other GP hyperparameters are fixed; joint optimization in practice is non-convex.
- Backward variant lacks theoretical guarantees.
Experimental setup
Datasets: Friedman10 Hartmann6 Dow Jones Industrial Average April 2013 data UCI machine learning repository datasets
Benchmarks: Friedman10 Hartmann6 Twitter flash crash DJIA UCI regression datasets Bayesian optimization Hartmann6 3D SVM 5D CNN 20D rover trajectory planning
Metrics: predictive test-set log likelihood best in-sample true objective value per BO iteration
Baselines: Standard GP Adaptive winsorization Power transform Student-t likelihood GP Trimmed MLL RCGP Sobol Oracle GP
Builds on: Gaussian processes orthogonal matching pursuit sparse Bayesian learning restricted strong convexity
Closest papers by meaning
- Robust and Computation-Aware Gaussian Processes (NeurIPS 2025 · similarity 0.91)
- Robust and Conjugate Gaussian Process Regression (ICML 2024 · similarity 0.90)
- New Bounds for Sparse Variational Gaussian Processes (ICML 2025 · similarity 0.89)
- Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference (NeurIPS 2024 · similarity 0.88)
- Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited (NeurIPS 2025 · similarity 0.88)
- Sparse Variational Student-t Processes (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.