Research map / Optimization / Optimizers
Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Consolidates three amortisation techniques for iterative Gaussian process hyperparameter optimisation (pathwise gradient estimator, warm-started linear solvers, and early stopping under compute budgets) and shows speed-ups up to 72x across CG, AP and SGD on UCI regression.
Problem
Hyperparameter optimisation of Gaussian processes on large datasets is expensive because the marginal likelihood gradient requires many iterative linear system solves, which converge slowly when the kernel matrix is poorly conditioned.
Approach
Introduces a pathwise gradient estimator whose probe vectors absorb H^{-1} so that solver initial distance is constant (n) rather than tr(H^{-1}); warm-starts solvers from the previous marginal likelihood step; and combines these with early stopping under a fixed compute budget so solver progress accumulates across outer steps. The pathwise solutions also yield posterior samples without extra solv
Key results
- Pathwise estimator with warm starting gives average speed-up of 72.1x for alternating projections relative to the standard estimator without warm starting when solving to tolerance on five UCI datasets.
- Warm starting alone gives average speed-ups of 2.1x (CG), 18.9x (AP) and 5.1x (SGD); test log-likelihoods are unchanged.
- Under limited budgets (10 solver epochs), pathwise estimator with warm starting reaches lower residual norms than the standard estimator with 50 epochs; average residual norm reduced up to 7x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the pathwise estimator instead of the standard estimator (ablation) | CG · UCI datasets (average) | speed-up: 2.1 × | Helped |
| add warm starting to the standard estimator (ablation) | CG · UCI datasets (average) | speed-up: 1.2 × | Helped |
| use the pathwise estimator and warm starting instead of the standard estimator without warm starting (ablation) | CG · UCI datasets (average) | speed-up: 1.9 × | Helped |
| use the pathwise estimator instead of the standard estimator (ablation) | AP · UCI datasets (average) | speed-up: > 18.9 × | Helped |
| add warm starting to the standard estimator (ablation) | AP · UCI datasets (average) | speed-up: > 5.4 × | Helped |
| use the pathwise estimator and warm starting instead of the standard estimator without warm starting (ablation) | AP · UCI datasets (average) | speed-up: > 72.1 × | Helped |
| use the pathwise estimator instead of the standard estimator (ablation) | SGD · UCI datasets (average) | speed-up: 5.1 × | Helped |
| add warm starting to the standard estimator (ablation) | SGD · UCI datasets (average) | speed-up: 2.1 × | Helped |
| use the pathwise estimator and warm starting instead of the standard estimator without warm starting (ablation) | SGD · UCI datasets (average) | speed-up: 7.2 × | Helped |
| combine early stopping with warm starting | iterative GP solvers · large UCI datasets | relative residual norm: up to 7× | Helped |
| stop after a limited compute budget instead of solving to tolerance | iterative GP solvers · large UCI datasets | relative residual norm | Hurt |
| use warm starting with early stopping | CG · Buzz and Houseelectric | test log-likelihood | Hurt |
Surprising findings
- Good predictive performance is often obtained even when the relative residual norm is well above the tolerance 0.01, so residual norm is a weak proxy for predictive quality.
- CG is brittle under early stopping, performing much worse than AP and SGD on buzz and houseelectric despite lower residual norms.
Limitations
- Warm starting introduces a small bias because probe vectors are fixed across outer steps; theory only gives convergence in probability as the number of probes grows.
- Relative residual norm is not reliably predictive of test performance under early stopping.
- Pathwise estimator relies on random features to approximate prior samples, causing deviations for some hyperparameters (e.g. elevators).
Experimental setup
Datasets: UCI pol UCI elevators UCI bike UCI protein UCI keggdirected UCI buzz UCI houseelectric
Benchmarks: UCI pol UCI elevators UCI bike UCI protein UCI keggdirected UCI buzz UCI houseelectric
Metrics: test log-likelihood total training time relative residual norm solver iterations to tolerance speed-up
Baselines: standard estimator with zero initialisation Cholesky-based exact marginal likelihood CG AP SGD
Builds on: pathwise conditioning (Wilson et al.) Hutchinson trace estimator Lin et al. SGD for GPs Maddox et al. iterative GP methods
Closest papers by meaning
- Turbocharging Gaussian Process Inference with Approximate Sketch-and-Project (NeurIPS 2025 · similarity 0.87)
- Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference (NeurIPS 2024 · similarity 0.86)
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.86)
- Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation (ICML 2024 · similarity 0.85)
- Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.85)
- Learning to Relax: Setting Solver Parameters Across a Sequence of Linear System Instances (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.