Research map / Learning theory and algorithms / Prediction and decision losses
Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends computation-aware GPs to linear-time model selection and inference by learning sparse block actions end-to-end with an ELBO-style training loss, outperforming SVGP, SGPR and CGGP on UCI regression up to 1.8M points.
Problem
Gaussian process model selection and inference scale poorly with dataset size; existing approximations (SVGP, SGPR, CGGP) incur approximation error that causes overconfidence and biased hyperparameters such as oversmoothing and overestimated noise.
Approach
Optimize sparse block-structured actions jointly with kernel hyperparameters using a computation-aware ELBO training loss, so the number of nonzero action entries equals n and inference/model selection cost becomes linear in n with GPU parallelism.
Key results
- On Protein, CaGP-Opt achieves test NLL 0.829 versus 0.851 for SVGP (Adam), while CGGP and CaGP-CG are best on RMSE and NLL respectively.
- On Power (1.84M points), CaGP-Opt matches SVGP's NLL (-2.103 vs -2.104) in 4h32min on one A100 GPU.
- CaGP-Opt learns lower observation noise than SVGP, which tends to overestimate noise and lengthscale; CaGP-Opt has better 95% coverage than SVGP on Power.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CaGP-Opt instead of CaGP-CG | Computation-aware GP · Parkinsons | Test NLL: -2.936 -> -3.449 | Helped |
| Use CaGP-Opt instead of SGPR | GP regression · Bike | Test NLL: -3.017 -> -2.438 | Hurt |
| Use CaGP-Opt instead of CaGP-CG | Computation-aware GP · Protein | Test NLL: 0.820 -> 0.829 | Hurt |
| Use CaGP-Opt instead of CGGP | GP regression · Protein | Test RMSE: 0.517 -> 0.545 | Hurt |
| Use CaGP-Opt instead of SGPR | GP regression · KEGGu | Test NLL: -0.712 -> -0.694 | Hurt |
| Use CaGP-Opt instead of SVGP | GP regression · Road | Test NLL: 0.149 -> -0.291 | Helped |
| Use CaGP-Opt instead of SVGP | GP regression · Power | Test NLL: -2.104 -> -2.103 | No clear effect |
| Use the ELBO training loss instead of projected-data log-marginal likelihood | CaGP model selection · UCI regression datasets | Test NLL | Helped |
Surprising findings
- CaGP-Opt with i=512 actions outperforms SVGP with m=1024 inducing points, despite using half the rank.
- Naive projected-data log marginal likelihood generalizes poorly because it only trains on the projected data, motivating the full-data ELBO loss.
Limitations
- Not amenable to stochastic minibatching, so it cannot achieve SVGP's constant time and memory per step; requires larger GPUs on datasets over one million points.
- Requires choosing the number of actions i, a tunable rank parameter.
- Limited to GP regression with conjugate Gaussian observation noise.
Experimental setup
Datasets: Parkinsons Bike Protein KEGGu 3DRoad (Road) Power
Benchmarks: Parkinsons Bike Protein KEGGu Road Power
Metrics: test NLL test RMSE wall-clock runtime coverage error
Baselines: SGPR SVGP CGGP CaGP-CG CholeskyGP
Builds on: computation-aware Gaussian processes (IterGP) stochastic variational gaussian processes sparse gaussian process regression conjugate gradient gaussian processes gpytorch
Closest papers by meaning
- Exact Gaussian Processes on a Million Data Points (NeurIPS 2019 · similarity 0.90)
- Robust and Computation-Aware Gaussian Processes (NeurIPS 2025 · similarity 0.89)
- Turbocharging Gaussian Process Inference with Approximate Sketch-and-Project (NeurIPS 2025 · similarity 0.89)
- Robust Gaussian Processes via Relevance Pursuit (NeurIPS 2024 · similarity 0.88)
- Conditioning Gaussian Processes on Almost Anything (NeurIPS 2026 · similarity 0.88)
- Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.