Research map / Architectures and efficiency / RNNs and normalization
Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIPS, a stopping criterion based on an online ELBO/upper-bound gap that automatically sets the number of inducing points for streaming sparse GP regression, with a single pre-set hyperparameter.
Problem
Choosing model size (number of inducing points, analogous to neurons) for sparse Gaussian processes in continual learning, where the final dataset size is unknown and cross-validation cannot be used because past data cannot be stored.
Approach
Derives an online upper bound on the log marginal likelihood and adds greedy-variance inducing points until the gap between the online ELBO and the upper bound falls below a threshold relative to a noise-model baseline, giving a KL guarantee to the best achievable approximation and a single hyperparameter delta that transfers across datasets.
Key results
- On UCI at the 10% RMSE operating point, VIPS used fewer inducing points on most datasets (e.g. Kin8nm 1904 vs CV 6316 and OIPS 6458; Bike 650 vs OIPS 1964).
- VIPS reached near-exact GP performance on the synthetic task with an adaptive count, avoiding the waste of a fixed M=30 early on.
- Naval proved hard: OIPS failed the accuracy threshold for all hyperparameters, and VIPS missed the 5% NLPD threshold by 0.48%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIPS instead of CV for adaptive inducing-point selection | Online GP · UCI datasets, 10% RMSE threshold | inducing points | Helped |
| Use VIPS instead of OIPS for adaptive inducing-point selection | Online GP · UCI datasets, 10% RMSE threshold | inducing points | Mixed |
| Use VIPS instead of CV for adaptive inducing-point selection | Online GP · UCI datasets, 10% NLPD threshold | inducing points | Mixed |
| Use VIPS instead of OIPS for adaptive inducing-point selection | Online GP · UCI datasets, 10% NLPD threshold | inducing points | Mixed |
| Adapt inducing-point count with VIPS instead of fixed M=10 or M=20 | Continual GP on synthetic data · Synthetic continual-learning dataset | RMSE | Helped |
| Adapt inducing-point count with VIPS instead of fixed M=30 | Continual GP on synthetic data · Synthetic continual-learning dataset | RMSE | No clear effect |
| Adapt model size with VIPS instead of fixed M=100 or M=1000 | Online GP on Naval · Naval | training time | Helped |
| Use VIPS with a single preset hyperparameter across datasets | Online GP · UCI datasets | model size at the accuracy threshold | Helped |
| Select candidate locations from the current batch only rather than from old inducing points and the current batch (ablation) | VIPS · Continual GP experiments | model size | No clear effect |
| Use VIPS rather than CV for magnetic-field mapping | Online GP · Magnetic field mapping | model capacity | Helped |
Surprising findings
- The Bike dataset causes CV to hit the 7000-point maximum capacity at the 10% RMSE operating point.
- VIPS (recommended delta=0.035) can grow fewer inducing points than a fixed oracle size of M=100 on Naval in training time, beating even the oracle.
Limitations
- Naval dataset, nearly noiseless, was not solved by any compared method under the tested hyperparameter range.
- Optimises only the number of inducing points with fixed earlier inducing locations; inducing point positions are not jointly optimised.
- Upper bound computed with the optimal LML is intractable for large batch sizes, requiring a looser stopping criterion.
Experimental setup
Datasets: UCI Concrete UCI Skillcraft UCI Kin8nm UCI Naval UCI Elevators UCI Bike synthetic sin/cos function magnetic field robot mapping data
Benchmarks: UCI Concrete UCI Skillcraft UCI Kin8nm UCI Naval UCI Elevators UCI Bike synthetic regression magnetic field mapping
Metrics: RMSE NLPD number of inducing points training time
Baselines: Conditional Variance (CV) OIPS fixed-size streaming sparse GP exact full-batch GP
Builds on: online sparse variational GP (streaming sparse GP) greedy variance inducing point selection Online Inducing Points Selection (OIPS)
Closest papers by meaning
- TinySubNets: An Efficient and Low Capacity Continual Learning Strategy (AAAI 2025 · similarity 0.86)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.85)
- How Does Critical Batch Size Scale in Pre-training? (ICLR 2025 · similarity 0.85)
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.84)
- Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference (NeurIPS 2024 · similarity 0.84)
- Bayesian Adaptation of Network Depth and Width for Continual Learning (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.