Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Recurrent Memory for Online Interdomain Gaussian Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OHSVGP, which uses HiPPO-LegS time-varying orthogonal projections as interdomain inducing variables in a sparse variational GP so inducing variables memorize history, enabling long-term memory in online/continual GP learning.
Problem
Online sparse GP learning forgets past data because inducing points shift with new data, so long-term memory is lost during sequential/continual learning.
Approach
Interpret HiPPO time-varying orthogonal projections as interdomain inducing variables with time-dependent Legendre basis functions; update the prior cross- and inducing covariances via HiPPO ODE recurrences (using random Fourier features for the double integral), then perform standard online SVGP variational updates.
Key results
- On Solar and Audio online regression, OHSGPR consistently achieves the best NLPD over learned tasks, while OSGPR shows catastrophic forgetting around task 5 with small M.
- On ERA5 continual learning, OHSVGP remains stable across tasks while OSVGP, OVC, and OVC-per-dim show clear forgetting.
- Wall-clock runtime on Solar (M=50) for OHSGPR is 0.297 s versus 140 s for OSGPR, since closed-form updates need no training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiPPO-based recurrent memory and interdomain inducing variables instead of standard online inducing points | OHSGPR/OHSVGP · Solar Irradiance and Audio time-series prediction | NLPD | Helped |
| Use HiPPO-based recurrent memory and interdomain inducing variables instead of standard online inducing points | OHSVGP · COVID death-count prediction | NLPD | Helped |
| Use oracle ordering rather than kernel-similarity ordering for the HiPPO recurrence (ablation) | OHSVGP continual learning · Skillcraft and Powerplant UCI datasets | NLPD | Helped |
| Use kernel-similarity sorting rather than oracle sorting (ablation) | OHSVGP continual learning · Skillcraft | NLPD | Hurt |
| Replace standard inducing points with HiPPO inducing variables and recurrence updates | SVGPVAE · ERA5 continual high-dimensional time-series prediction | NLPD | Helped |
| Use direct ODE evolution of the inducing covariance instead of the RFF approximation (ablation) | OHSVGP · Kernel covariance updates | numerical stability | Hurt |
| Use fixed predefined-interval Fourier inducing variables rather than adaptive HiPPO inducing variables | OVFF versus OHSGPR · Early tasks in Solar Irradiance and Audio prediction | NLPD | Hurt |
Surprising findings
- The direct matrix ODE evolution of K_uu is numerically unstable (stiff Lyapunov operator), so the RFF approximation was used instead.
- OHSVGP outperformed a traditional weekly AR(2) renewal-equation infectious disease model fit offline on the full history, though the authors attribute this partly to the renewal model's strong inductive biases.
Limitations
- Continual learning on non-sequential data requires a pseudo time order, and performance depends on the sorting criterion.
- Stationary-kernel RFF approximation is needed for the double integral; non-stationary kernels need additional Fourier feature techniques not developed here.
- Direct ODE evolution of K_uu is numerically unstable, so the approximation relies on RFF samples.
Experimental setup
Datasets: Solar Irradiance Audio (TIMIT) COVID-19 Santa Catarina deaths Skillcraft Powerplant ERA5
Benchmarks: Solar Irradiance Audio signal COVID-19 deaths Skillcraft Powerplant ERA5
Metrics: NLPD RMSE ECE wall-clock runtime
Baselines: OSVGP OVC OVC-optZ OVFF OSGPR OVC-SVGPVAE OVC-per-dim
Builds on: SVGP interdomain GPs OSVGP HiPPO random Fourier features SVGPVAE
Closest papers by meaning
- HiPPO: Recurrent Memory with Optimal Polynomial Projections (NeurIPS 2020 · similarity 0.89)
- Memory-Based Dual Gaussian Processes for Sequential Learning (ICML 2023 · similarity 0.88)
- Incremental Variational Sparse Gaussian Process Regression (NeurIPS 2016 · similarity 0.87)
- Variational Auto-Regressive Gaussian Processes for Continual Learning (ICML 2021 · similarity 0.86)
- Bayesian Basis Function Approximation for Scalable Gaussian Process Priors in Deep Generative Models (ICML 2025 · similarity 0.86)
- Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.