Research map / Continual learning / Online and replay-based
Function-space Parameterization of Neural Networks for Sequential Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts a trained neural network into a sparse function-space GP via a dual parameterization (dual parameters from the NN's MAP), enabling scalable uncertainty, continual learning regularization, and incorporation of new data via dual updates without retraining.
Problem
Gradient-based deep learning struggles with sequential learning (forgetting, incorporating new data without retraining), while GPs that offer these benefits do not scale to large data sets or rich inputs such as images.
Approach
Linearize the trained NN around its MAP weights to obtain a zero-mean GP with NTK kernel, express the posterior via dual parameters (alpha, beta) computed from the NN's loss derivatives, and sparsify by projecting these duals onto inducing points using all training data, avoiding a second optimization loop; new data is added by simple dual updates.
Key results
- On CIFAR-10 with CNN and M=3200 inducing points, SFR achieves NLPD 0.72 and accuracy 78.48%, versus GP subset NLPD 1.08 and accuracy 69.75%.
- On HouseElectric, SFR reaches NLPD -0.153 vs 0.024 for SVGP and wall-clock 6129 s vs 11982 s.
- On Split-MNIST (40 pts/task) continual learning, SFR achieves 89.22% accuracy, the best among compared methods including DER (85.26%) and S-FSVI (84.51%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse dual parameters rather than a GP subset | SFR · Fashion-MNIST and CIFAR-10 | NLPD: Fashion-MNIST: 0.79 ± 0.09 -> 0.29 ± 0.02; CIFAR-10: 1.08 ± 0.05 -> 0.72 ± 0.02 | Helped |
| Use sparse dual parameters rather than Laplace GLM predictions | SFR · Fashion-MNIST and CIFAR-10 | accuracy (%): Fashion-MNIST: 84.79 ± 1.96 -> 91.74 ± 0.47; CIFAR-10: 75.56 ± 1.63 -> 78.48 ± 0.98 | Helped |
| Replace the covariance regularizer with identity (L2) (ablation) | SFR continual-learning regularizer · Split-MNIST (SH), 40 points/task | accuracy: 89.22 ± 0.76 -> 87.53 ± 0.36 | Hurt |
| Replace the covariance regularizer with identity (L2) (ablation) | SFR continual-learning regularizer · Split-MNIST (SH), 200 points/task | accuracy: 94.19 ± 0.26 -> 93.72 ± 0.10 | Hurt |
| Replace the covariance regularizer with identity (L2) (ablation) | SFR continual-learning regularizer · Split-FashionMNIST (SH), 200 points/task | accuracy: 81.96 ± 0.24 -> 81.21 ± 0.36 | Hurt |
| Replace the covariance regularizer with identity (L2) (ablation) | SFR continual-learning regularizer · Permuted-MNIST (SH), 200 points/task | accuracy: 95.58 ± 0.08 -> 94.96 ± 0.15 | Hurt |
| Incorporate new data using dual updates instead of retraining | SFR · Airfoil | NLPD: 0.60 ± 0.02 -> 0.50 ± 0.02; retrain: 0.47 ± 0.03 | Mixed |
| Incorporate new data using dual updates instead of retraining | SFR · Boston | NLPD: 0.23 ± 0.01 -> 0.16 ± 0.02; retrain: 0.13 ± 0.02 | Mixed |
| Incorporate new data using dual updates instead of retraining | SFR · Protein | NLPD: 0.42 ± 0.01 -> 0.15 ± 0.01; retrain: 0.14 ± 0.00 | Mixed |
| Incorporate new data using dual updates instead of retraining | SFR · Airfoil | time (s): 0.04 ± 0.00 vs. 18.22 ± 2.27 | Helped |
| Use SFR rather than SVGP | HouseElectric · HouseElectric (5-fold split) | NLPD: 0.024 ± 0.984 -> -0.153 ± 0.001 | Helped |
| Use SFR rather than SVGP | HouseElectric · HouseElectric (5-fold split) | wall-clock time: 11982 ± 455 s -> 6129 ± 996 s | Helped |
Surprising findings
- Dual updates did not improve predictive performance on image datasets, which the authors attribute to the NTK not being accurate outside training data.
- Weight-space regularization methods (Online-EWC, SI) fail almost entirely on Split-MNIST and Split-FashionMNIST single-head settings.
Limitations
- Linearization around MAP weights makes the prior and posterior only locally linear approximations of the NN.
- Unlike vanilla GPs, no straightforward way to specify the prior covariance function; it is implicitly set by architecture.
- Dual updates do not improve predictions on image data and the model becomes outdated as new data arrives.
Experimental setup
Datasets: UCI Fashion-MNIST CIFAR-10 HouseElectric Split-MNIST Split-FashionMNIST Permuted-MNIST MNIST (OOD) cartpole swingup (MuJoCo)
Benchmarks: UCI Fashion-MNIST CIFAR-10 HouseElectric Split-MNIST Split-FashionMNIST Permuted-MNIST cartpole swingup
Metrics: NLPD accuracy AUROC wall-clock time average accuracy in continual learning episode return
Baselines: Laplace diag Laplace kron GP subset GP predictive Online-EWC SI VCL DER FROMP S-FSVI SVGP DDPG
Builds on: Laplace approximation Generalized Gauss-Newton Sparse variational GP (DTC) DNN2GP VCL FROMP
Closest papers by meaning
- FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning (NeurIPS 2024 · similarity 0.88)
- Function-Space Learning Rates (ICML 2025 · similarity 0.86)
- BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning (CVPR 2025 · similarity 0.86)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.86)
- Linearization Turns Neural Operators into Function-Valued Gaussian Processes (ICML 2025 · similarity 0.86)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.