Research map / Continual learning / Online and replay-based
Learning to Continually Learn with the Bayesian Principle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sequential Bayesian Meta-Continual Learning (SB-MCL), where meta-learned neural networks stay fixed while continual learning happens as exact sequential Bayesian updates of an exponential-family posterior over a latent z.
Problem
Continual learning with SGD suffers catastrophic forgetting on non-stationary streams; the paper seeks a meta-continual learning method that is immune to forgetting by design while retaining expressive neural models.
Approach
Meta-train neural networks (learner and model) with SGD only in the outer loop, while continual learning is done in closed form by sequentially updating an exponential-family (factorized Gaussian) posterior over a latent z; the Fisher-Darmois-Koopman-Pitman theorem motivates this choice. Unifies GeMCL, PN, and ALPaCA as special cases.
Key results
- On CASIA classification with 10 tasks and 10 shots, SB-MCL reaches 0.002 error versus 0.015 for OML and 0.004 for TF, while on Celeb TF (0.228) is slightly better than SB-MCL (0.231).
- With 500 tasks on CASIA classification, SB-MCL's error grows only to 0.036 while TF reaches 0.980 and OML 0.564.
- Meta-training 50K steps on a single A40 takes 40 min for SB-MCL versus 6.5 hr for OML on classification and 8 hr versus 5 days for DDPM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SB-MCL instead of TF | Continual image classification · Omniglot classification | error rate: $.014^{\pm .001}$ -> $.008^{\pm .000}$ | Helped |
| Use SB-MCL instead of TF | Continual image classification · CASIA classification | error rate: $.004^{\pm .000}$ -> $.002^{\pm .000}$ | Helped |
| Use SB-MCL instead of TF | Continual image classification · Celeb classification | error rate: $.228^{\pm .003}$ -> $.231^{\pm .004}$ | Hurt |
| Use SB-MCL instead of TF | Continual regression · Sine regression | loss: $.0009^{\pm .0001}$ -> $.0011^{\pm .0002}$ | Hurt |
| Use SB-MCL instead of TF | Continual image completion · CASIA image completion | loss: $.097^{\pm .000}$ -> $.100^{\pm .001}$ | Hurt |
| Use SB-MCL instead of TF | Continual rotation prediction · CASIA rotation prediction | loss: $.034^{\pm .001}$ -> $.039^{\pm .001}$ | Hurt |
| Use SB-MCL instead of TF | Continual image completion · Celeb image completion | loss: $.094^{\pm .000}$ -> $.096^{\pm .000}$ | Hurt |
| Use SB-MCL instead of OML | Continual generation · CASIA VAE | loss: $.442^{\pm .003}$ -> $.428^{\pm .001}$ | Helped |
| Use SB-MCL instead of OML | Continual generation · CASIA DDPM | loss: $.0353^{\pm .0001}$ -> $.0345^{\pm .0001}$ | Helped |
| Use SB-MCL instead of OML-Rep | Continual generation · Celeb DDPM | loss: $.0307^{\pm .0004}$ -> $.0302^{\pm .0004}$ | Helped |
| Replace Monte Carlo posterior predictive estimation with MAP estimation (ablation) | SB-MCL · CASIA classification, 10 tasks, 10 shots | error rate: $.002^{\pm .000}$ -> $.002^{\pm .000}$ | No clear effect |
Surprising findings
- SB-MCL performance improves slightly as the number of shots increases, while OML and TF degrade with more shots.
- Offline and online learning baselines perform far worse than meta-continual learning methods because MCL leverages a large meta-training set.
Limitations
- Exponential-family posterior ignores the order of training data, which may be harmful when curriculum or order matters.
- Constant-memory constraint restricts posterior to exponential families; more flexible non-parametric posteriors are left for future work.
- On Celeb classification, SB-MCL is slightly worse than TF (0.231 vs 0.228).
Experimental setup
Datasets: CASIA MS-Celeb-1M Omniglot Sine wave regression
Benchmarks: CASIA classification Omniglot classification Celeb classification Sine regression CASIA completion CASIA rotation Celeb completion CASIA VAE CASIA DDPM Celeb DDPM
Metrics: error rate loss (MSE / negative log-likelihood-based loss) meta-training time
Baselines: OML OML-Rep Transformer Linear Transformer Offline learning Online learning
Builds on: OML Reptile GeMCL ALPaCA Prototypical Networks
Closest papers by meaning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.90)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.90)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.90)
- Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.89)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.