Research map / Continual learning / Online and replay-based
BECAME: Bayesian Continual Learning with Adaptive Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BECAME, a two-stage continual learning method that merges a gradient-projection model with an unconstrained fine-tuned model using a closed-form, Laplace-approximation-based adaptive merging coefficient.
Problem
Continual learning must balance stability (retaining old tasks) and plasticity (learning new tasks); gradient projection methods preserve stability but limit plasticity, and existing model merging approaches use hand-tuned or fixed merging coefficients.
Approach
Reformulates model merging as MAP estimation under Bayesian continual learning with a Laplace approximation, deriving a closed-form optimal merging coefficient from Fisher information; merges a gradient-projected model with an unconstrained model trained on the new task.
Key results
- GPM + Ours reaches 80.57% ACC on 20-Split CIFAR-100 vs 77.34% for GPM; SGP + Ours reaches 76.74% on 10-Split CIFAR-100 (best) and 70.06% on 20-Split MiniImageNet.
- Adam-NSCL + Ours achieves 81.66% ACC on 10-Split CIFAR-100 vs 72.91% for Adam-NSCL and 79.79% for Connector.
- On 10-Split CIFAR-100 without gradient projection, the adaptive merge reaches 65.03% ACC, above CoMA (64.42%) and 1/t (60.69%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BECAME adaptive merging to GPM | GPM · 20-Split CIFAR-100 | ACC (%): 77.34 ± 0.55 -> 80.57 ± 0.18 | Helped |
| Add BECAME adaptive merging to GPM | GPM · 10-Split CIFAR-100 | ACC (%): 71.81 ± 0.10 -> 75.05 ± 0.27 | Helped |
| Add BECAME adaptive merging to GPM | GPM · 20-Split MiniImageNet | ACC (%): 63.90 ± 1.05 -> 67.62 ± 1.40 | Helped |
| Add BECAME adaptive merging to TRGP | TRGP · 20-Split CIFAR-100 | ACC (%): 81.68 ± 0.69 -> 82.61 ± 0.38 | Helped |
| Add BECAME adaptive merging to TRGP | TRGP · 10-Split CIFAR-100 | ACC (%): 75.01 ± 0.63 -> 75.87 ± 0.43 | Helped |
| Add BECAME adaptive merging to TRGP | TRGP · 20-Split MiniImageNet | ACC (%): 62.68 ± 2.49 -> 65.09 ± 1.81 | Helped |
| Add BECAME adaptive merging to SGP | SGP · 20-Split CIFAR-100 | ACC (%): 80.21 ± 0.68 -> 81.94 ± 0.40 | Helped |
| Add BECAME adaptive merging to SGP | SGP · 10-Split CIFAR-100 | ACC (%): 74.97 ± 0.31 -> 76.74 ± 0.36 | Helped |
| Add BECAME adaptive merging to SGP | SGP · 20-Split MiniImageNet | ACC (%): 66.99 ± 2.33 -> 70.06 ± 2.12 | Helped |
| Add BECAME adaptive merging to Adam-NSCL | Adam-NSCL · 20-Split CIFAR-100 | ACC (%): 75.66 ± 0.39 -> 81.88 ± 0.37 | Helped |
| Add BECAME adaptive merging to Adam-NSCL | Adam-NSCL · 10-Split CIFAR-100 | ACC (%): 72.91 ± 0.30 -> 81.66 ± 0.51 | Helped |
| Add BECAME adaptive merging to Adam-NSCL | Adam-NSCL · 25-Split TinyImageNet | ACC (%): 58.57 ± 0.40 -> 66.49 ± 0.48 | Helped |
Surprising findings
- Fixed 1/t merging can improve accuracy yet the adaptive method gives the best balance; CoMA's fixed coefficient yields poor BWT (-2.79 on 10-Split CIFAR-100).
- Gains are smaller on 5-Datasets, attributed to the merging coefficient favoring the hard CIFAR-10 task.
Limitations
- Second training stage increases training time and additional memory for storing the precision matrix.
- Gains on 5-Datasets are limited, as the coefficient can be biased toward hard tasks such as CIFAR-10.
- Relies on a diagonal Fisher approximation and a local quadratic/convexity argument along the merging path.
Experimental setup
Datasets: 20-Split CIFAR-100 10-Split CIFAR-100 20-Split MiniImageNet 25-Split TinyImageNet 5-Datasets CIFAR-100 Superclass
Benchmarks: 20-Split CIFAR-100 10-Split CIFAR-100 20-Split MiniImageNet 25-Split TinyImageNet 5-Datasets CIFAR-100 Superclass
Metrics: ACC BWT IM AAA AOA STD training time GPU memory
Baselines: GPM TRGP SGP GPCNS Adam-NSCL Connector EWC SI LwF OWM A-GEM ER-Res
Builds on: GPM Adam-NSCL Connector IMM CoMA
Closest papers by meaning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.93)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.91)
- Temporal-Difference Variational Continual Learning (NeurIPS 2025 · similarity 0.90)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.90)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.90)
- MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.