Research map / Continual learning / Online and replay-based
Bayesian Adaptation of Network Depth and Width for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a non-parametric Bayesian continual learner that infers network depth with a beta process (stick-breaking) and adapts width via drop-connect masks from a conjugate Bernoulli process, trained with sequential structured variational inference.
Problem
Dynamic-architecture continual learning methods adapt only network width and keep depth fixed, limiting model capacity and performance as tasks arrive.
Approach
Model network depth growth with a beta process and width via conjugate Bernoulli weight-activation masks, performing full sequential Bayesian inference over weights and structure with skip connections so deep layers do not collapse, plus task-specific masks for task-incremental learning.
Key results
- On split fashion MNIST the method reaches 99.33 ± 0.06% accuracy, the best among compared methods in Table 1.
- On CIFAR100-20 with AlexNet it reaches 79.93 ± 0.28%, outperforming IBPCL (77.13 ± 0.94%) and other baselines.
- Ablation on CIFAR10-5 with AlexNet shows Bayesian weight inference plus task-specific masks gives 91.44% vs 77.71% for MAP weights without structure adaptation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use depth and width adaptation with Bayesian inference | Ours vs. IBPCL · permuted MNIST | test accuracy (%): 98.18 ± 0.19 -> 97.98 ± 0.04 | Hurt |
| use depth and width adaptation with Bayesian inference | Ours vs. IBPCL · split MNIST | test accuracy (%): 99.79 ± 0.03 -> 99.76 ± 0.05 | No clear effect |
| use depth and width adaptation with Bayesian inference | Ours vs. IBPCL · split fashion MNIST | test accuracy (%): 99.27 ± 0.02 -> 99.33 ± 0.06 | Helped |
| use depth and width adaptation with Bayesian inference | Ours vs. HAT · CIFAR10-5 | test accuracy (%): 92.29 ± 0.19 -> 91.44 ± 0.16 | Hurt |
| use depth and width adaptation with Bayesian inference | Ours vs. HAT · CIFAR100-10 | test accuracy (%): 72.42 ± 0.36 -> 71.83 ± 0.48 | Hurt |
| use depth and width adaptation with Bayesian inference | Ours vs. IBPCL · CIFAR100-20 | test accuracy (%): 77.13 ± 0.94 -> 79.93 ± 0.28 | Helped |
| use depth and width adaptation with Bayesian inference | Ours vs. SPG · TinyImagenet-10 | test accuracy (%): 49.72 ± 0.22 -> 50.59 ± 0.42 | Helped |
| add sequential Bayesian weight inference instead of MAP estimation (ablation) | network structure adaptation (ada-st) · CIFAR10-5 | test accuracy (%): 77.71 ± 2.40 -> 84.48 ± 0.56 (Alexnet); 79.78 ± 0.98 -> 86.54 ± 0.79 (fullyConv-7) | Helped |
| add task-specific masks (ablation) | network structure adaptation with MAP weight estimation · CIFAR10-5 | test accuracy (%): 77.71 ± 2.40 -> 87.34 ± 0.74 (Alexnet); 79.78 ± 0.98 -> 86.55 ± 0.76 (fullyConv-7) | Helped |
| add task-specific masks (ablation) | network structure adaptation with Bayesian weight inference · CIFAR10-5 | test accuracy (%): 84.48 ± 0.56 -> 91.44 ± 0.16 (Alexnet); 86.54 ± 0.79 -> 91.02 ± 0.29 (fullyConv-7) | Helped |
| add structure adaptation (ablation) | ER-ACE · CIFAR10-5 class-incremental | test accuracy (%): 50.86 ± 2.44 -> 53.83 ± 1.34 | Helped |
| add Bayesian weight inference (ablation) | ER-ACE with structure adaptation · CIFAR10-5 class-incremental | test accuracy (%): 53.83 ± 1.34 -> 56.55 ± 0.61 | Helped |
Surprising findings
- Hard-attention HAT's performance collapses faster than the proposed method as network depth grows, which the authors attribute to accumulated sparsity across layers.
Limitations
- Pooling layers are hard to combine with stochastic depth because feature-map dimensionality varies with the inferred layers.
- Truncation of the variational distribution is still needed, though it could be relaxed with Russian Roulette.
- Performance is not uniformly best; it is second to IBPCL on permuted and split MNIST and third on CIFAR10-5 with AlexNet.
Experimental setup
Datasets: permuted MNIST split MNIST split fashion MNIST CIFAR10-5 CIFAR100-10 CIFAR100-20 TinyImagenet-10 one-MNIST not-MNIST CIFAR10-5 (class-incremental)
Benchmarks: permuted MNIST split MNIST split fashion MNIST CIFAR10-5 CIFAR100-10 CIFAR100-20 TinyImagenet-10 one-MNIST not-MNIST
Metrics: final test accuracy test log-likelihood
Baselines: EWC VCL UCL SFSVI DEN HAT HIBNN IBPCL SPG
Builds on: VCL IBPCL HIBNN Bayes-by-Backprop structured stochastic variational inference
Closest papers by meaning
- Bayesian Structural Adaptation for Continual Learning (ICML 2021 · similarity 0.89)
- On the Diminishing Returns of Width for Continual Learning (ICML 2024 · similarity 0.87)
- TinySubNets: An Efficient and Low Capacity Continual Learning Strategy (AAAI 2025 · similarity 0.85)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.85)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.85)
- Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.