Research map / Continual learning / Online and replay-based
Continual learning with the neural tangent ensemble
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interprets a neural network in the lazy regime as a Bayesian ensemble of N fixed "neural tangent experts" and shows the posterior update is approximately single-example SGD projected to the L1 simplex, used to reduce catastrophic forgetting.
Problem
Neural networks forget previous tasks when trained sequentially with gradient descent; the paper asks whether a single network can be viewed as a Bayesian ensemble of fixed experts that learns without forgetting.
Approach
A first-order Taylor expansion around a seed point decomposes a network with N parameters into a weighted ensemble of N classifiers (neural tangent experts), each a valid label distribution; Bayesian posterior updating over these experts equals a scaled, projected single-example SGD on the weights.
Key results
- Momentum in SGD monotonically increases forgetting of the first task on Permuted MNIST, so any momentum is harmful for retaining past tasks.
- Adam's forgetting is not helped by increasing network width on Permuted MNIST, while SGD without momentum forgets less as width grows.
- The NTE update using current gradients improves both single-task and multitask accuracy with width on Permuted MNIST and CIFAR-100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase SGD momentum without resetting momentum buffers between tasks | SGD on a 2-layer MLP · Permuted MNIST, first-task test accuracy after 5 tasks | accuracy | Hurt |
| Use SGD with momentum instead of no-momentum SGD | ResNet18 and ConvNeXtTiny · Task-incremental CIFAR100 | accuracy | Hurt |
| Increase network width while using Adam | 2-layer MLP · Permuted MNIST | forgetting | No clear effect |
| Increase network width while using the NTE posterior update rule with current gradients | MLP · Permuted MNIST | accuracy | Helped |
| Increase network width while using the NTE posterior update rule with current gradients | ConvNeXtTiny · Task-incremental CIFAR100 | accuracy | Helped |
| Use the NTE posterior update rule with initialization gradients as gradients diverge from initialization | Finite-width neural network · MNIST | performance | Hurt |
| Reset the SGD momentum buffer between tasks (ablation) | 2-layer MLP · Permuted MNIST | accuracy | Mixed |
Surprising findings
- Any amount of momentum with SGD harms retention of past tasks, which prior empirical studies had not noted.
- The posterior update rule for neural tangent experts is nearly identical to single-example SGD with mean-centered gradients.
Limitations
- Theorem 1 holds only for small parameter perturbations and low curvature (zL<1).
- Posterior-weighted ensembles are optimal only for independent experts, so the NTE update is suboptimal.
- The NTE update with fixed Jacobian degrades in finite-width networks where gradients change during training.
Experimental setup
Datasets: MNIST CIFAR-100
Benchmarks: Permuted MNIST task-incremental CIFAR-100
Metrics: test accuracy on first task multitask accuracy forgetting
Baselines: SGD with momentum Adam EWC synaptic intelligence
Builds on: neural tangent kernel elastic weight consolidation exponentiated gradient descent
Closest papers by meaning
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.88)
- KAC: Kolmogorov-Arnold Classifier for Continual Learning (CVPR 2025 · similarity 0.88)
- Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification (ICLR 2025 · similarity 0.87)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.87)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.87)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.