Research map / Continual learning / Online and replay-based
On the Diminishing Returns of Width for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a finite-width theoretical bound showing continual-learning error (forgetting) of sparse feed-forward networks decays as W^{-beta} with width, implying diminishing returns, and validates it on rotated MNIST/SVHN/Fashion-MNIST/GTSRB experiments up to width 2^16.
Problem
Prior work observed that wider networks forget less in continual learning, but the exact finite-width relationship between width and catastrophic forgetting was not characterized, and width scaling tested only up to 2048 hidden units.
Approach
Combines a shared-active-row analysis under row-wise sparsity with an empirical assumption that wider networks move less from initialization (relative distance ~ W^{-beta}), then uses perturbation analysis to bound the output difference between sequentially trained models, yielding error O(t W^{-beta} alpha^{(1-2beta)/2}) and hence diminishing returns in width.
Key results
- Average forgetting on MNIST at width 2^14 with sparsity alpha=0.1 drops to 0.2, versus 27.9 in the dense setting.
- Empirical distance-from-initialization fits the predicted W^{-beta} trend with gamma=0.013, beta=0.311 for Rotated MNIST and gamma=2.5, beta=0.12 for Fashion MNIST.
- Wide ResNet forgetting on rotated SVHN and GTSRB slowly decreases with width to 40.1 and 40.9 respectively, showing a similar diminishing-returns pattern.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase width | Feed-forward networks · Rotated MNIST and Rotated Fashion MNIST | Average Forgetting | Helped |
| Increase width | Feed-forward networks · Rotated SVHN, two- and three-layer networks | Average Forgetting | Hurt |
| Increase width | Feed-forward networks · Rotated SVHN, one-layer network | Average Forgetting | No clear effect |
| Increase width | Wide ResNet · Rotated SVHN and Rotated GTSRB | Average Forgetting | Helped |
| Use row-wise sparsity with α=0.1 instead of dense training (ablation) | Feed-forward networks · MNIST, width 2^14 | Average Forgetting: 27.9 -> 0.2 | Helped |
| Increase depth from 1 to 3 layers (ablation) | Feed-forward networks · Rotated Fashion MNIST and Rotated GTSRB | Average Forgetting | Hurt |
| Increase depth (ablation) | Feed-forward networks · Rotated MNIST and Rotated SVHN | Average Forgetting | No clear effect |
| Increase depth beyond 4 hidden layers (ablation) | Feed-forward networks · Rotated Fashion MNIST, Rotated SVHN, and Rotated GTSRB | Average Forgetting | Mixed |
| Use Adam instead of SGD (ablation) | Feed-forward networks · Continual learning benchmarks | Average Forgetting | No clear effect |
| Increase the number of sequential tasks (ablation) | Feed-forward networks · Rotated GTSRB and Rotated Fashion MNIST | Accuracy | Hurt |
| Apply dropout at probabilities 0.1, 0.2, or 0.3 (ablation) | Feed-forward networks · Distance from initialization across widths | Distance from initialization | No clear effect |
Surprising findings
- Increasing width does not always reduce forgetting; on SVHN the first-layer forgetting barely changes with width and increases with width in two- and three-layer networks.
- Forgetting decreases at depths beyond 4 hidden layers, which the authors attribute to lower accuracy from vanishing gradients rather than better retention.
Limitations
- The key distance-from-initialization assumption is empirically motivated and not rigorously proven.
- Analysis is restricted to feed-forward networks and does not extend to residual or attention architectures.
- The bound does not perfectly predict the exact continual learning error.
Experimental setup
Datasets: Rotated MNIST Rotated Fashion MNIST Rotated SVHN Rotated GTSRB
Benchmarks: Rotated MNIST Rotated Fashion MNIST Rotated SVHN Rotated GTSRB
Metrics: average forgetting average accuracy learning accuracy joint accuracy
Baselines: Wide ResNet
Builds on: Lazy training Adam Ideal Continual Learner
Closest papers by meaning
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.90)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.89)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.89)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.88)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.88)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.