Research map / Continual learning / Online and replay-based
Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically compares concurrent vs. sequential rehearsal in continual learning with overparameterized linear models, then proposes a hybrid rehearsal method that shows gains on DNN benchmarks.
Problem
Rehearsal-based continual learning almost always trains on new and memory data concurrently, but it is unclear whether this is optimal; prior theory did not analyze rehearsal at all, so the trade-off between concurrent and sequential rehearsal was unknown.
Approach
Introduces sequential rehearsal (train new task first, then revisit each old task's memory in order) and derives closed-form forgetting and generalization error for concurrent and sequential rehearsal in overparameterized linear regression, showing sequential is better when tasks are dissimilar; uses these insights to build a hybrid method that trains similar tasks concurrently and dissimilar task
Key results
- Hybrid rehearsal improves final accuracy over concurrent rehearsal on Split-TinyImagenet200 by 2.19 points and reduces forgetting by 3.66 points (5-task setting).
- On corrupted Split-TinyImagenet200, hybrid rehearsal reduces forgetting by 13.32 points versus concurrent rehearsal.
- Theory: sequential rehearsal has lower forgetting and generalization error than concurrent rehearsal when the task gap is large relative to the noise, with simulations matching theory (T=5, p=500, n=24, M=24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-CIFAR-10 | Acc: +1.07 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-CIFAR-100 | Fgt: -3.27 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-TinyImagenet200 | Acc: +2.19 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Corrupted Split-CIFAR-100 | Fgt: -7.86 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Corrupted Split-TinyImagenet200 | Fgt: -13.32 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-CIFAR-100, 20 tasks | Acc: +4.32 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-TinyImagenet200, 20 tasks | Fgt: -2.02 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal as label corruption increases | ResNet-18 continual learning · Split-TinyImagenet200, p_cor=20% | Acc: +4.66 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal as label corruption increases | ResNet-18 continual learning · Split-TinyImagenet200, p_cor=20% | Fgt: -4.87 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Split-MNIST | Acc: +0.15 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal | ResNet-18 continual learning · Corrupted Split-MNIST | Fgt: -3.91 | Helped |
| Use hybrid rehearsal instead of concurrent rehearsal as corrupted-task count increases | ResNet-18 continual learning · Split-CIFAR-100, 10 tasks, corruption number 3 | Acc: +0.79 | Helped |
Surprising findings
- Concurrent rehearsal can be worse than sequential rehearsal; the advantage of concurrent rehearsal holds only when tasks are similar and noise is small.
Limitations
- /invoke
Experimental setup
Datasets: Split-MNIST Split-CIFAR-10 Split-CIFAR-100 Split-TinyImagenet200 Corrupted variants of these datasets
Benchmarks: Split-MNIST Split-CIFAR-10 Split-CIFAR-100 Split-TinyImagenet200
Metrics: average final accuracy average final forgetting forgetting (theory) generalization error (theory)
Baselines: concurrent rehearsal
Builds on: concurrent rehearsal reservoir sampling overparameterized linear model CL theory
Closest papers by meaning
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.91)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.90)
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.90)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.89)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.89)
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.