Research map / Continual learning / Online and replay-based
Do Your Best and Get Enough Rest for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the view-batch model (VBM), which replays multiple augmented views of a single sample and adds a one-to-many KL self-supervised loss, to lengthen recall intervals and reduce forgetting in continual learning, as a drop-in plug-in for existing methods.
Problem
Continual learning suffers from catastrophic forgetting because current methods retrain the same samples at short recall intervals, which prevents the spacing effect from improving long-term memory retention.
Approach
Inspired by the forgetting curve, replace the sample-batch replay unit with a view-batch of V augmented views of one sample, which multiplies the recall interval by V while keeping the total number of samples seen constant (epochs reduced by V), plus a one-to-many KL self-supervised loss between weak and strong views.
Key results
- On S-CIFAR-10 with buffer 200, VBM-iCaRL improves CIL accuracy from 64.11% to 69.73% (avg +4.09 over iCaRL).
- On S-CIFAR-100 10-step CIL, VBM-TCIL improves last accuracy from 52.37% to 54.69% in non-rehearsal (Table 4, 5-step), and up to +5.61 on 10-step last accuracy.
- On DomainNet DIL, VBM-DER++ improves average accuracy from 34.56% to 42.16% (+7.24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add view-batch replay and self-supervised learning | DER++ · S-CIFAR-10, buffer 200 | Avg accuracy: 76.14 -> 80.65 | Helped |
| Add view-batch replay and self-supervised learning | iCaRL · S-Tiny-ImageNet, buffer 200 | Avg accuracy: 36.08 -> 38.85 | Helped |
| Add view-batch replay and self-supervised learning | DER++ · DomainNet, DIL | Avg accuracy: 39.84±0.15 -> 42.16±0.20 | Helped |
| Add view-batch replay and self-supervised learning | DER · S-CIFAR-100, 5-step rehearsal CIL | Avg accuracy: 76.77±0.01 -> 78.60±0.23 | Helped |
| Add view-batch model | SLCA · S-CIFAR-100, pre-trained CIL | Last top-1 accuracy: 91.57±0.46 -> 91.78±0.18 | Helped |
| Add view-batch model | SLCA · S-ImageNet-R, pre-trained CIL | Last top-1 accuracy: 78.81±0.21 -> 80.07±0.12 | Helped |
| Add replay without strong augmentation or SSL (ablation) | iCaRL · S-CIFAR-10, CIL | Last accuracy: 64.11 -> 62.04 | Hurt |
| Add replay with weak augmentation only (ablation) | iCaRL · S-CIFAR-10, CIL | Last accuracy: 64.11 -> 67.51 | Helped |
| Add strong augmentation to replay without SSL (ablation) | iCaRL · S-CIFAR-10, CIL | Last accuracy: 64.11 -> 66.78 | Helped |
| Add replay, strong augmentation, and SSL (ablation) | iCaRL · S-CIFAR-10, CIL | Last accuracy: 64.11 -> 69.73 | Helped |
| Add replay without strong augmentation or SSL (ablation) | DER++ · S-CIFAR-10, CIL | Last accuracy: 61.67 -> 62.66 | Helped |
| Add replay, strong augmentation, and SSL (ablation) | DER++ · S-CIFAR-10, CIL | Last accuracy: 61.67 -> 66.99 | Helped |
Surprising findings
- Strong augmentation alone (without view-batch replay) slightly hurts iCaRL CIL accuracy (-1.05), indicating gains come from the replay and SSL components rather than augmentation.
Limitations
- Gains are shown mainly on small-scale sequential benchmarks (S-CIFAR, S-Tiny-ImageNet, S-ImageNet-R, DomainNet) with ResNet-18 backbones.
- Improvements are sometimes marginal or negative (e.g., VBM-DER++ on S-Tiny-ImageNet 500 buffer lower by -0.92 avg).
Experimental setup
Datasets: S-CIFAR-10 S-CIFAR-100 S-Tiny-ImageNet S-ImageNet-R DomainNet
Benchmarks: S-CIFAR-10 S-CIFAR-100 S-Tiny-ImageNet S-ImageNet-R DomainNet
Metrics: last top-1 accuracy average accuracy forgetting
Baselines: iCaRL ER DER DER++ TCIL LwF SLCA L2P DualPrompt CODA-Prompt S-iPrompt UCIR
Builds on: iCaRL DER++ TCIL SLCA Ebbinghaus forgetting curve
Closest papers by meaning
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.91)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.91)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.90)
- Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters (ICML 2025 · similarity 0.90)
- When to Learn and When to Stop: Quitting at the Optimal Time (Student Abstract) (AAAI 2025 · similarity 0.90)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.