Research map / Continual learning / Online and replay-based
Saliency-driven Experience Replay for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SER, which uses a continually trained saliency prediction network whose features multiplicatively modulate a paired classifier's features, boosting online continual learning methods on Mini-ImageNet and FG-ImageNet.
Problem
Online continual learning suffers catastrophic forgetting; existing bio-inspired methods ignore early visual saliency processes that may be forgetting-free.
Approach
Pair a classifier with a saliency prediction network (trained with a KL saliency loss on oracle maps) and multiply classifier pre-activations by the saliency encoder's features, with gradients from the classification loss stopped from reaching the saliency encoder, so saliency features remain stable across tasks.
Key results
- SER improves Class-IL accuracy of DER++, ER-ACE and CoPE across buffer sizes, e.g. ER-ACE on Mini-ImageNet at buffer 1000 rises from 20.86 to 27.48 and CoPE at buffer 5000 from 24.77 to 45.04.
- On a spurious-feature benchmark, SER recovers most of the ~22-point drop caused by spurious features for ER-ACE.
- Under PGD attacks, ER-ACE with SER (Class-IL 44.08) is far more robust than ER-ACE with spurious features (28.46) at the reported setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add saliency-driven feature modulation to ER-ACE | ER-ACE · Split Mini-ImageNet Class-IL, buffer 2000 | accuracy: 24.93±3.20 -> 33.09±1.28 | Helped |
| Add saliency-driven feature modulation to ER-ACE | ER-ACE · Split FG-ImageNet Class-IL, buffer 2000 | accuracy: 16.45±1.24 -> 23.80±2.11 | Helped |
| Add saliency-driven feature modulation to DER++ | DER++ · Split Mini-ImageNet Class-IL, buffer 2000 | accuracy: 12.82±4.97 -> 22.92±2.25 | Helped |
| Add saliency-driven feature modulation to DER++ | DER++ · Split FG-ImageNet Class-IL, buffer 2000 | accuracy: 8.27±1.72 -> 12.97±1.62 | Helped |
| Add saliency-driven feature modulation to CoPE | CoPE · Split Mini-ImageNet Class-IL, buffer 2000 | accuracy: 23.58±4.39 -> 33.35±4.67 | Helped |
| Add saliency-driven feature modulation to CoPE | CoPE · Split FG-ImageNet Class-IL, buffer 2000 | accuracy: 16.81±0.83 -> 27.14±1.62 | Helped |
| Add SER when training on biased images with spurious features | ER-ACE-SF · Spurious-feature benchmark Class-IL | accuracy: 28.46±3.46 -> 44.08±3.67 | Helped |
| Modulate all five feature points instead of only the first three (ablation) | SER + DER++ · Split Mini-ImageNet Class-IL, buffer 2000 | accuracy: 12.97±2.62 -> 22.92±2.25 | Helped |
| Modulate all five feature points instead of only the first three (ablation) | SER + ER-ACE · Split Mini-ImageNet Class-IL, buffer 2000 | accuracy: 23.72±0.77 -> 33.09±1.28 | Helped |
| Modulate all five feature points instead of only the first three (ablation) | SER + DER++ · Split FG-ImageNet Class-IL, buffer 2000 | accuracy: 6.54±0.67 -> 12.97±1.62 | Helped |
| Modulate all five feature points instead of only the first three (ablation) | SER + ER-ACE · Split FG-ImageNet Class-IL, buffer 2000 | accuracy: 18.08±0.96 -> 23.80±2.11 | Helped |
| Use multiplicative feature modulation instead of input multiplication, saliency as an extra channel, or learned 1×1-convolution fusion (ablation) | ER-ACE with saliency integration · Split Mini-ImageNet and Split FG-ImageNet Class-IL | accuracy | Helped |
Surprising findings
- Saliency prediction accuracy does not degrade and slightly improves as CL tasks are added, while GradCAM attribution maps suffer forgetting.
- SER outperforms DualNet and TwF, two existing multi-branch CL methods, at comparable or lower training cost.
Limitations
- Requires an external saliency oracle (ground-truth maps or pseudo-labels from a pre-trained predictor), and high-resolution inputs; low-resolution datasets like CIFAR are not usable.
- Saliency encoder and classifier must share identical architectures, limiting use with heterogeneous networks.
- Inference is slower than baselines due to the second branch.
Experimental setup
Datasets: Mini-ImageNet FG-ImageNet ImageNet
Benchmarks: Split Mini-ImageNet Split FG-ImageNet
Metrics: final average accuracy (Class-IL) Task-IL accuracy similarity (SIM) CC KLD accuracy drop under PGD attack
Baselines: DER++ ER-ACE CoPE LwF oEWC DualNet TwF PEC OnPro TASS
Builds on: DER++ ER-ACE CoPE DualNet TwF
Closest papers by meaning
- Understanding and Visualizing Deep Visual Saliency Models (CVPR 2019 · similarity 0.86)
- Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning (CVPR 2024 · similarity 0.85)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.85)
- Re-evaluating Continual Learning with Few-Shot Adaptation (NeurIPS 2026 · similarity 0.85)
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.84)
- Continual Learning in the Frequency Domain (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.