Research map / Continual learning / Online and replay-based
Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCLIS, a rehearsal-based prototype contrastive continual learner that uses importance-sampling-based replay buffer selection (minimizing estimated variance) plus a self-distilled prototype-instance relation loss (PRD) to counter catastrophic forgetting.
Problem
Catastrophic forgetting in rehearsal-based contrastive continual learning, where the small replay buffer poorly recovers previous task distributions and hard negatives are poorly selected and preserved.
Approach
Reformulates a prototype-based supervised InfoNCE under continual learning as a biased importance-sampling estimator, selects replay buffer samples by minimizing the KL divergence between proposal and target distributions (equivalently hard-negative retention), and adds a prototype-instance relation distillation loss from the previous model.
Key results
- On Seq-CIFAR-10 with buffer 200 (Class-IL), CCLIS reaches 74.95% vs. 65.57% for Co2L and 49.16% for ER.
- On Seq-CIFAR-100 Class-IL with buffer 500, CCLIS reaches 46.08% vs. 45.91% for GCR (near tie), and on Task-IL 74.51% vs. 71.73% for DER.
- Ablation on Seq-CIFAR-10 (buffer 200, Class-IL): IS only 58.37%, PRD only 73.95%, both 74.95%, neither 56.43%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add importance sampling to the prototype-based contrastive loss without PRD (ablation) | CCLIS with random sampling baseline · Seq-Cifar-10 Class-IL, buffer 200 | accuracy: 56.43$\pm$1.54 -> 58.37$\pm$ 1.17 | Helped |
| Add PRD to random sampling without importance sampling (ablation) | CCLIS · Seq-Cifar-10 Class-IL, buffer 200 | accuracy: 56.43$\pm$1.54 -> 73.95$\pm$1.12 | Helped |
| Add PRD to importance sampling (ablation) | CCLIS · Seq-Cifar-10 Class-IL, buffer 200 | accuracy: 58.37$\pm$ 1.17 -> 74.95$\pm$0.61 | Helped |
| Use importance sampling instead of RBS alone (ablation) | CCLIS · Seq-Cifar-10 Class-IL and Task-IL | accuracy | Helped |
| Use symmetric prototype-based InfoNCE instead of asymmetric instance-wise InfoNCE (ablation) | Contrastive continual learning without distillation or importance sampling · Seq-Cifar-10 and Seq-Cifar-100 Class-IL | accuracy | Helped |
| Increase buffer size (ablation) | CCLIS · Seq-Cifar-10 Class-IL | accuracy | Mixed |
Surprising findings
- Symmetric prototype-based loss outperforms asymmetric instance-wise loss of Co2L in the ablation, and the paper argues prototypes preserve knowledge better than instance-wise contrastive loss.
- Importance sampling alone gives only a small gain (56.43% to 58.37%), while PRD distillation contributes most of the improvement.
Limitations
- Data distribution recovery only covers the last visible samples, far from the entire data distribution.
- Benefit of importance sampling diminishes as buffer size grows, and estimation bias is significant with very small buffers.
Experimental setup
Datasets: Seq-CIFAR-10 Seq-CIFAR-100 Seq-Tiny-ImageNet
Benchmarks: Seq-CIFAR-10 Seq-CIFAR-100 Seq-Tiny-ImageNet
Metrics: accuracy average forgetting
Baselines: ER iCaRL GEM GSS DER Co2L GCR
Builds on: Co2L GCR SupCon InfoNCE ProtoNCE
Closest papers by meaning
- Provable Contrastive Continual Learning (ICML 2024 · similarity 0.91)
- Predicting the Susceptibility of Examples to Catastrophic Forgetting (ICML 2025 · similarity 0.88)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.88)
- Summarizing Stream Data for Memory-Constrained Online Continual Learning (AAAI 2024 · similarity 0.87)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.87)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.