Research map / Continual learning / Online and replay-based
Provable Contrastive Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives theoretical bounds for contrastive continual learning (contrastive plus distillation loss) and proposes CILA, which sets the distillation coefficient per task from the ratio of past distillation to contrastive losses.
Problem
Contrastive continual learning combining contrastive and distillation losses works well empirically but lacks theoretical explanation of its performance guarantees on the final model across all seen tasks.
Approach
Derives recursive bounds linking contrastive losses of consecutive models via the distillation loss, yielding a test-loss bound as a weighted sum of training losses; this motivates adaptive per-task distillation coefficients computed as the ratio of accumulated distillation loss to contrastive loss (CILA).
Key results
- CILA reaches 76.03% Class-IL accuracy on Seq-CIFAR-10 with a 500 buffer, versus 74.26% for Co2L.
- CILA reaches 98.36% Domain-IL accuracy on R-MNIST with a 200 buffer, versus 97.90% for Co2L.
- Ablation on Seq-CIFAR-10 (200 buffer) shows adaptive variants gain about 1 point over the fixed-coefficient Co2L baseline in Class-IL (65.57 to 66.52-67.06).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pure adaptive distillation coefficients instead of the fixed coefficient (ablation) | CILA · Seq-CIFAR-10 Class-IL, buffer 200 | accuracy: 65.57 -> 66.52 | Helped |
| cap the adaptive distillation coefficient at 1 (ablation) | CILA · Seq-CIFAR-10 Class-IL, buffer 200 | accuracy: 65.57 -> 66.36 | Helped |
| floor the adaptive distillation coefficient at 1 (ablation) | CILA · Seq-CIFAR-10 Class-IL, buffer 200 | accuracy: 65.57 -> 67.06 | Helped |
| use pure adaptive distillation coefficients instead of the fixed coefficient (ablation) | CILA · Seq-CIFAR-10 Task-IL, buffer 200 | accuracy: 93.43 -> 94.27 | Helped |
| cap the adaptive distillation coefficient at 1 (ablation) | CILA · Seq-CIFAR-10 Task-IL, buffer 200 | accuracy: 93.43 -> 94.21 | Helped |
| floor the adaptive distillation coefficient at 1 (ablation) | CILA · Seq-CIFAR-10 Task-IL, buffer 200 | accuracy: 93.43 -> 94.29 | Helped |
Surprising findings
- The empirical ratio of accumulated distillation to contrastive losses stays close to one, which the authors use to justify the adaptive coefficient.
- The theory shows that a fixed lambda=1 can be suboptimal in extreme task-weight settings, where the appropriate lambda may be near 10.
Limitations
- Theoretical bounds are derived for a simplified setting (one negative sample in the main text, normalized embeddings, bounded losses) and the adaptive rule is motivated by heuristic reasoning rather than optimized by the bounds directly.
- Gains over Co2L are modest, around 1 to 2 points, and results are reported only on small benchmarks with ResNet-18 and a simple CNN.
- The hyperparameter kappa is needed and its selection is not systematically studied.
Experimental setup
Datasets: Seq-CIFAR-10 Seq-Tiny-ImageNet R-MNIST CIFAR-10 Tiny-ImageNet MNIST
Benchmarks: Seq-CIFAR-10 Seq-Tiny-ImageNet R-MNIST
Metrics: classification accuracy
Baselines: ER GEM A-GEM iCaRL FDR GSS HAL DER DER++ Co2L
Builds on: Co2L supervised contrastive loss IRD distillation DER++
Closest papers by meaning
- Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation (AAAI 2024 · similarity 0.91)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.88)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.88)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.87)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.