Research map / Continual learning / Online and replay-based
Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes task-modulated contrastive learning (TMCL), which learns per-class affine modulations from sparse labels to orthogonalize classes, then consolidates them into feedforward weights via view- and modulation-invariant contrastive learning for sparsely supervised class-incremental learning.
Problem
Continual representation learning from mostly unlabeled streams with sparse labels, where supervised fine-tuning causes catastrophic forgetting and degrades generalization.
Approach
Learn class-specific affine top-down modulations on frozen feedforward weights with an orthogonal projection loss using labeled samples, then train feedforward weights to be invariant between unmodulated and modulated views (modulation invariance), consolidating class information without labels and stabilizing past knowledge.
Key results
- On CIFAR-100 with 1% labels, VI+MI (TMCL) reaches 60.7% linear readout vs 39.1% for SupCon and 41.6% for CE (5 sessions).
- On ImageNet-100 with 1% labels, VI+SI+MI reaches 62.0% vs 60.2% for VI+SupCon and 33.6% for SupCon.
- TMCL remains robust to 99% label noise (59.4%) where VI+SupCon drops to 58.7%, and transfer kNN improves across most of 9 downstream datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add modulation-invariance loss (MI) to VI | ConViT continual learning · CIFAR-100/5, 1% labels | linear readout accuracy: 59.3±0.2 -> 60.7±0.3 | Helped |
| add modulation-invariance loss (MI) to VI | ConViT continual learning · ImageNet-100/5, 1% labels | linear readout accuracy: 59.7±0.3 -> 62.0±0.2 | Helped |
| add MI rather than SupCon to VI | ConViT continual learning · CIFAR-100/5, 1% labels | linear readout accuracy: 59.3±0.2 -> 60.7±0.3 | Helped |
| add MI rather than SupCon to VI | ConViT continual learning · ImageNet-100/5, 1% labels | linear readout accuracy: 60.2±0.3 -> 62.0±0.2 | Helped |
| add MI to VI (ablation) | ConViT continual learning · CIFAR-100/5, all labels | linear readout accuracy: 59.3±0.2 -> 60.7±0.4 | Helped |
| add MI to VI (ablation) | ConViT continual learning · ImageNet-100/5, all labels | linear readout accuracy: 59.7±0.3 -> 64.5±0.3 | Helped |
| train orthogonalizing modulations rather than leave them untrained (ablation) | VI + MI (TMCL) · CIFAR-100/5, all labels | linear readout accuracy: 60.1±0.5 -> 60.7±0.4 | Helped |
| train orthogonalizing modulations rather than redraw random modulations (ablation) | VI + MI (TMCL) · CIFAR-100/5, all labels | linear readout accuracy: 59.7±0.5 -> 60.7±0.4 | Helped |
| replace BCE with OPL for modulation orthogonalization (ablation) | VI + MI (TMCL) · CIFAR-100/5, all labels | linear readout accuracy: 59.5±0.5 -> 60.7±0.4 | Helped |
| add a predictor to MI (ablation) | VI + MI (TMCL) · CIFAR-100/5, all labels | linear readout accuracy: 55.1 ±0.1 -> 60.7 ±0.4 | Helped |
| add stop-gradient to the modulated branches (ablation) | VI + MI (TMCL) · CIFAR-100/5, all labels | linear readout accuracy: 60.3 ±0.3 -> 60.7 ±0.4 | Helped |
| increase MI strength | VI + λMI MI · CIFAR-100 continual learning | forward and backward transfer | Mixed |
Surprising findings
- Modulations that are untrained or randomly redrawn still give moderate improvement over pure view-invariant learning, suggesting a regularization effect of perturbed model states.
- Standard BatchNorm-based ResNet-18 hinders stable training of modulations, while LayerNorm ViT architectures do not.
Limitations
- Modulation memory scales with the number of classes (about 4.1M parameters for 100 classes).
- Representational quality on CIFAR-100 improves only moderately with more labels and underperforms SupCon in the fully labeled regime.
- Experiments limited to standard class-incremental CIFAR-100 and ImageNet-100; domain-incremental settings not studied.
Experimental setup
Datasets: CIFAR-100 ImageNet-100 Aircraft CIFAR-10 CUBirds DTD EuroSAT GTSRB STL-10 SVHN VGGFlower
Benchmarks: CIFAR-100 ImageNet-100 Aircraft CIFAR-10 CUBirds DTD EuroSAT GTSRB STL-10 SVHN VGGFlower
Metrics: linear readout accuracy kNN accuracy backward transfer forward transfer CDNV
Baselines: SupCon CE CaSSLe PNR CLS-ER VI (view-invariant only)
Builds on: orthogonal projection loss Barlow Twins CaSSLe PNR DyTox
Closest papers by meaning
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.87)
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.86)
- MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data (NeurIPS 2026 · similarity 0.86)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.85)
- TRACE: Contrastive learning for multi-trial time series data in neuroscience (NeurIPS 2025 · similarity 0.85)
- InTAct: Interval-based Task Activation Consolidation for Continual Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.