Research map / Continual learning / Online and replay-based
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the "CL on Diet" setting (per-step compute budget plus sparse labels in a streaming setting) and DietCL, a baseline that jointly trains MAE reconstruction on unlabeled data, masked classification on current labeled data, and a balanced buffer loss.
Problem
Continual learning on large streams where each time step has a restricted computational budget and only a very small fraction of samples are labeled, causing supervised CL methods to overfit and semi-supervised CL methods to underuse compute.
Approach
Allocate a per-step compute budget across labeled current data, unlabeled current data (via MAE masked reconstruction, chosen over contrastive SSL for its single-view efficiency), and a task-balanced replay buffer, then spend leftover budget only on buffer data to balance old and new classes.
Key results
- On 20-split ImageNet10k (1% labels, 500 steps), DietCL reaches A(T)=16.82 and average accuracy 24.90, versus 14.89 / 22.84 for Replay and 5.78 / 13.25 for CaSSLe.
- On CLOC, DietCL average accuracy 4.98 vs 4.59 for Replay; on CGLM, 20.26 vs 19.40 for Replay (A(T) 24.34).
- Ablation on ImageNet10k: adding masked classification loss gives +0.9 average accuracy, balanced buffer +0.13, and MAE reconstruction loss +0.86.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · ImageNet10k | last-step accuracy: 14.89 -> 16.82 | Helped |
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · ImageNet10k | average accuracy: 22.84 -> 24.90 | Helped |
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · CLOC | last-step accuracy: 4.64 -> 5.63 | Helped |
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · CLOC | average accuracy: 4.59 -> 4.98 | Helped |
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · CGLM | last-step accuracy: 22.81 -> 24.34 | Helped |
| Use DietCL instead of Replay | Continual learning on sparsely labeled streams · CGLM | average accuracy: 19.40 -> 20.26 | Helped |
| Add masked classification loss (ablation) | Replay on ImageNet10k with 1% labels and 500 steps · ImageNet10k | average accuracy: 14.97 -> 15.83 | Helped |
| Add a task-balanced buffer (ablation) | Replay with masked classification loss · ImageNet10k | average accuracy: 15.83 -> 15.96 | Helped |
| Add unlabeled-data reconstruction loss (ablation) | Replay with masked classification loss and balanced buffer · ImageNet10k | average accuracy: 15.96 -> 16.82 | Helped |
| Add a task-balanced buffer (ablation) | Replay · ImageNet10k | average accuracy: 14.97 -> 15.01 | Helped |
| Add unlabeled reconstruction loss without masked classification loss (ablation) | Replay with balanced buffer · ImageNet10k | average accuracy: 15.01 -> 15.24 | Helped |
Surprising findings
- Contrastive SSL methods like CaSSLe collapse under the small per-step budget, achieving only 5.78% last-step accuracy on ImageNet10k despite being competitive with full training budgets.
- Increasing the label rate from 5% to 10% did not improve DietCL when the computational budget was insufficient for later steps.
Limitations
- Balanced fine-tuning budget threshold chosen by cross-validation on the first few tasks, adding a tuned hyperparameter.
- Several baselines (DualNet, CCIC, CaSSLe) were re-implemented under the budget constraint and performed poorly, which may reflect implementation choices rather than intrinsic limits.
- Real-world cost analysis (Snap example) is back-of-envelope and depends on GPU type and data-loading speed.
Experimental setup
Datasets: ImageNet10k (constructed from ImageNet21k V2 minus ImageNet1k classes) CLOC CGLM ImageNet1k
Benchmarks: ImageNet10k CLOC CGLM
Metrics: last-step accuracy A(T) average accuracy A over stream
Baselines: ER ER-ACE EWC MAS GDumb L2P CaSSLe DualNet CCIC
Builds on: MAE CaSSLe ER GDumb L2P
Closest papers by meaning
- Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning (AAAI 2024 · similarity 0.90)
- Cut out and Replay: A Simple yet Versatile Strategy for Multi-Label Online Continual Learning (ICML 2025 · similarity 0.89)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.87)
- Persistence Homology Distillation for Semi-supervised Continual Learning (NeurIPS 2024 · similarity 0.87)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.87)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.