Research map / Continual learning / Online and replay-based
Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a total-budget (FLOPs plus bytes of memory and model storage) evaluation for online continual learning, and aL-SAR, which combines batch-wise adaptive layer freezing via Fisher information with use-frequency and class-gradient-similarity-aware memory retrieval.
Problem
Online continual learning methods are compared under inconsistent computation and storage budgets: single-epoch training hides differing per-method FLOPs, and storage for logits or model copies is usually ignored when setting memory budgets.
Approach
Measure budgets as training FLOPs and total bytes (replay memory plus extra stored logits/models), then reduce per-iteration cost by adaptively freezing layers per batch using a Fisher-information-per-computation criterion, and improve sample efficiency with a retrieval rule that favors samples with low discounted, similarity-adjusted use frequency at no extra inference cost.
Key results
- On CIFAR-10 Gaussian with 7.6MB memory, aL-SAR reaches 64.38 A_AUC and 72.57 A_last at 146.80 TFLOPs versus 171.94 TFLOPs for no freezing.
- On ImageNet-1K Gaussian, aL-SAR achieves 33.24 A_AUC versus 27.65 for the next best (LiDER) under a shared total budget.
- On LLaVA-1.5-7B with LoRA on Bongard-HOI, aL reduces training TFLOPs by 10.3% (1578.02 to 1431.27) while keeping accuracy comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use aL-SAR instead of the strongest reported baseline | Online continual learning · ImageNet-1K Gaussian | A_AUC: 27.65 -> 33.24 | Helped |
| Use aL-SAR instead of ER | Online continual learning · CLEAR-10 | A_AUC: 64.00$\pm$0.06 -> 67.61$\pm$0.45 | Helped |
| Use aL-SAR instead of ER | Online continual learning · CLEAR-100 | A_AUC: 31.56$\pm$0.42 -> 38.65$\pm$0.30 | Helped |
| Add adaptive layer freezing (ablation) | Vanilla replay · CIFAR-10 Gaussian | TFLOPs: 163.74 -> 142.23 | Helped |
| Add adaptive layer freezing (ablation) | Vanilla replay · CIFAR-10 Gaussian | A_AUC: 60.76$\pm$0.11 -> 60.38$\pm$0.54 | Hurt |
| Add similarity-aware retrieval (ablation) | Vanilla replay · CIFAR-10 Gaussian | A_AUC: 60.76$\pm$0.11 -> 64.60$\pm$0.83 | Helped |
| Add similarity-aware retrieval (ablation) | Vanilla replay · CIFAR-100 Gaussian | A_AUC: 31.97$\pm$0.89 -> 37.60$\pm$0.40 | Helped |
| Use sample use-frequency without temporal discounting (ablation) | Vanilla random retrieval · CIFAR-10 continuous | A_AUC: 60.76$\pm$0.11 -> 56.15$\pm$0.43 | Hurt |
| Discount sample use-frequency over time (ablation) | Use-frequency retrieval · CIFAR-10 continuous | A_AUC: 56.15$\pm$0.43 -> 62.72$\pm$0.18 | Helped |
| Include class similarity in effective use-frequency (ablation) | Discounted-use-frequency retrieval · CIFAR-10 continuous | A_AUC: 62.72$\pm$0.18 -> 64.60$\pm$0.83 | Helped |
| Use sample use-frequency without temporal discounting (ablation) | Vanilla random retrieval · CIFAR-100 continuous | A_AUC: 31.97$\pm$0.89 -> 30.74$\pm$1.09 | Hurt |
| Discount sample use-frequency over time (ablation) | Use-frequency retrieval · CIFAR-100 continuous | A_AUC: 30.74$\pm$1.09 -> 33.76$\pm$0.67 | Helped |
Surprising findings
- Several high-performing online CL methods fall behind simple experience replay when compared under the same total FLOPs and memory budget.
- Adaptive freezing saves FLOPs without reducing accuracy in several settings, with BFC mostly negative on average, meaning freezing is rarely beneficial.
Limitations
- Hyperparameters tuned only on CIFAR-10 Gaussian and reused elsewhere.
- Fisher trace estimate requires periodic full-gradient updates (every 4 iterations), adding overhead.
- Freezing savings are modest; BFC is mostly negative so freezing is rarely beneficial.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CLEAR-10 CLEAR-100 ImageNet-1K Bongard-HOI Bongard-OpenWorld
Benchmarks: CIFAR-10 CIFAR-100 CLEAR-10 CLEAR-100 ImageNet-1K Bongard-HOI Bongard-OpenWorld
Metrics: A_AUC A_last TFLOPs memory budget (MB)
Baselines: ER DER++ MIR MEMO REMIND EWC OCS LiDER X-DER CCL-DC CAMA PTLF
Builds on: experience replay MIR ASER EGERIA REMIND
Closest papers by meaning
- Summarizing Stream Data for Memory-Constrained Online Continual Learning (AAAI 2024 · similarity 0.89)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.89)
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.88)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.88)
- Do Your Best and Get Enough Rest for Continual Learning (CVPR 2025 · similarity 0.88)
- Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.