Research map / Continual learning / Online and replay-based
Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ferret, an online continual learning framework that combines fine-grained asynchronous pipeline parallelism with iterative gradient compensation and a bi-level memory-constrained partitioning/planning search.
Problem
Online continual learning on high-frequency data streams needs low latency and high online accuracy, but operating under varying memory budgets is hard, and asynchronous pipeline parallelism introduces stale gradients.
Approach
Apply asynchronous 1F1B pipeline parallelism with multiple interleaved workers to process each arriving stream item immediately, compensate stale gradients iteratively via a Fisher-based first-order Taylor approximation with an automatically optimized coefficient, and choose model partition and pipeline configuration by bi-level optimization under a memory budget.
Key results
- Ferret achieves up to 3.7x lower memory overhead to reach the same online accuracy compared to competing methods (abstract).
- FerretM+ achieves online accuracy gain comparable to the Oracle on many settings, e.g., CORe50/ConvNet agm 24.49 vs Oracle 26.01 (Table 1).
- Iter-Fisher gradient compensation consistently improves online accuracy across settings, while Step-Aware and Gap-Aware reduce it substantially (Table 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FerretM+ rather than 1-Skip | OCL framework · MNIST/MNISTNet | agm: 0±0 -> 26.34±0.7 | Helped |
| Use FerretM+ rather than 1-Skip | OCL framework · CIFAR10/ConvNet | agm: 0±0 -> 9.44±0.12 | Helped |
| Use FerretM+ rather than 1-Skip | OCL framework · CORe50/ConvNet | agm: 0±0 -> 24.49±0.43 | Helped |
| Use minimal-memory FerretM→ rather than 1-Skip | OCL framework · CIFAR100/ConvNet | agm: 0±0 -> 1.58±0.04 | Helped |
| Use FerretM+ rather than Oracle | OCL framework · MNIST/MNISTNet | agm: 27.32±0.71 -> 26.34±0.7 | Hurt |
| Use FerretM+ rather than Oracle | OCL framework · CIFAR100/ConvNet | agm: 5.24±0.01 -> 4.39±0.05 | Hurt |
| Use FerretM+ rather than Oracle | OCL framework · CORe50/ConvNet | agm: 26.01±0.42 -> 24.49±0.43 | Hurt |
| Replace Iter-Fisher with Step-Aware compensation (ablation) | FerretM+ · evaluated settings in Table 4 | online accuracy difference: -56.04±2.78 | Hurt |
| Replace Iter-Fisher with Gap-Aware compensation (ablation) | FerretM+ · evaluated settings in Table 4 | online accuracy difference: -14.03±1.24 | Hurt |
| Replace Iter-Fisher with Fisher compensation (ablation) | FerretM+ · evaluated settings in Table 4 | online accuracy difference: -0.02±0.01 | Mixed |
| Add Iter-Fisher gradient compensation (ablation) | Ferret pipeline parallelism · evaluated settings in Table 4 | online accuracy difference: 0.01±0 | Helped |
Surprising findings
- Asynchronous pipeline parallelism strategies significantly outperform synchronous ones, even Zero-Bubble, which claims to eliminate pipeline bubbles.
- Plain Fisher compensation yields only marginal accuracy decrease relative to no compensation, while Step-Aware and Gap-Aware compensation significantly reduce online accuracy.
Limitations
- Gradient compensation via Fisher approximation can still be unhelpful for some configurations (Fisher gives marginal decrease).
- Experiments limited to image classification datasets and small-to-medium models (MLP, ConvNet, ResNet-18, MobileNet).
Experimental setup
Datasets: MNIST FMNIST CIFAR10 CIFAR100 SVHN Tiny-ImageNet CORe50 Split-MNIST Split-FMNIST Split-CIFAR10 Split-CIFAR100 Split-SVHN
Benchmarks: MNIST FMNIST EMNIST CIFAR10 CIFAR100 SVHN Tiny-ImageNet CORe50 CLEAR10 CLEAR100 Covertype
Metrics: online accuracy gain per unit of memory test accuracy gain per unit of memory
Baselines: Oracle 1-Skip Random-N B-Skip Last-N B-Skip Camel DAPPLE Pipedream Pipedream2BW Zero-Bubble Hanayo ER MIR
Builds on: 1F1B pipeline scheduling (PipeDream) Fisher information gradient compensation Camel Pipedream2BW experience replay (ER)
Closest papers by meaning
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.88)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.87)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.87)
- Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning (ICML 2025 · similarity 0.86)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.86)
- Active Learning for Continual Learning: Keeping the Past Alive in the Present (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.