Research map / Continual learning / Online and replay-based
Cut out and Replay: A Simple yet Versatile Strategy for Multi-Label Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CUTER, a replay strategy for multi-label online continual learning that uses MCut-based object cutting from a ViT backbone, a class-balanced buffer, and a low-rank adjacency regularizer to replay single-label object regions.
Problem
Multi-label online continual learning (MOCL) suffers from catastrophic forgetting, missing labels, and class imbalance in endless multi-label streams; existing methods ignore label-specific region identification and feature learning.
Approach
Select pre-trained models by the average Fiedler value of patch-similarity graphs, use MCut to cut out foreground object regions, keep only confidently single-label crops in a class-balanced replay buffer, and add a nuclear-norm regularizer on the patch adjacency matrix to preserve localization during continual learning.
Key results
- On PASCAL VOC, CUTER w/ R_l reaches 82.07 average mAP and 67.89 last mAP, versus 76.24 average mAP for the strongest baseline APPLE.
- On MS-COCO, CUTER w/ R_l achieves 60.14 average mAP and 47.82 last mAP, versus 56.45 and 40.56 for AGCN and OCDM.
- On NUS-WIDE, CUTER w/ R_l achieves 51.14 average mAP and 37.57 last mAP, outperforming AGCN (49.16 average mAP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add label-specific cut-and-replay to experience replay (ablation) | RS · PASCAL VOC 2007 | Avg mAP: 75.05±1.28 -> 77.92±0.78 | Helped |
| add label-specific cut-and-replay to experience replay (ablation) | RS · MS-COCO 2014 | Avg mAP: 48.12±1.78 -> 53.40±0.82 | Helped |
| add label-specific cut-and-replay to experience replay (ablation) | RS · NUS-WIDE | Avg mAP: 43.90±1.08 -> 46.30±1.02 | Helped |
| add low-rank regularization (ablation) | Cut.Rep · PASCAL VOC 2007 | Avg mAP: 77.92±0.78 -> 78.24±0.93 | Helped |
| add low-rank regularization (ablation) | Cut.Rep · MS-COCO 2014 | Avg mAP: 53.40±0.82 -> 56.37±0.64 | Helped |
| add low-rank regularization (ablation) | Cut.Rep · NUS-WIDE | Avg mAP: 46.30±1.02 -> 48.12±0.85 | Helped |
| continuously update the model instead of using a fixed localization backbone (ablation) | CUTER · PASCAL VOC 2007 | Avg mAP: 78.62±0.77 -> 79.45±0.92 | Helped |
| continuously update the model instead of using a fixed localization backbone (ablation) | CUTER · MS-COCO 2014 | Avg mAP: 59.01±0.45 -> 59.23±0.72 | Helped |
| continuously update the model instead of using a fixed localization backbone (ablation) | CUTER · NUS-WIDE | Avg mAP: 49.67±0.72 -> 50.51±0.64 | Helped |
| replace low-rank regularization with sparse regularization (ablation) | CUTER · PASCAL VOC 2007 | Avg mAP: 82.07±0.53 -> 78.34±0.76 | Hurt |
| replace low-rank regularization with sparse regularization (ablation) | CUTER · MS-COCO 2014 | Avg mAP: 60.14±0.60 -> 58.68±1.03 | Hurt |
| replace low-rank regularization with smooth regularization (ablation) | CUTER · MS-COCO 2014 | Avg mAP: 60.14±0.60 -> 58.31±0.68 | Hurt |
Surprising findings
- Sparse and smooth regularization on the adjacency matrix often degrade performance, while low-rank regularization consistently helps.
- Reconstruction-based pre-training (MAE) and iBOT show weaker zero-shot localization than contrastive multi-crop methods such as DINO.
Limitations
- Relies on ViT patch structure; performance degrades with CNN backbones such as ResNet50.
- Cut-out operations add computational overhead, reducing throughput.
- Some baselines (AGCN++, OCDM) were reproduced by the authors since code was unavailable.
Experimental setup
Datasets: PASCAL VOC 2007 MS-COCO 2014 NUS-WIDE
Benchmarks: PASCAL VOC 2007 MS-COCO 2014 NUS-WIDE
Metrics: mAP CF1 OF1 average mAP last mAP
Baselines: RS GSS iCarl NsCE KRT APPLE PRS OCDM AGCN AGCN++
Builds on: Asymmetric loss Partitioning Reservoir Sampling Optimizing Class Distribution in Memory Mask Cut AGCN
Closest papers by meaning
- Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (ICLR 2024 · similarity 0.89)
- Towards Macro-AUC Oriented Imbalanced Multi-Label Continual Learning (AAAI 2025 · similarity 0.87)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.87)
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.87)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.