Research map / Continual learning / Online and replay-based
Adaptive Shortcut Debiasing for Online Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DropTop, a model-agnostic add-on for replay-based online continual learning that drops top-attention features from a fused first/last-layer attention map, with the drop intensity adapted by a loss-reduction t-test, to suppress shortcut bias.
Problem
Shortcut bias in online continual learning hurts transferability and increases forgetting, but existing debiasing methods need prior knowledge of the bias or auxiliary OOD data, which are unavailable in OCL.
Approach
Identify shortcut features as highly-activated features by fusing first- and last-layer attention maps, drop the top-kappa% of them, and adaptively shift the per-class drop intensity kappa by comparing loss reductions of increment vs. decrement options with a t-test, all without prior knowledge or auxiliary data.
Key results
- DropTop raises ER average accuracy by 10.2% on Split CIFAR-10 and DER++ forgetting falls by 16.0% on the same dataset, with up to 10.4% accuracy and 63.2% forgetting relative improvements overall.
- On pretrained ViT, DropTop with DualPrompt reduces forgetting by 63.2% relative on Split ImageNet-9 (3.7 vs 10.1 range reported via relative improvement).
- Ablation: removing multi-level feature fusion lowers average accuracy by 4.9% on average; replacing adaptive intensity shifting with fixed or random dropping lowers it by 2.1% and 4.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DropTop to ER | ER · Split CIFAR-100 | average accuracy: 25.3 -> 26.5 | Helped |
| Add DropTop to ER | ER · Split CIFAR-10 | average accuracy: 55.4 -> 61.0 | Helped |
| Add DropTop to ER | ER · Split ImageNet-9 | average accuracy: 46.1 -> 48.9 | Helped |
| Add DropTop to ER | ER · Split ImageNet-OnlyFG | forgetting: 45.0 -> 47.8 | Hurt |
| Add DropTop to ER | ER · Split ImageNet-Stylized | forgetting: 49.7 -> 38.9 | Helped |
| Add multi-level feature fusion using low- and high-level features (ablation) | DropTop · Biased datasets average | average accuracy: 43.8 -> 45.5 | Helped |
| Replace attention-based feature dropping with random dropping (ablation) | DropTop · Biased datasets average | average accuracy: 45.5 -> 43.9 | Hurt |
| Use adaptive rather than fixed attention-based drop intensity (ablation) | DropTop · Biased datasets average | average accuracy: 44.1 -> 45.5 | Helped |
| Replace the hard drop mask with a soft drop mask (ablation) | ER + DropTop · Split CIFAR-10 | average accuracy: 61.0 -> 60.3 ($\pm$0.4) | Hurt |
| Replace class-wise drop intensities with one common intensity (ablation) | ER + DropTop · Split CIFAR-10 | average accuracy: 61.0 -> 57.6 ($\pm$0.1) | Hurt |
Surprising findings
- Fixed top-attention dropping (no adaptive intensity shifting) outperforms random dropping, showing gains are not merely from regularization.
- On Split ImageNet-OnlyFG, DropTop lowers MIR-like ASER/GSS gains are small, and ER+DropTop is 6.2% worse than ER on the MIR-style OnlyFG ablation case reported for one method (ER+DropTop on OnlyFG accuracy drop of -6.2%).
Limitations
- Relies on the assumption that shortcut features are highly activated early in training; the attention-based identification is heuristic.
- Some methods show no gain or slight degradation on certain datasets (e.g., GSS forgetting on Split CIFAR-100, MIR accuracy on ImageNet-OnlyFG).
Experimental setup
Datasets: Split CIFAR-10 Split CIFAR-100 Split ImageNet-9 Split ImageNet-OnlyFG Split ImageNet-Stylized Biased MNIST MNIST
Benchmarks: Split CIFAR-10 Split CIFAR-100 Split ImageNet-9 Split ImageNet-OnlyFG Split ImageNet-Stylized
Metrics: average accuracy forgetting
Baselines: ER DER++ MIR GSS ASER L2P DualPrompt Rand Drop Fixed Drop No MF
Builds on: ER DER++ MIR GSS ASER L2P DualPrompt
Closest papers by meaning
- Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning (NeurIPS 2024 · similarity 0.87)
- Adaptive Retention & Correction: Test-Time Training for Continual Learning (ICLR 2025 · similarity 0.85)
- LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · similarity 0.85)
- Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization (CVPR 2025 · similarity 0.85)
- RanDumb: Random Representations Outperform Online Continually Learned Representations (NeurIPS 2024 · similarity 0.85)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.