Research map / Continual learning / Online and replay-based
Attention Retention for Continual Learning with Vision Transformers
Yue Lu, Xiangyu Zhou, Shizhou Zhang, Yinghui Xing, Guoqiang Liang, Wencong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive thresholding instead of fixed top-20% masks (raw attention) (ablation) | ARCL-ViT · 10S-ImageNet-R | accuracy: 77.63 -> 80.33 | Improved |
| add adaptive thresholding instead of fixed top-20% masks (layer-wise rollout) (ablation) | ARCL-ViT · 10S-ImageNet-R | accuracy: 76.72 -> 81.17 | Improved |
| add adaptive thresholding instead of fixed (layer-wise rollout) (ablation) | ARCL-ViT · 10S-DomainNet | forgetting: 21.61 -> 7.04 | Improved |
| use layer-wise rollout instead of raw attention (ablation) | ARCL-ViT · 10S-ImageNet-R | accuracy: 80.33 -> 81.17 | Improved |
| use layer-wise rollout instead of naive rollout (ablation) | ARCL-ViT · 10S-DomainNet | accuracy: 82.28 -> 83.94 | Improved |
| mask previous attention regions instead of random positions (ablation) | ARCL-ViT · 10S-ImageNet-R | accuracy: 73.27 -> 81.17 | Improved |
| mask previous attention regions instead of non-attention regions (ablation) | ARCL-ViT · 10S-DomainNet | accuracy: 76.08 -> 83.94 | Improved |
| mask random regions instead of attention regions (ablation) | ARCL-ViT · 10S-ImageNet-R | forgetting: 4.22 vs 4.84 | No clear effect |
| apply gradient masking of attention regions to sequential fine-tuning | Seq-FT · 10S-ImageNet-R | accuracy: 49.43 -> 81.17 | Improved |
| use ARCL-ViT instead of VPT-CPG with DINO-1k weights | ARCL-ViT · 10S-ImageNet-R (DINO-1k) | accuracy: 68.32 -> 72.37 | Improved |
| use ARCL-ViT instead of VPT-CPG with iBOT-1k weights | ARCL-ViT · 10S-DomainNet (iBOT-1k) | accuracy: 77.79 -> 80.88 | Improved |
| use ARCL-ViT instead of VPT-CPG on 100-task ImageNet-R | ARCL-ViT · 100S-ImageNet-R | accuracy: 64.63 -> 70.72 | Improved |
Closest papers by meaning
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.91)
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.90)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.89)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.89)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.89)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
