Research map / Continual learning / Online and replay-based
Overcoming Catastrophic Forgetting with Hard Attention to the Task
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-conditioned hard attention masks and cumulative masks to constrain updates | HAT · Task Mixture, after 2 tasks | forgetting ratio: -0.08 -> -0.02 | Helped |
| Use task-conditioned hard attention masks and cumulative masks to constrain updates | HAT · Task Mixture, after 8 tasks | forgetting ratio: -0.11 -> -0.06 | Helped |
| Use task-conditioned hard attention masks and cumulative masks to constrain updates | HAT · Incremental CIFAR, after 10 tasks | forgetting ratio: -0.18 -> -0.09 | Helped |
| Use task-conditioned hard attention | Permuted MNIST, 10 tasks · Permuted MNIST, 10 tasks | accuracy: 97.1% -> 98.6% | Helped |
| Use task-conditioned hard attention | Split MNIST · Split MNIST, after 2 tasks | accuracy: 94.9% -> 99.0% | Helped |
| Use an adaptive optimizer for task embeddings instead of the constant-factor SGD boost (ablation) | HAT embedding learning · Sequential image classification | performance | Hurt |
| Use the first tangent-based annealing schedule instead of the proposed annealing (ablation) | HAT · Sequential image classification | performance | Hurt |
| Use the parametric tangent schedule with α=1 and β<1 instead of the proposed annealing (ablation) | HAT · Sequential image classification | performance | Hurt |
| Use alternative embedding initialization U(0,k1) or N(k2,1) instead of N(0,1) (ablation) | HAT · Sequential image classification | performance | Hurt |
| Replace attention-weighted L1 regularization with normalized L1 regularization (ablation) | HAT · Sequential image classification | accuracy | Hurt |
| Replace attention-weighted L1 regularization with L2 regularization (ablation) | HAT · Sequential image classification | accuracy | No clear effect |
| Add dropout to the split-MNIST configuration (ablation) | HAT · Split MNIST | accuracy | Helped |
Closest papers by meaning
- Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks (ICML 2025 · similarity 0.88)
- ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think (ICML 2025 · similarity 0.87)
- When to Learn and When to Stop: Quitting at the Optimal Time (Student Abstract) (AAAI 2025 · similarity 0.87)
- Kolmogorov-Arnold Networks Still Catastrophically Forget but Differently from MLP (AAAI 2025 · similarity 0.87)
- The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting — An Analytical Model (ICLR 2024 · similarity 0.87)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.