Research map / Continual learning / Online and replay-based
Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orthogonal Sparse Network (OSN), which splits the network into three parts (old-task, shared, free) and trains shared parameters with a sharpness-aware orthogonal projection, achieving near-zero forgetting and higher plasticity in task-incremental learning.
Problem
Parameter-pruning continual learning methods (e.g., WSN) freeze old-task sub-networks, which wastes capacity and limits plasticity, especially in long-term task-incremental learning with many tasks.
Approach
Partition the network into three parts by adding a shared-parameter part selected from the old-task sub-network by importance to the new task; update these shared parameters via a sharpness-aware orthogonal projection onto the complement of old-task feature space, while free parameters are updated unconstrained.
Key results
- OSN reaches 97.19% ACC on PMNIST and 79.78% on Split CIFAR-100, versus 75.86% for WSN, with BWT near 0.
- On Split TinyImageNet (40 tasks), OSN achieves 75.42% ACC, 3.46% above the second-best WSN.
- OSN lowers capacity metric CAP to 25.09% on Split CIFAR-100 versus 41.88% for WSN (as reported in Table 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OSN instead of WSN | Continual learning model · PMNIST | ACC(%): 96.51 -> 97.19 ± 0.01 | Helped |
| use OSN instead of Connector | Continual learning model · Split CIFAR-100 | ACC(%): 78.10 -> 79.78 ± 0.06 | Helped |
| use OSN instead of SupSup | Continual learning model · CIFAR-100 Superclass | ACC(%): 61.70 -> 63.81 ± 0.06 | Helped |
| use OSN instead of SupSup | Continual learning model · 5-Datasets | ACC(%): 93.28 -> 93.32 ± 0.09 | Helped |
| use OSN instead of WSN | Continual learning model · Split TinyImageNet | ACC(%): 71.96 -> 75.42 ± 0.38 | Helped |
| add network partitioning to pruning (ablation) | OSN · Split CIFAR-100 | ACC(%): 75.86 -> 77.12 | Helped |
| add network partitioning to pruning (ablation) | OSN · CIFAR-100 Superclass | ACC(%): 61.34 -> 62.13 | Helped |
| add orthogonal projection to pruning with network partitioning (ablation) | OSN · Split CIFAR-100 | ACC(%): 77.12 -> 78.23 | Helped |
| add orthogonal projection to pruning with network partitioning (ablation) | OSN · CIFAR-100 Superclass | ACC(%): 62.13 -> 63.21 | Helped |
| add sharpness-aware orthogonal projection (ablation) | OSN with pruning, network partitioning, and OP · Split CIFAR-100 | ACC(%): 78.23 -> 79.78 | Helped |
| add sharpness-aware orthogonal projection (ablation) | OSN with pruning, network partitioning, and OP · CIFAR-100 Superclass | ACC(%): 63.21 -> 63.81 | Helped |
Surprising findings
- Selecting more shared parameters can cause a small amount of forgetting (small negative BWT), indicating a stability-plasticity trade-off in the shared part.
Limitations
- Evaluated only in task-incremental learning; class-incremental setting left for future work.
- Relies on several hyperparameters (sparsity c, partition percentage k, threshold epsilon, rho, gamma) that trade off stability and plasticity.
- Small backbones (MLP, AlexNet, LeNet, ResNet-18) on standard benchmarks only.
Experimental setup
Datasets: PMNIST Split CIFAR-100 CIFAR-100 Superclass 5-Datasets Split TinyImageNet CIFAR-100-50
Benchmarks: PMNIST Split CIFAR-100 CIFAR-100 Superclass 5-Datasets Split TinyImageNet CIFAR-100-50
Metrics: ACC BWT CAP diagonal accuracy
Baselines: EWC PackNet SupSup La-MaML FS-DGPM GPM TRGP WSN Connector DualGPM API DFGP
Builds on: Winning Subnetworks (WSN) Gradient Projection Memory (GPM) Lottery Ticket Hypothesis Sharpness-aware minimization
Closest papers by meaning
- Efficient Continual Learning with Modular Networks and Task-Driven Priors (ICLR 2021 · similarity 0.89)
- Learning Bayesian Sparse Networks With Full Experience Replay for Continual Learning (CVPR 2022 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.87)
- Learning without Isolation: Pathway Protection for Continual Learning (ICML 2025 · similarity 0.87)
- Task-Free Dynamic Sparse Vision Transformer for Continual Learning (AAAI 2024 · similarity 0.86)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.