Research map / Continual learning / Online and replay-based
TinySubNets: An Efficient and Low Capacity Continual Learning Strategy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TinySubNets, a forget-free continual learning method combining sparsity-level pruning, adaptive non-linear quantization with KL-divergence-based weight sharing, and replay, achieving high accuracy at much lower capacity than pruning baselines.
Problem
Architectural continual learning methods waste model sparsity and saturate capacity because each weight can hold only one task value and layers are pruned at a uniform rate, limiting how many tasks can be learned.
Approach
Combines per-layer pruning with different sparsity levels, adaptive non-linear quantization that splits a weight's bits into banks/prefixes assigned to tasks, and KL-divergence-driven weight sharing between similar tasks (with an optional replay buffer), plus a greedy post-training pruning step without retraining.
Key results
- TSN achieves 97.14% on p-MNIST, 77.27% on s-CIFAR100, and 80.10% on TinyImageNet, the best among compared methods in the paper's main table.
- TSN-wr uses 22.65% capacity vs 77.73% for WSN on p-MNIST, and 17.62% vs 78.6% (Ada-QPacknet) on s-CIFAR100 while staying close in accuracy.
- On Imagenet100, TSN reaches 77.86% accuracy with 4.94M parameters vs DyTox 75.54% (10.73M) and DER 75.36% (112.27M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TSN instead of AdaQPacknet | p-MNIST · p-MNIST | average accuracy: 97.1 -> 97.14 | Helped |
| Use TSN instead of AdaQPacknet | s-CIFAR100 · s-CIFAR100 | average accuracy: 74.1 -> 77.27 | Helped |
| Use TSN instead of AdaQPacknet | 5 datasets · 5 datasets | average accuracy: 94.1 -> 93.76 | Hurt |
| Use TSN instead of WSN | TinyImagenet · TinyImagenet | average accuracy: 71.96 -> 80.10 | Helped |
| Remove replay memory (ablation) | TSN · TinyImagenet | capacity: 93.6% -> 32.15% | Helped |
| Remove replay memory (ablation) | TSN · TinyImagenet | average accuracy: 80.10 -> 79.81 | Hurt |
| Use 3-bit rather than 5-bit weights (ablation) | TSN · TinyImagenet | accuracy: 77.19 -> 68.01 | Hurt |
| Use 4-bit rather than 5-bit weights (ablation) | TSN · CIFAR100 | accuracy: 74.82 -> 75.14 | Helped |
| Disable post-pruning (ablation) | TSN-wr · 5 datasets | capacity: 24.68% -> 30.62% | Hurt |
| Add replay memory (ablation) | TSN-wr · CIFAR100 | capacity: 17.62% -> 41.92% | Hurt |
| Reduce weight and activation quantization from 8-bit to 4-bit (ablation) | TSN · TinyImagenet | accuracy: 79.66 -> 75.84 | Hurt |
| Reduce weight and activation quantization from 8-bit to 4-bit (ablation) | TSN · p-MNIST | accuracy: 97.06 -> 84.08 | Hurt |
Surprising findings
- Ada-QPacknet, which lacks weight sharing, achieves the best accuracy on the heterogeneous 5 datasets scenario (94.1%), outperforming TSN's replay variant by 2.3%.
- Table of capacity for WSN and Packnet exceeds 100% on some scenarios in the reported capacity table (e.g. Packnet 188.67% on TinyImageNet), which the paper does not explain.
Limitations
- Performance may degrade when tasks are highly dissimilar, as adaptive pruning may struggle to find shareable weights and memory saturates.
- Behaviour in very long scenarios with hundreds of tasks is not evaluated; quantization may be the main memory reducer there.
- Forward transfer is hard to evaluate in the task-incremental setting, and the method requires task identifiers.
Experimental setup
Datasets: Permuted MNIST split CIFAR100 5 datasets (MNIST, SVHN, FashionMNIST, CIFAR10, notMNIST) TinyImageNet Imagenet100
Benchmarks: Permuted MNIST split CIFAR100 5 datasets TinyImageNet Imagenet100
Metrics: average lifelong accuracy capacity (memory occupied per task) FLOPs backward transfer forward transfer
Baselines: Naive CWRStar SI Replay Cumulative Packnet WSN Ada-QPacknet DyTox DER NPCL QDI
Builds on: PackNet WSN Ada-QPacknet
Closest papers by meaning
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.88)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
- TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree (ICML 2025 · similarity 0.87)
- A Probabilistic Framework for Modular Continual Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.