Research map / Continual learning / Domain and prompt-based
MIND: Multi-Task Incremental Network Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIND, a rehearsal-free parameter isolation method that distills knowledge from a per-task teacher (or itself, via self-distillation) into sparse sub-networks of a shared gated ResNet, with task-specific BatchNorm, reaching state-of-the-art class-incremental accuracy on CIFAR100/10, TinyImageNet/10, Core50, and Synbols.
Problem
Class-incremental and domain-incremental continual learning without replay data, where catastrophic forgetting must be avoided under memory constraints.
Approach
Combines PackNet-style sub-network pruning with Jensen-Shannon distillation from a freshly trained teacher (or from the network itself in a memory-saving self-distillation variant), adds a binary gating mask on the backbone, and learns task-specific BatchNorm parameters per sub-network.
Key results
- On CIFAR-100/10, MIND reaches 39.9% task-agnostic accuracy versus 33.8% for PASS and 28.5% for PackNet (approx. +6% over the best memory-free baseline, per the text).
- On TinyImageNet/10, MIND reaches 35.0% task-agnostic accuracy, against 29.0% for PackNet.
- On Core50 domain-incremental, MIND achieves 79.28% versus 31.38% for LwF, more than doubling the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MIND instead of PackNet | CIFAR100/10 CI · CIFAR100/10 CI, task-agnostic | ACC_TAG: 28.5 $\pm$ 2.2 -> 39.9 $\pm$ 0.9 | Helped |
| Use MIND instead of PackNet | CIFAR100/10 CI · CIFAR100/10 CI, task-aware | ACC_TAW: 72.4 $\pm$ 1.4 -> 82.3 $\pm$ 0.6 | Helped |
| Use MIND instead of PackNet | TinyImageNet/10 CI · TinyImageNet/10 CI, task-agnostic | ACC_TAG: 29.0 $\pm$1.2 -> 35.0 $\pm$ 0.8 | Helped |
| Use MIND instead of PackNet | TinyImageNet/10 CI · TinyImageNet/10 CI, task-aware | ACC_TAW: 65.0 $\pm$ 0.9 -> 71.1 $\pm$ 0.7 | Helped |
| Use MIND instead of PackNet | Core50/10 CI · Core50/10 CI, task-agnostic | ACC_TAG: 40.0 $\pm$ 4.3 -> 57.9 $\pm$ 2.1 | Helped |
| Use MIND instead of PackNet | Core50/10 CI · Core50/10 CI, task-aware | ACC_TAW: 95.7 $\pm$ 1.2 -> 99.7 $\pm$ 0.08 | Helped |
| Use MIND instead of PackNet | Synbols/10 CI · Synbols/10 CI, task-agnostic | ACC_TAG: 60.4 $\pm$ 4.7 -> 76.5 $\pm$ 2.6 | Helped |
| Use MIND instead of PackNet | Synbols/10 CI · Synbols/10 CI, task-aware | ACC_TAW: 95.7 $\pm$ 1.1 -> 98.4 $\pm$ 0.2 | Helped |
| Use MIND instead of LwF | Core50 DI · Core50 DI | ACC_TAG: 31.38 $\pm$ 0.02 -> 79.28$\pm$2.63 | Helped |
| Use self-distillation instead of the standard new-model teacher | MIND on CIFAR100/10 CI · CIFAR100/10 CI, task-agnostic | ACC_TAG: 39.9 $\pm$ 0.9 -> 35.7 $\pm$ 0.7 | Hurt |
| Add weight sharing between sub-networks (ablation) | MIND on CIFAR100/10 CI · CIFAR100/10 CI | ACC_TAG: +2.9% | Helped |
| Add distillation loss (ablation) | MIND on CIFAR100/10 CI · CIFAR100/10 CI | ACC_TAG: +2.7% | Helped |
Surprising findings
- The self-distillation variant, which needs no extra teacher network, is nearly on par with standard MIND in task-aware accuracy (e.g., 82.2% vs 82.3% on CIFAR-100/10).
Limitations
- Self-distillation loses task-agnostic accuracy in the last tasks as the trainable parameters shrink.
- Hyperparameters were grid-searched per dataset, and competitor baselines were tuned with FACIL/Avalanche defaults.
- Experiments are limited to small ResNet backbones on small-resolution image benchmarks.
Experimental setup
Datasets: CIFAR-100 TinyImageNet Core50 Synbols
Benchmarks: CIFAR-100/10 TinyImageNet/10 Core50/10 (CI) Synbols/10 Core50 (DI)
Metrics: ACC_TAG ACC_TAW
Baselines: Finetuning LwF EWC SI MAS RWalk LwM PackNet PASS
Builds on: PackNet Learning without Forgetting FACIL Avalanche
Closest papers by meaning
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.86)
- Rethinking Momentum Knowledge Distillation in Online Continual Learning (ICML 2024 · similarity 0.86)
- Defying Imbalanced Forgetting in Class Incremental Learning (AAAI 2024 · similarity 0.86)
- Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning (ICML 2024 · similarity 0.86)
- GraphKeeper: Graph Domain-Incremental Learning via Knowledge Disentanglement and Preservation (NeurIPS 2025 · similarity 0.86)
- Autoencoder-Based Hybrid Replay for Class-Incremental Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.