Research map / Continual learning / Online and replay-based
Task-Free Dynamic Sparse Vision Transformer for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TFDSViT, a task-free continual learning Vision Transformer that grows sparse experts via an energy-based (autoencoder reconstruction error) expansion criterion and a dynamic dual attention (sparse attention plus knowledge transfer attention).
Problem
Continual learning when task boundaries are unavailable during training and testing (task-free continual learning), where ViT-based continual learners typically require task information and either have static architectures or grow unboundedly.
Approach
A dynamic-expansion ViT whose experts consist of self-attention blocks reused by multiple submodels; sparse binary masks freeze some attention blocks to prevent forgetting, a Gumbel-softmax knowledge transfer attention reuses all previous experts' blocks, and autoencoder reconstruction energy of incoming batches decides when to add submodels or experts.
Key results
- On Split CIFAR100, TFDSViT reaches 32.86 ± 0.56 accuracy versus 29.87 ± 0.69 for Dynamic-OCM over five runs.
- On Split MNIST, TFDSViT achieves 98.12 ± 0.18 versus 95.67 ± 0.22 for Dynamic-OCM.
- On Split MiniImageNet (20 runs), TFDSViT achieves 35.62 ± 1.9 versus 28.03 ± 2.1 for Dynamic-OCM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TFDSViT instead of Dynamic-OCM | Task-free continual learning · Split MNIST | accuracy: 95.67 ± 0.22 -> 98.12 ± 0.18 | Helped |
| Use TFDSViT instead of Dynamic-OCM | Task-free continual learning · Split CIFAR10 | accuracy: 51.27 ± 1.47 -> 55.46 ± 1.02 | Helped |
| Use TFDSViT instead of Dynamic-OCM | Task-free continual learning · Split CIFAR100 | accuracy: 29.87 ± 0.69 -> 32.86 ± 0.56 | Helped |
| Use TFDSViT instead of Dynamic-OCM | Task-free continual learning · Split MiniImageNet | accuracy: 28.03 ± 2.1 -> 35.62 ± 1.9 | Helped |
| Use TFDSViT instead of MIR+GMED | Task-free continual learning with fuzzy task boundaries · Fuzzy Split MNIST | accuracy: 86.17 ± 1.7 -> 93.62 ± 1.3 | Helped |
| Use TFDSViT instead of MIR+GMED | Task-free continual learning with fuzzy task boundaries · Fuzzy Split CIFAR10 | accuracy: 41.22 ± 1.1 -> 49.23 ± 1.2 | Helped |
| Use TFDSViT instead of ER + GMED | Task-free continual learning with fuzzy task boundaries · Fuzzy Split MiniImageNet | accuracy: 28.20 ± 0.6 -> 32.62 ± 0.5 | Helped |
| Decrease the expansion threshold γ (ablation) | TFDSViT · Split CIFAR10 | accuracy | No clear effect |
Surprising findings
- Using more experts does not bring significant performance improvement; three experts can handle ten categories on Split MNIST with good accuracy.
Limitations
- Fixed memory buffer with simple earliest-removed replay policy.
- Expansion threshold gamma must be tuned to balance model size and performance.
- Uses a CNN feature extractor for MiniImageNet rather than a pure ViT there.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 MiniImageNet
Benchmarks: Split MNIST Split CIFAR10 Split CIFAR100 Split MiniImageNet
Metrics: average classification accuracy
Baselines: finetune GEM iCaRL reservoir MIR GSS CoPE ER + GMED ERa + GMED WGF-SVGD CURL CNDPM
Builds on: Vision Transformer DyTox OCM (Online Cooperative Memorization) CNDPM Gumbel-softmax
Closest papers by meaning
- Continual Learning With Lifelong Vision Transformer (CVPR 2022 · similarity 0.90)
- Task-Free Continual Generation and Representation Learning via Dynamic Expansionable Memory Cluster (AAAI 2024 · similarity 0.87)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.86)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.86)
- Visual Prompt Tuning in Null Space for Continual Learning (NeurIPS 2024 · similarity 0.86)
- Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.