Research map / Continual learning / Class-incremental
eTag: Class-Incremental Learning via Embedding Distillation and Task-Oriented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes eTag, a data-free class-incremental learning method combining block-wise embedding distillation for the feature extractor with a VAE feature generator trained under classifier guidance (task-oriented generation).
Problem
Class-incremental learning suffers catastrophic forgetting; existing strong methods store exemplars or prototypes, which conflicts with data privacy and limits fidelity of the incremental feature distribution.
Approach
Distills outputs of intermediate blocks (with rotation-based self-supervised auxiliary classifiers) from the frozen extractor to the new one, and trains a conditional VAE feature generator guided by the frozen classifier so generated features fit the classifier's needs, requiring no exemplars or prototypes.
Key results
- On CIFAR-100 and ImageNet-100, eTag improves average incremental accuracy by more than 5% and 7% over GFR respectively (e.g., 61.63% on CIFAR-100 with 10 tasks... vs GFR 56.76% on 10 tasks in Table 1 as reported for 10 tasks).
- On CIFAR-100 with 5 tasks, eTag reaches 67.99% average incremental accuracy versus 62.74% for GFR.
- Naively adding embedding distillation to other methods (eGFR, eLwF, eEWC) does not consistently help, e.g., eGFR's ACCs drop on 25 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-oriented feature generation instead of naive feature generation (ablation) | eTag · CIFAR-100, 4 equally divided tasks | A: 57.30 -> 59.82 | Helped |
| Use embedding KD instead of naive KD and add the SS task (ablation) | eTag · CIFAR-100, 4 equally divided tasks | A: 57.30 -> 61.18 | Helped |
| Add the SS task (ablation) | eTag with task-oriented generation and embedding KD · CIFAR-100, 4 equally divided tasks | A: 61.18 -> 64.10 | Helped |
| Use task-oriented generation and embedding KD instead of naive generation and embedding KD (ablation) | eTag · CIFAR-100, 4 equally divided tasks | A: 56.89 -> 58.38 | No clear effect |
| Use eTag instead of GFR | CIL with ResNet-18 · CIFAR-100, 10 tasks | average incremental accuracy (A): 59.85 -> 65.50 | Helped |
| Use eTag instead of GFR | CIL with ResNet-18 · ImageNet-100, 10 tasks | average incremental accuracy (A): 67.30 -> 75.17 | Helped |
| Introduce embedding distillation into GFR | GFR · CIFAR-100, 10 tasks | average incremental accuracy (A): 59.85 -> 64.62 | Helped |
| Introduce embedding distillation into GFR | GFR · CIFAR-100, 10 tasks | forgetting measure (F): 21.74 -> 23.95 | Hurt |
| Introduce embedding distillation into GFR | GFR · CIFAR-100, 25 tasks | ACC | Hurt |
Surprising findings
- Directly incorporating embedding distillation into existing SOTA methods does not consistently improve incremental performance and can increase forgetting (e.g., eGFR forgetting rises from 21.74% to 23.95% on CIFAR-100 10 tasks).
Limitations
- Performance remains well below the joint-training upper bound.
- Training efficiency is a concern; the authors plan to improve it.
- Does not yet leverage large pre-trained backbones.
Experimental setup
Datasets: CIFAR-100 ImageNet-100
Benchmarks: CIFAR-100 ImageNet-100
Metrics: average incremental accuracy average forgetting measure classification accuracy
Baselines: Fine-tuning Joint training EWC MAS LwF LwM IL2M Lucir PASS GFR
Builds on: Generative Feature Replay (GFR) PASS PodNet Lucir Deep Generative Replay
Closest papers by meaning
- Class Incremental Learning via Likelihood Ratio Based Task Prediction (ICLR 2024 · similarity 0.89)
- Compress to One Point: Neural Collapse for Pre-Trained Model-Based Class-Incremental Learning (AAAI 2025 · similarity 0.88)
- Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning (ICML 2025 · similarity 0.88)
- Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning (NeurIPS 2024 · similarity 0.88)
- Gradient Reweighting: Towards Imbalanced Class-Incremental Learning (CVPR 2024 · similarity 0.88)
- Long-Tail Class Incremental Learning via Independent Sub-prototype Construction (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.