Research map / Continual learning / Class-incremental
M2SD:Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2SD, a self-distillation framework for few-shot class-incremental learning that uses dual-branch mixup/CutMix virtual classes and attention-enhanced FPN features to shape a base-session feature space, then freezes the backbone and updates only the classifier.
Problem
Few-shot class-incremental learning (FSCIL): learning new classes from few examples while retaining accuracy on previously learned classes, with catastrophic forgetting.
Approach
Applies self-distillation to FSCIL: a dual-branch virtual-class mixing (mixup and CutMix) plus an attention-enhanced BiFPN feature module distills enriched features back to the base network, so the frozen feature space is more extensible for new classes; only the prototype classifier is updated incrementally.
Key results
- On miniImageNet, session 0 accuracy 82.11% and final session 8 accuracy 56.51%, versus SAVC's 81.02% and 54.59% (reported in Table 1).
- Compared to the baseline, M2SD reduces average intra-class distance by 27% (1.35 to 0.98) and increases inter-class distance by 22% (6.49 to 7.92).
- Ablation on CUB200 shows the self-distillation stage adds roughly 10% per-session accuracy on average over no distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M2SD instead of the strongest listed baseline, SAVC† | FSCIL on miniImageNet · miniImageNet session 8 | accuracy: 54.59 -> 56.51 | Helped |
| Use M2SD instead of the strongest listed baseline, SAVC† | FSCIL on miniImageNet · miniImageNet session 0 | accuracy: 81.02 -> 82.11 | Helped |
| Use M2SD instead of previous methods | FSCIL on CUB200 · CUB200, average across stages | accuracy: more than 2.0% | Helped |
| Use M2SD instead of previous methods | FSCIL on CIFAR100 · CIFAR100, average across stages | accuracy: more than 2.1% | Helped |
| Use M2SD instead of previous methods | FSCIL on miniImageNet · miniImageNet, average across stages | accuracy: more than 3.2% | Helped |
| Add base-session pretraining before the self-distillation stage (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 52.37 -> 60.96 | Helped |
| Use the 20%:80% stage1:stage2 ratio rather than 100%:0% (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 48.35 -> 60.96 | Helped |
| Add CutMix as the second virtual-class branch alongside mixup (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 60.19 -> 60.96 | Helped |
| Use CutMix alone rather than the mixup + CutMix branches (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 60.96 -> 52.37 | Hurt |
| Add the dual-branch virtual-class structure (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 59.84 -> 60.96 | Helped |
| Add feature enhancement (FE) (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 60.02 -> 60.96 | Helped |
| Use classifier updating rather than retaining the original classifier parameters (ablation) | M2SD on CUB200 · CUB200 session 10 | accuracy: 40.17 -> 60.96 | Helped |
Surprising findings
- CutMix-only virtual classes degrade rapidly across incremental sessions (last session 8.4% below the full method), though combining CutMix and mixup with distillation helps.
Limitations
- Some baseline results (CEC, FACT, SAVC) were reproduced by the authors rather than taken from papers.
- Ablation is conducted on CUB200 only, and the authors note that CutMix virtual classes are not suited to FSCIL on their own.
Experimental setup
Datasets: CIFAR100 CUB200 miniImageNet
Benchmarks: CIFAR100 CUB200 miniImageNet
Metrics: session-wise classification accuracy average accuracy intra-class distance inter-class distance
Baselines: Finetune iCaRL NCM TOPIC CEC ERDFR MetaFSCIL FACT ALICE SSFE-Net WaRP CABD
Builds on: CEC FACT SAVC IL2A mixup
Closest papers by meaning
- Few-Shot Class-Incremental Learning via Class-Aware Bilateral Distillation (CVPR 2023 · similarity 0.91)
- Tripartite Weight-Space Ensemble for Few-Shot Class-Incremental Learning (CVPR 2025 · similarity 0.87)
- Pseudo Informative Episode Construction for Few-Shot Class-Incremental Learning (AAAI 2025 · similarity 0.87)
- Adaptive Decision Boundary for Few-Shot Class-Incremental Learning (AAAI 2025 · similarity 0.86)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.86)
- Class Incremental Learning with Multi-Teacher Distillation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.