Research map / Continual learning / Online and replay-based
Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Sub-graph Distillation (DSGD), a graph-structure distillation loss based on K-order personalized PageRank vectors over batch-dynamic similarity graphs, to reduce catastrophic forgetting of unlabeled data in semi-supervised class-incremental learning when added to iCaRL and DER with FixMatch.
Problem
Semi-supervised continual learning suffers catastrophic forgetting, especially on unlabeled data, because unreliable pseudo-labels and distribution bias make conventional logit or representation distillation unstable.
Approach
Instead of distilling absolute representations or predictions on unlabeled replayed samples, distill the structural relations: build old and new similarity graphs over replayed exemplars, compute K-step PPR distillation vectors per sample, and match them between old and new networks; also blends previous-model predictions into FixMatch pseudo-labels with a logistic schedule.
Key results
- On CIFAR100-20 (4% labels), DSGD raises iCaRL&Fix average accuracy from 45.75 to 52.80 and last accuracy from 23.40 to 35.47.
- On CIFAR10-30 (0.6% labels), DSGD raises iCaRL&Fix average accuracy from 45.98 to 77.33 and last accuracy from 30.71 to 76.41.
- On ImageNet-100-100, DSGD raises iCaRL&Fix average accuracy from 37.49 to 50.53, though JointTrain (71.56 last) remains far higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DSGD to iCaRL&Fix | iCaRL&Fix · CIFAR100-20, CIFAR100-25, CIFAR100-80, CIFAR100-125 | average accuracy: +7.05; +3.59; +4.46; +1.21 | Helped |
| add DSGD to iCaRL&Fix | iCaRL&Fix · CIFAR100-20, CIFAR100-25, CIFAR100-80, CIFAR100-125 | last accuracy: +12.07; +4.70; +5.60; +1.76 | Helped |
| add DSGD to iCaRL&Fix | iCaRL&Fix · CIFAR10-30, CIFAR10-150 | average accuracy: +31.35; +5.78 | Helped |
| add DSGD to iCaRL&Fix | iCaRL&Fix · CIFAR10-30, CIFAR10-150 | last accuracy: +45.70; +10.61 | Helped |
| add DSGD to iCaRL&Fix | iCaRL&Fix · ImageNet-100-13, ImageNet-100-100 | average accuracy: + 1.98; +13.04 | Helped |
| add DSGD to iCaRL&Fix | iCaRL&Fix · ImageNet-100-13, ImageNet-100-100 | last accuracy: + 3.56; +11.8 | Helped |
| add DSGD (ablation) | iCaRL&Fix · CIFAR100-5, CIFAR100-20, CIFAR100-80 | average accuracy: 31.79 -> 35.91; 45.75 -> 48.44; 53.46 -> 54.40 | Helped |
| add PseDis (ablation) | iCaRL&Fix · CIFAR100-5, CIFAR100-20, CIFAR100-80 | average accuracy: 31.79 -> 35.42; 45.75 -> 47.67; 53.46 -> 53.71 | Helped |
| add PseDis alongside DSGD (ablation) | iCaRL&Fix + DSGD · CIFAR100-5, CIFAR100-20, CIFAR100-80 | average accuracy: 35.91 -> 39.85; 48.44 -> 52.80; 54.40 -> 57.92 | Helped |
| increase K from 1 to 6 (ablation) | DSGD · CIFAR100-20 | average accuracy: 47.34% to 50.98% | Helped |
| increase K from 1 to 6 (ablation) | DSGD · CIFAR100-5 | average accuracy: 31.37% to 37.25% | Helped |
| vary gamma across 0.9, 0.95, 1, 1.5, and 2 (ablation) | DSGD · CIFAR100-20 | performance | No clear effect |
Surprising findings
- Replacing unlabeled-sample targets with previous-model predictions (PseDis-style pseudo-label distillation) reduced accuracy in the authors' systematic study, suggesting conventional distillation on unlabeled data introduces unreliable information.
Limitations
- Gains on ImageNet-100 remain far below joint training, indicating forgetting persists in realistic settings.
- Distillation relies on a fully connected similarity graph and K-step PPR, with the design choices (e.g., alpha schedule) heuristically set.
- Comparisons with some baselines (e.g., CCIC, DistillMatch) are reported only partially, with missing entries in tables.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100
Metrics: average incremental accuracy last incremental accuracy
Baselines: iCaRL DER iCaRL&Fix DER&Fix DistillMatch CCIC NNCSL ORDisCo JointTrain
Builds on: iCaRL DER FixMatch ORDisCo
Closest papers by meaning
- Persistence Homology Distillation for Semi-supervised Continual Learning (NeurIPS 2024 · similarity 0.92)
- Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (ICLR 2024 · similarity 0.90)
- A Selective Learning Method for Temporal Graph Continual Learning (ICML 2025 · similarity 0.89)
- Rethinking Momentum Knowledge Distillation in Online Continual Learning (ICML 2024 · similarity 0.89)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.88)
- Exploring Rationale Learning for Continual Graph Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.