Research map / Continual learning / Online and replay-based
Persistence Homology Distillation for Semi-supervised Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Persistence Homology Distillation (PsHD), a topology-based distillation loss on unlabeled memory-buffer samples that preserves multi-scale persistence diagrams of k-hop neighborhoods, with an approximate algorithm to cut computation, for semi-supervised continual learning.
Problem
Catastrophic forgetting in semi-supervised continual learning, where traditional relation/feature distillation on noisy or inaccurate unlabeled data is unstable when feature spaces change substantially.
Approach
Distill persistence homology (persistence diagrams of weighted k-hop neighborhood graphs built from logit cosine similarity) using p-Wasserstein distance between old and new models, instead of pairwise sample or similarity distillation; an approximate grouping scheme reduces the number of diagrams computed.
Key results
- On 5-task CIFAR-10 at 25% labels with buffer 2000 (Table 1, DER_Fix setting), PsHD reaches 88.0 avg vs 85.0 for DSGD; on CIFAR-100 5% (iCaRL setting), PsHD gets 56.6 avg vs 53.4 for DSGD.
- Average 3.9% improvement over NNCSL across six SSCL settings with buffer 5120 (Table 7), and 1.5% gain with 60% smaller buffer (2000).
- Under Gaussian noise (sigma=1.2) on CIFAR-10 5%, PsHD keeps AA 67.8 with BWT 18.3, versus 61.8 for LUCIR and 65.8 for TopKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace unlabeled-sample relation distillation with persistence homology distillation | iCaRL-based SSCL · CIFAR-10, CIFAR-100, and ImageNet-100; 5% and 25% on CIFAR datasets, 1% and 7.7% on ImageNet-100 | average incremental accuracy: +1.6; +2.9; +3.2; +1.3; +1.4; +2.9 | Helped |
| replace unlabeled-sample relation distillation with persistence homology distillation | DER-based SSCL · CIFAR-10, CIFAR-100, and ImageNet-100; 5% and 25% on CIFAR datasets, 1% and 7.7% on ImageNet-100 | average incremental accuracy: -0.3; +3.0; +0.4; +0.4; +0.9; +0.7 | Mixed |
| use PsHD instead of DSGD | SSCL · CIFAR-10 and CIFAR-100, default class order, buffer 500 | average accuracy: +1.1; +0.4; +0.5; +0.3; +0.8; +0.7 | Helped |
| replace logits, feature, or relation distillation with persistence homology distillation | SSCL · CIFAR-10 and CIFAR-100, 5% and 25% | average accuracy | Helped |
| add Gaussian noise to distillation | distillation-based SSCL methods · CIFAR-10 5% supervision, σ 0.2, 1.0, and 1.2 | average accuracy | Mixed |
| include persistence homology distillation loss Lhd (ablation) | iCaRL_Fix SSCL · CIFAR-10 5%, 25%; CIFAR-100 5%, 25% | average accuracy: 83.7 -> 85.3+1.5; 82.9 -> 85.4+2.6; 54.2 -> 56.4+2.2; 58.4 -> 59.2+0.8 | Helped |
| include persistence homology distillation loss Lhd (ablation) | DER_Fix SSCL · CIFAR-10 5%, 25%; CIFAR-100 5%, 25% | average accuracy: 86.1 -> 86.2+0.1; 84.9 -> 87.9+3.0; 57.8 -> 58.2+0.4; 68.1 -> 68.3+0.2 | Helped |
| consider both 0- and 1-dimensional holes instead of only 0-dimensional holes (ablation) | PsHD · CIFAR-10 | average incremental accuracy | Helped |
| consider both 0- and 1-dimensional holes instead of only 0-dimensional holes (ablation) | PsHD · CIFAR-100 | average incremental accuracy | Hurt |
| replace 20%-40% labeled memory samples with unlabeled samples (ablation) | SSCL memory buffer · CIFAR-100 5% supervision | average accuracy: 1.49% | Helped |
| replace labeled memory samples with unlabeled samples (ablation) | SSCL memory buffer · CIFAR-100 25% supervision | average accuracy: 1.27% | Mixed |
| reduce memory buffer size from 5120 to 2000 (ablation) | PsHD · CIFAR-100 5% supervision | average accuracy: 47.7 -> 46.0 | No clear effect |
Surprising findings
- Adding unlabeled-data distillation to iCaRL and PodNet can degrade average incremental accuracy in the motivating study (Figure 1).
- On DER_Fix CIFAR-10 5% the gain is slightly negative (86.0 vs 86.3 for DSGD in avg accuracy).
Limitations
- Persistence diagram computation relies on the Gudhi package running on CPU, which is a computational bottleneck.
- Theoretical stability bound is stated with a loose constant and the accelerated algorithm's cluster-count approximation (|S| approximately equal to number of classes) is empirical.
- Results reported only on small-resolution and one higher-resolution benchmark with no large-scale or non-vision evaluation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100
Metrics: average incremental accuracy (AIA) average accuracy at last task (AAT) backward transfer (BWT)
Baselines: iCaRL iCa_Fix DSGD DER_Fix PseudoER CCIC CSL NNCSL FOSTER X-DER PAWS PodNet
Builds on: FixMatch DSGD iCaRL DER NNCSL
Closest papers by meaning
- Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning (AAAI 2024 · similarity 0.92)
- Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (ICLR 2024 · similarity 0.87)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.86)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.86)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.86)
- Continual Generalized Category Discovery: Learning and Forgetting from a Bayesian Perspective (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.