Research map / Label-efficient and robust learning / Knowledge distillation
Do Topological Characteristics Help in Knowledge Distillation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TopKD, a knowledge distillation method that transfers the global topology of latent feature spaces, using persistence diagrams approximated by persistence images via RipsNet, and reports gains over vanilla KD on CIFAR-100 and ImageNet-1K.
Problem
Existing knowledge distillation methods transfer only point-to-point or pairwise (relational) structure of embedding features, which fails to capture the global structure of complex latent spaces efficiently.
Approach
Defines global topology knowledge as the persistence diagram of teacher embedding features, approximates persistence images with a pretrained RipsNet so they can be computed per batch, and distills them with an L2 topology distillation loss added to vanilla KD and cross-entropy.
Key results
- On CIFAR-100 with ResNet56 teacher and ResNet20 student, TopKD reaches 71.58% top-1 vs 70.66% for vanilla KD and 69.06% for the student trained alone.
- On ImageNet-1K with ResNet34 teacher and ResNet18 student, TopKD reaches 73.60% top-1, above the 73.31% teacher and the 70.66-level KD baseline (KD 70.66 on ImageNet per Table 3 KD row 70.66 as listed).
- Matching 0-dim persistence diagrams on the teacher's fourth-stage features gives the best accuracy in ablations (Table 5 and Table 7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill global topology with TopKD instead of vanilla KD | ResNet56→ResNet20 · CIFAR-100 | top-1 accuracy: 70.66 -> 71.58 | Helped |
| distill global topology with TopKD instead of vanilla KD | ResNet34→ResNet18 · ImageNet-1K val | top-1 accuracy: 71.89 -> 73.60 | Helped |
| use only 0-dim persistence diagrams rather than only 1-dim diagrams (ablation) | TopKD, ResNet56→ResNet20 · CIFAR-100 | top-1 accuracy: 71.22 -> 71.58 | Helped |
| add 0-dim persistence diagrams to 1-dim diagrams (ablation) | TopKD, ResNet56→ResNet20 · CIFAR-100 | top-1 accuracy: 71.22 -> 71.36 | Helped |
| add topology distillation loss to classification loss (ablation) | ResNet56→ResNet20 · CIFAR-100 | top-1 accuracy: 69.06 -> 69.72 | Helped |
| add vanilla KD loss to classification and topology distillation losses (ablation) | ResNet56→ResNet20 · CIFAR-100 | top-1 accuracy: 69.72 -> 71.58 | Helped |
Surprising findings
- On ImageNet-1K, the TopKD student (73.60% top-1) surpasses its ResNet34 teacher (73.31%).
- Using the topology loss alone without the KD loss underperforms vanilla KD, but combining it with KD gives a larger gain than either term alone.
Limitations
- Performance depends on RipsNet approximation quality; the student-side approximation error is larger than the teacher's.
- Requires adequate batch size (performance drops below the student at batch size 8).
- Only 0-dim and 1-dim persistence diagrams are explored, with other topological features left to future work.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-100 ImageNet-1K
Metrics: top-1 accuracy top-5 accuracy approximation error of persistence images
Baselines: KD FitNet AT CRD CRCD OFD RKD DKD ReviewKD NORM
Builds on: vanilla KD FitNet RKD RipsNet persistence images
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.85)
- Geometry Awakening: Cross-Geometry Learning Exhibits Superiority over Individual Structures (NeurIPS 2024 · similarity 0.84)
- TopInG: Topologically Interpretable Graph Learning via Persistent Rationale Filtration (ICML 2025 · similarity 0.84)
- Towards Global-Topology Relation Graph for Inductive Knowledge Graph Completion (AAAI 2025 · similarity 0.84)
- Disentanglement Learning via Topology (ICML 2024 · similarity 0.84)
- Topological Neural Networks go Persistent, Equivariant, and Continuous (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.