Research map / Label-efficient and robust learning / Knowledge distillation
Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TGeo-KD, which learns a sample-wise knowledge fusion ratio between distillation and ground-truth losses from intra- and inter-sample trilateral geometry of student, teacher, and ground-truth predictions, via a bilevel formulation.
Problem
Knowledge distillation requires balancing soft teacher supervision and hard ground-truth supervision, but existing methods use constant, annealed, or heuristic sample-independent fusion ratios that are suboptimal.
Approach
Model the student, teacher, and ground-truth predictions as a triangle per sample (edges and vertices), add the class-average teacher prediction as an inter-sample vertex, and let a small MLP map these geometric features to a sample-wise fusion ratio learned in a bilevel optimization.
Key results
- On CIFAR-100 (ResNet-32x4 teacher to ShuffleNetV1 student), TGeo-KD reaches 77.27% top-1 vs 76.05% for the strongest same-architecture-style baseline WLS-KD in the table.
- On ImageNet ResNet-34 to ResNet-18, TGeo-KD reaches 72.89% top-1, about 1.10 points above WLS-KD (71.52%) in the MobileNet setting; ResNet-18 setting improvement is 0.85 over WLS-KD 72.04.
- On HIL with a 12-layer BERT teacher and 4-layer student, TGeo-KD gets 92.39% ACC vs 90.15% for ADA-KD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TGeo-KD instead of ADA-KD | ResNet-56→ResNet-20 distillation · CIFAR-100 | Top-1 accuracy (%): 72.22 ±0.21 -> 72.98 ±0.14 | Helped |
| Use TGeo-KD instead of WLS-KD | ResNet-110→ResNet-32 distillation · CIFAR-100 | Top-1 accuracy (%): 74.12 -> 75.09 ±0.13 | Helped |
| Use TGeo-KD instead of WLS-KD | ResNet-32×4→ShuffleNetV1 distillation · CIFAR-100 | Top-1 accuracy (%): 75.46 -> 76.83 ±0.17 | Helped |
| Use TGeo-KD instead of WLS-KD | ResNet-34→ResNet-18 distillation · ImageNet | Top-1 accuracy: 72.04 -> 72.89 ±0.15 | Helped |
| Use TGeo-KD instead of WLS-KD | ResNet-34→ResNet-18 distillation · ImageNet | Top-5 accuracy: 90.70 -> 91.80 ±0.04 | Helped |
| Use TGeo-KD instead of WLS-KD | ResNet-50→MobileNetV1 distillation · ImageNet | Top-1 accuracy: 71.52 -> 72.46 ±0.14 | Helped |
| Use TGeo-KD instead of ADA-KD | 12-layer BERT→4-layer BERT distillation · HIL | ACC (%): 90.15 -> 92.39 | Helped |
| Use TGeo-KD instead of WLS-KD | 12-layer BERT→4-layer BERT distillation · Criteo | ACC (%): 75.30 -> 77.80 | Helped |
| Add the intra-sample STG relation (ablation) | TGeo-KD on ResNet-32×4→ShuffleNetV1 · CIFAR-100 | Top-1 accuracy (%): 73.92 -> 75.28 | Helped |
| Add the inter-sample relation to intra-sample STG (ablation) | TGeo-KD on ResNet-32×4→ShuffleNetV1 · CIFAR-100 | Top-1 accuracy (%): 75.28 -> 76.83 | Helped |
| Use a 2-layer rather than a 1-layer MLP for the fusion-ratio generator (ablation) | TGeo-KD with ResNet-110→ResNet-32 · CIFAR-100 | Top-1 accuracy (%): 72.69 -> 74.31 | Helped |
| Use combined vertex-and-edge representation R3 rather than R1 or R2 (ablation) | TGeo-KD · CIFAR-100 | Top-1 accuracy (%): 70.32; 68.44 -> 72.98 | Helped |
Surprising findings
- Adaptive methods such as ADA-KD and RW-KD sometimes underperform vanilla KD or the student on some architecture pairs (e.g., RW-KD 70.33 on ResNet-56 to ResNet-20 CIFAR-100 vs 70.66 vanilla KD).
Limitations
- Inter-sample relations only consider samples within the same class; cross-class relations are not modeled.
- Relies on oversampling for imbalanced datasets (HIL, Criteo), and results depend on that preprocessing.
Experimental setup
Datasets: CIFAR-100 ImageNet HIL Criteo DomainNet
Benchmarks: CIFAR-100 ImageNet HIL Criteo DomainNet
Metrics: top-1 accuracy top-5 accuracy ACC AUC NLL
Baselines: Vanilla KD ANL-KD ADA-KD WLS-KD RW-KD FitNet AT SP CC VID RKD PKT
Builds on: knowledge distillation WLS-KD RW-KD ADA-KD ANL-KD
Closest papers by meaning
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.88)
- Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence (ICML 2025 · similarity 0.88)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.