Research map / Label-efficient and robust learning / Knowledge distillation
DisWOT: Student Architecture Search for Distillation WithOut Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search the student using semantic and sample-relation similarity metrics instead of NWOT | NAS-Bench-201 distillation search · CIFAR-100 | distillation accuracy: 73.31 -> 74.21 | Helped |
| Search the student using semantic and sample-relation similarity metrics instead of NWOT | NAS-Bench-201 distillation search · CIFAR-10 | distillation accuracy: 93.73 -> 93.55 | No clear effect |
| Search the student using semantic and sample-relation similarity metrics instead of NWOT | NAS-Bench-201 distillation search · ImageNet-16-120 | distillation accuracy: 45.43 -> 47.30 | Helped |
| Add semantic similarity to relation-similarity search (ablation) | DisWOT on NAS-Bench-201 · CIFAR-100 | distillation accuracy: 73.62 -> 74.21 | Helped |
| Remove semantic similarity from the search score (ablation) | DisWOT on NAS-Bench-201 · CIFAR-100 | search speed-up: 180× -> 300× | Helped |
| Remove semantic similarity from the search score (ablation) | DisWOT on NAS-Bench-201 · CIFAR-100 | distillation accuracy: 74.21 -> 73.62 | Hurt |
| Replace FitNets feature distance with Grad-CAM similarity matrices (ablation) | Semantic proxy on CIFAR-100 S0 · CIFAR-100, S0 | Spearman correlation: 64.06±6.11 -> 73.68±5.45 | Helped |
| Replace RKD relation distance with sample-relation similarity matrices (ablation) | Relation proxy on CIFAR-100 S0 · CIFAR-100, S0 | Spearman correlation: 13.52±11.51 -> 72.36±3.42 | Helped |
| Add relation similarity to semantic similarity (ablation) | DisWOT proxy on CIFAR-100 S0 · CIFAR-100, S0 | Spearman correlation: 73.68±5.45 -> 77.51±2.76 | Helped |
| Replace Kaiming initialization with Gaussian initialization (ablation) | DisWOT proxy on CIFAR-100 S0 · CIFAR-100, S0 | Spearman correlation: 84.08±0.03 -> 91.38±0.03 | Helped |
| Use DisWOT-search-selected student instead of fixed ResNet20 | KD with ResNet110 teacher · CIFAR-100 | Top-1 accuracy: 70.79 -> 71.84 | Helped |
| Add DisWOT semantic and relation losses to the distillation objective (ablation) | DisWOT† on CIFAR-100 S1 · CIFAR-100, 1M parameters | distillation accuracy: 74.23 -> 75.25 | Helped |
Closest papers by meaning
- UniADS: Universal Architecture-Distiller Search for Distillation Gap (AAAI 2024 · similarity 0.92)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.90)
- PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (ICML 2026 · similarity 0.90)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.89)
- Distillation Scaling Laws (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.