Research map / Label-efficient and robust learning / Knowledge distillation
Direct Distillation between Different Domains
Jialiang Tang, Shuo Chen, Gang Niu, Hongyuan Zhu, Joey Tianyi Zhou, Chen Gong, Masashi Sugiyama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add domain-invariant feature transfer loss (ablation) | 4Ds · Office-Home | accuracy: 80.25 -> 82.77 | Improved |
| add categorical knowledge transfer loss (ablation) | 4Ds · Office-Home | accuracy: 81.80 -> 82.77 | Improved |
| use fusion-activation weighting instead of no weights (ablation) | 4Ds · Office-Home | accuracy: 80.07 -> 82.77 | Improved |
| use fusion-activation instead of block-wise plus channel-wise weights (ablation) | 4Ds · Office-Home | accuracy: 81.83 -> 82.77 | Improved |
| use adapters on the teacher instead of fixed or fully learnable teacher (ablation) | 4Ds · Office-Home | accuracy: 79.64 / 78.55 -> 82.77 | Improved |
| train adapted teacher with cross-entropy and knowledge transfer losses (ablation) | 4Ds · Office-Home | accuracy: 81.22 / 81.86 -> 82.77 | Improved |
| use one-stage 4Ds instead of two-stage DAA-SHOT | student ResNet18 · Average of four datasets | accuracy: 85.61 -> 87.80 | Improved |
| apply two-stage NRC-based adaptation plus distillation | student ResNet18 · Average of four datasets | accuracy: 84.60 -> 82.77 (DAA-NRC) | Hurt |
| apply traditional KD methods (VKD, Fitnets, VID, SPKD, SemCKD, SRRL) from a source-trained teacher | student ResNet18 · Average of four datasets | accuracy: 84.60 -> 80.48 to 83.10 | Hurt |
| use 4Ds instead of training the compact student on target data | student ResNet18 · PACS | accuracy: 94.67 -> 97.65 | Improved |
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.91)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.89)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.89)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
