Research map / Label-efficient and robust learning / Knowledge distillation
AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AuG-KD, an anchor-based mixup generative distillation method for out-of-domain knowledge distillation where the teacher is frozen and only student-domain data is available.
Problem
Data-free knowledge distillation degrades badly when the teacher's training domain differs from the student's deployment domain (out-of-domain KD), with no access to teacher training data.
Approach
Learn a class-conditioned latent mask and mapping (AnchorNet) that moves student-domain samples' latent codes toward the teacher domain using a teacher-uncertainty (energy) score, then generate mixup samples between anchors and student data with a linearly scheduled factor so the student gradually shifts from teacher-domain knowledge to student-domain information.
Key results
- On Office-31 Amazon,Webcam->DSLR (ResNet34->MobileNet-V3-Small), AuG-KD reaches 84.3% top-1 vs. 80.4% for the best DFKD baseline DFQ+ and 67.1% for the teacher on the student domain.
- On VisDA-2017 train->validation, AuG-KD achieves 55.5% top-1 vs. 54.9% for PRE-DFKD+ and 50.7% for the no-KD baseline.
- Ablation on Office-31 shows the full M1+M2+M3 framework reaches 84.3% vs. 33.7% for the DFKD-only module M1 and 80.1% without mixup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AuG-KD instead of the strongest listed baseline in each Office-31 domain setting | ResNet34 → MobileNet-V3-Small distillation · Office-31 AW→D, AD→W, and DW→A | Acc: 80.4$\pm$5.7 -> 84.3$\pm$3.1; 86.5$\pm$5.7 -> 87.8$\pm$7.6; 46.6$\pm$4.5 -> 58.8$\pm$3.7 | Helped |
| Use AuG-KD instead of the strongest listed baseline in each Office-Home domain setting | ResNet34 → MobileNet-V3-Small distillation · Office-Home ACP→R, ACR→P, APR→C, and CPR→A | Acc: 33.3$\pm$1.3 -> 35.2$\pm$2.5; 60.0$\pm$3.8 -> 65.3$\pm$1.6; 50.6$\pm$2.8 -> 53.4$\pm$3.0; 21.0$\pm$3.4 -> 21.2$\pm$4.7 | Helped |
| Use AuG-KD instead of PRE-DFKD+ | ResNet34 → MobileNet-V3-Small distillation · VisDA-2017 validation→test | Acc: 54.9$\pm$1.0 -> 55.5$\pm$0.3 | Helped |
| Add Anchor Learning Module (M2) and Mixup Learning Module (M3) to M1 (ablation) | AuG-KD · Office-31 AW→D | Acc: 33.7$\pm$5.1 -> 84.3$\pm$3.1 | Helped |
| Add Anchor Learning Module (M2) to M1+M3 (ablation) | AuG-KD · Office-31 AW→D | Acc: 83.5$\pm$4.1 -> 84.3$\pm$3.1 | Helped |
| Add mixup samples to M1+M2+M3 without Mixup (ablation) | AuG-KD · Office-31 AW→D | Acc: 46.4$\uparrow\pm$0.6$\uparrow$ -> 84.3$\pm$3.1 | Helped |
| Vary mixup scheduler parameters a and b across the tested grid (ablation) | AuG-KD · Office-31 AW→D | Accuracy | Mixed |
| Use ResNet50→ResNet18, ResNet34→ShuffleNet-V2-X0-5, or ResNet34→EfficientNet-B0 teacher-student pairs (ablation) | AuG-KD · Office-31 AW→D | Acc | Helped |
Surprising findings
- Removing mixup samples (M1+M2+M3 w/o Mixup) lowers top-1 accuracy to 80.1%, but the authors report the w/o KD baseline is competitive in some settings (e.g., 82.7% on AW->D vs PRE-DFKD+ at 66.5%).
Limitations
- Experiments are limited to image classification with a single teacher-student architecture in the main results and small datasets at 32x32 resolution.
- DFKD baselines show high variance across seeds, and the authors note stability is not fully guaranteed.
- Compared baselines are adapted DFKD methods because no prior OOD-KD baselines exist, so comparisons are not like-for-like.
Experimental setup
Datasets: Office-31 Office-Home VisDA-2017
Benchmarks: Office-31 Office-Home VisDA-2017
Metrics: top-1 accuracy top-3 accuracy top-5 accuracy
Baselines: DFQ CMI DeepInv ZSKT PRE-DFKD w/o KD DFND C-SFDA SFDA-DE U-SFDA
Builds on: data-free knowledge distillation DeepInv CMI IRM curriculum learning
Closest papers by meaning
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.90)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
- When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need (ICML 2025 · similarity 0.89)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.88)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.88)
- De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.