Research map / Label-efficient and robust learning / Knowledge distillation
CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification
Yuan Gong, Sameer Khurana, Andrew Rouditchenko, James Robert Glass
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CNN teacher instead of same-architecture teacher (cross-model vs self-KD) (ablation) | AST-Base · FSD50K eval | mAP: 59.6 -> 61.7 | Improved |
| use a smaller AST-Tiny teacher instead of the strongest AST-Base teacher (ablation) | EfficientNet-B2 · FSD50K eval | mAP: 57.3 -> 58.2 | Improved |
| distill from KD-trained teacher (iterative KD) instead of no-KD teacher (ablation) | EfficientNet-B2 / AST-Base · FSD50K eval | mAP: 58.2 -> 58.0 and 61.7 -> 61.4 | Hurt |
| raise temperature from 1.0 to 2.5 (ablation) | EfficientNet-B2 (AST to CNN KD) · FSD50K eval | mAP: 57.2 -> 58.2 | Improved |
| lower temperature from 1.0 to 0.5 (ablation) | AST-Base (CNN to AST KD) · FSD50K eval | mAP: 61.7 -> 60.9 | Hurt |
| add ground-truth loss to teacher soft labels (lambda 0 -> 0.5) (ablation) | AST-Base · FSD50K eval | mAP: 59.7 -> 61.7 | Improved |
| use consistent teacher (same augmented input as student) instead of conventional teacher (ablation) | AST-Base (CNN to AST KD) · FSD50K eval | mAP: 60.4 -> 61.7 | Improved |
| use consistent teacher instead of conventional teacher (ablation) | EfficientNet-B2 (AST to CNN KD) · FSD50K eval | mAP: 58.2 -> 58.2 | No clear effect |
| add a separate distillation head (ablation) | AST-Base · FSD50K eval | mAP: 61.7 -> 61.4 | Hurt |
| add a separate distillation head (ablation) | EfficientNet-B2 · FSD50K eval | mAP: 58.2 -> 56.9 | Hurt |
| add a second diverse teacher (EfficientNet-B0 + DenseNet-121) with multi-loss KD | AST-Base · FSD50K eval | mAP: 61.7 -> 62.3 | Improved |
| add multi-teacher KD | EfficientNet-B2 · FSD50K eval | mAP: 58.2 -> 58.0 | No clear effect |
Closest papers by meaning
- Heuristic-free Knowledge Distillation for Streaming ASR via Multi-modal Training (AAAI 2025 · similarity 0.86)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.86)
- DTF-AT: Decoupled Time-Frequency Audio Transformer for Event Classification (AAAI 2024 · similarity 0.85)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.85)
- X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation (NeurIPS 2026 · similarity 0.84)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
