Research map / Vision-language and multimodal / Cross-modal learning
C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C2KD, a cross-modal knowledge distillation method that bidirectionally updates teacher and student, filters rank-misaligned samples via Kendall rank correlation, distills non-target classes, and uses proxy student/teacher heads to transfer knowledge across modalities.
Problem
Traditional knowledge distillation fails when knowledge must be transferred from a teacher modality to a different student modality (CMKD), due to modality imbalance and soft label misalignment across modalities.
Approach
Bidirectionally tune the pre-trained teacher with the student, select samples with Kendall-rank-consistent soft labels (OFSD), distill only non-target-class knowledge, and add proxy student/teacher heads that progressively pass cross-modal knowledge.
Key results
- On AVE, C2KD reaches 34.7% visual accuracy (vs 31.6% w/o KD and 32.9% for the best prior KD, NKD) and 54.9% audio accuracy (vs 52.8% w/o KD).
- On VGGSound, C2KD gives 40.9% visual and 61.9% audio accuracy, versus 38.7% and 59.4% without KD.
- On NYU-Depth V2 segmentation, C2KD improves depth-to-RGB mIoU to 37.5 (RN18 pair) vs 36.1 w/o KD and 37.3 for CIRKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C2KD instead of the strongest listed baseline | Cross-modal distillation · AVE Visual | top-1 accuracy (%): 32.9 -> 34.7 | Helped |
| Use C2KD instead of the strongest listed baseline | Cross-modal distillation · AVE Audio | top-1 accuracy (%): 52.2 -> 54.9 | Helped |
| Use C2KD instead of the strongest listed baseline | Cross-modal distillation · VGGSound Visual | top-1 accuracy (%): 39.2 -> 40.9 | Helped |
| Use C2KD instead of the strongest listed baseline | Cross-modal distillation · VGGSound Audio | top-1 accuracy (%): 59.3 -> 61.9 | Helped |
| Replace KL loss with DIST loss (ablation) | C2KD on NYU-Depth V2 · NYU-Depth V2 RGB, ResNet-18/ResNet-18 | mIoU (%): 37.5 -> 38.1 | Helped |
| Remove proxy teacher and proxy student (ablation) | C2KD · AVE Visual | top-1 accuracy (%): 34.6 -> 34.7 | Hurt |
| Remove proxy teacher and proxy student (ablation) | C2KD · AVE Audio | top-1 accuracy (%): 54.3 -> 54.9 | Hurt |
| Add bidirectional distillation to vanilla KD (ablation) | C2KD · AVE Audio | top-1 accuracy (%): 46.6 -> 47.9 | Helped |
| Add on-the-fly sample selection and non-target-class distillation (ablation) | C2KD · AVE Audio | top-1 accuracy (%): 47.9 -> 54.3 | Helped |
| Filter samples by Kendall Rank Correlation instead of random filtering (ablation) | Non-target-class KD · AVE Audio | top-1 accuracy (%): 50.2 -> 51.0 | Helped |
| Use target-class KD (ablation) | Cross-modal KD · AVE Audio | top-1 accuracy (%): 52.8 -> 48.0 | Hurt |
| Use target-class KD (ablation) | Cross-modal KD · VGGSound Audio | top-1 accuracy (%): 59.4 -> 55.8 | Hurt |
Surprising findings
- Naive traditional KD (e.g., KD, DIST, NKD) often degrades the high-accuracy audio student when distilling from a visual teacher, and multimodal teachers with high accuracy do not necessarily improve the distilled modality.
- Self-knowledge distillation methods improve each modality but still underperform C2KD, which the authors take as evidence for the necessity of cross-modal KD.
Limitations
- Feature-based cross-modal distillation is not solved; the authors observe large CKA divergence and leave it to future work.
- Hyperparameters (KRC threshold omega, lambda weights) are set heuristically and sensitivity varies across settings.
- Experiments cover only a few modality pairs and a 50-class VGGSound subset.
Experimental setup
Datasets: CREMA-D AVE VGGSound CrisisMMD NYU-Depth V2
Benchmarks: CREMA-D AVE VGGSound CrisisMMD NYU-Depth V2
Metrics: top-1 accuracy mIoU
Baselines: KD FitNet Review DML SHAKE RKD DKD DIST NKD CIRKD DLB ZipfKD
Builds on: Knowledge distillation (Hinton et al.) DKD DML SHAKE NKD Modality Focusing Hypothesis
Closest papers by meaning
- Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.89)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.88)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.87)
- Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement (CVPR 2025 · similarity 0.86)
- Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness (NeurIPS 2024 · similarity 0.86)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.