Research map / Vision-language and multimodal / Cross-modal learning
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Zihui Xue, Zhengqi Gao, Sucheng Ren, Hang Zhao
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a modality-general teacher (larger gamma) instead of a regular teacher | crossmodal KD student · NYU Depth V2 (depth teacher, RGB student) | student mIoU (%): +1.04 vs regular KD (46.89 to 47.93) | Improved |
| use a modality-general teacher instead of a regular teacher | crossmodal KD student · RAVDESS (audio teacher, image student) | student accuracy (%): +1.06 vs regular KD (77.22 to 78.28) | Improved |
| use a modality-general teacher instead of a regular teacher | crossmodal KD student · VGGSound ResNet-18 (audio teacher, video student) | mAP (%): +1.26 (30.62 to 31.88) | Improved |
| use a modality-general teacher instead of a regular teacher | crossmodal KD student · VGGSound ResNet-50 (video teacher, audio student) | mAP (%): +0.59 (58.87 to 59.46) | Improved |
| use a modality-specific teacher (nullify largest-salience channels) instead of regular teacher (ablation) | crossmodal KD student · RAVDESS | student accuracy (%): -34.56 (77.22 to 42.66) | Hurt |
| use a modality-specific video teacher instead of regular teacher (ablation) | crossmodal KD audio student · VGGSound ResNet-50 (video teacher) | mAP (%): -27.59 (58.87 to 31.28) | Hurt |
| use a modality-general teacher trained on synthetic data with only modality-general features (gamma=1) instead of regular teacher, despite lower teacher accuracy | crossmodal KD student · Synthetic Gaussian (gamma 0.25 to 0.75) | student accuracy (%): about +2 points over regular KD (e.g., 59.46 to 61.01 at gamma 0.25) | Improved |
| use a modality-specific multimodal teacher (biased to image via pseudo labels) instead of regular multimodal teacher (ablation) | text student on MM-IMDB · MM-IMDB | student micro F1 (%): -3.32 (65.09 to 61.77) | Hurt |
| use a multimodal (text+image) teacher instead of a unimodal text teacher | text student on MM-IMDB · MM-IMDB | student micro F1 (%): +4.05 (61.04 to 65.09) | Improved |
| use a multimodal (image+audio) teacher instead of unimodal audio teacher on AV-MNIST | audio student · AV-MNIST | student accuracy (%): -0.37 (70.10 to 69.73) | Hurt |
| use a multimodal (RGB+depth) teacher instead of unimodal RGB teacher on NYU Depth V2 | RGB student · NYU Depth V2 | student mIoU (%): -0.22 (48.00 to 47.78) | Hurt |
Closest papers by meaning
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation (CVPR 2024 · similarity 0.92)
- Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm (ICML 2026 · similarity 0.89)
- Distilling Cross-Modal Knowledge via Feature Disentanglement (AAAI 2026 · similarity 0.88)
- Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.88)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.86)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
