AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation

Zihui Xue, Zhengqi Gao, Sucheng Ren, Hang Zhao

ICLR 2023 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 12 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a modality-general teacher (larger gamma) instead of a regular teachercrossmodal KD student · NYU Depth V2 (depth teacher, RGB student)student mIoU (%): +1.04 vs regular KD (46.89 to 47.93)Improved
use a modality-general teacher instead of a regular teachercrossmodal KD student · RAVDESS (audio teacher, image student)student accuracy (%): +1.06 vs regular KD (77.22 to 78.28)Improved
use a modality-general teacher instead of a regular teachercrossmodal KD student · VGGSound ResNet-18 (audio teacher, video student)mAP (%): +1.26 (30.62 to 31.88)Improved
use a modality-general teacher instead of a regular teachercrossmodal KD student · VGGSound ResNet-50 (video teacher, audio student)mAP (%): +0.59 (58.87 to 59.46)Improved
use a modality-specific teacher (nullify largest-salience channels) instead of regular teacher (ablation)crossmodal KD student · RAVDESSstudent accuracy (%): -34.56 (77.22 to 42.66)Hurt
use a modality-specific video teacher instead of regular teacher (ablation)crossmodal KD audio student · VGGSound ResNet-50 (video teacher)mAP (%): -27.59 (58.87 to 31.28)Hurt
use a modality-general teacher trained on synthetic data with only modality-general features (gamma=1) instead of regular teacher, despite lower teacher accuracycrossmodal KD student · Synthetic Gaussian (gamma 0.25 to 0.75)student accuracy (%): about +2 points over regular KD (e.g., 59.46 to 61.01 at gamma 0.25)Improved
use a modality-specific multimodal teacher (biased to image via pseudo labels) instead of regular multimodal teacher (ablation)text student on MM-IMDB · MM-IMDBstudent micro F1 (%): -3.32 (65.09 to 61.77)Hurt
use a multimodal (text+image) teacher instead of a unimodal text teachertext student on MM-IMDB · MM-IMDBstudent micro F1 (%): +4.05 (61.04 to 65.09)Improved
use a multimodal (image+audio) teacher instead of unimodal audio teacher on AV-MNISTaudio student · AV-MNISTstudent accuracy (%): -0.37 (70.10 to 69.73)Hurt
use a multimodal (RGB+depth) teacher instead of unimodal RGB teacher on NYU Depth V2RGB student · NYU Depth V2student mIoU (%): -0.22 (48.00 to 47.78)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy