Research map / Vision-language and multimodal / Cross-modal learning
XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XKD, a self-supervised audio-visual framework combining masked autoencoding with cross-modal teacher-student distillation aided by MMD-based domain alignment and attention-based feature refinement, plus modality-agnostic shared-backbone variants.
Problem
Learning video representations from unlabelled audio-visual data is hampered by the large domain gap between audio and visual modalities, which makes effective cross-modal knowledge sharing difficult.
Approach
Cross-modal self-distillation between audio and video teachers/students, with a domain alignment step (attention-based feature refinement plus MMD loss) to make audio and visual distributions compatible, combined with masked data reconstruction; modality-agnostic variants share backbones across audio and video.
Key results
- Cross-modal distillation with domain alignment improves linear eval accuracy on UCF101, HMDB51 and Kinetics400 by 8.1-14.2% over the audio-visual MAE baseline.
- XKD pretrained on AudioSet reaches 96.5% top-1 finetuned accuracy on ESC50, state of the art as reported.
- XKD reaches 81.2% on Kinetics-Sound multimodal late fusion, 5.5% above the audio-visual masked autoencoder baseline and 14.5% above CrissCross.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal distillation and domain alignment instead of video-only masked reconstruction (ablation) | XKD video representations · UCF101, HMDB51, Kinetics-Sound, and Kinetics400 | top-1 accuracy: 8.6%; 8.2%; 13.9%; 15.7% | Helped |
| add cross-modal distillation and domain alignment instead of audio-only masked reconstruction (ablation) | XKD audio representations · FSD50K and ESC50 | mAP / top-1 accuracy: 1.5%; 1% | Helped |
| use joint audio-video masked reconstruction instead of video-only reconstruction (ablation) | XKD · UCF101, HMDB51, and Kinetics-Sound | top-1 accuracy: 0.1%-0.2% | No clear effect |
| add domain alignment while retaining cross-modal distillation (ablation) | Audio-video masked autoencoder · UCF101, HMDB51, and Kinetics400 | top-1 accuracy: 76.3 -> 76.3; 51.2 -> 51.2; 32.2 -> 32.2 | No clear effect |
| add cross-modal distillation while retaining domain alignment (ablation) | Audio-video masked autoencoder · UCF101, HMDB51, and Kinetics400 | top-1 accuracy: 77.8 -> 84.7; 51.4 -> 59.3; 33.7 -> 46.4 | Helped |
| add domain alignment without cross-modal distillation (ablation) | Audio-video masked autoencoder · UCF101, HMDB51, and Kinetics400 | top-1 accuracy: 76.3 -> 77.8; 51.2 -> 51.4; 32.2 -> 33.7 | No clear effect |
| add cross-modal feature refinement (ablation) | XKD · HMDB51, UCF101, and ESC50 | top-1 accuracy: 53.5 -> 59.3; 81.0 -> 84.7; 88.0 -> 91.0 | Helped |
| share audio and visual student backbones (ablation) | XKD-MAS · UCF101, HMDB51, and ESC50 | top-1 accuracy: 84.7 -> 81.5; 59.3 -> 55.2; 91.0 -> 89.5 | Hurt |
| share audio and visual teacher and student backbones (ablation) | XKD-MATS · UCF101, HMDB51, and ESC50 | top-1 accuracy: 84.7 -> 80.3; 59.3 -> 53.5; 91.0 -> 90.3 | Hurt |
| use teacher rather than student encoder (ablation) | XKD · HMDB51, UCF101, and ESC50 | top-1 accuracy: 57.8 -> 59.3; 84.4 -> 84.7; 90.3 -> 91.0 | Helped |
| increase video teacher temperature to 0.10 or 0.11 (ablation) | XKD · ESC50 versus HMDB51 and UCF101 | top-1 accuracy: ESC50: 87.0, 89.0, 89.0; HMDB51: 54.4, 55.9, 54.5; UCF101: 81.0, 81.9, 79.5 | Mixed |
| use 1-second rather than longer local temporal windows (ablation) | XKD · HMDB51, UCF101, and ESC50 | top-1 accuracy: 1-1: 58.5, 82.6, 88.0; 2-2: 55.2, 81.1, 86.8; 4-1: 58.4, 82.6, 82.3 | Mixed |
Surprising findings
- Video-to-audio distillation gives a much smaller gain than audio-to-video, attributed to the sharper audio distribution.
- Without domain alignment, cross-modal distillation fails and the model behaves as a plain audio-visual masked autoencoder.
Limitations
- Modality-agnostic variants show drops of 0.7-4.1% versus modality-specific ones, attributed to fixed backbone size.
- Video-to-audio distillation gives only modest improvements on audio tasks.
- Pretraining experiments use reduced resolution and 8 V100 GPUs with fixed hyperparameters tuned only on Kinetics400.
Experimental setup
Datasets: Kinetics-Sound Kinetics400 AudioSet UCF101 HMDB51 ESC50 FSD50K
Benchmarks: UCF101 HMDB51 Kinetics400 Kinetics-Sound ESC50 FSD50K
Metrics: top-1 accuracy mAP
Baselines: CrissCross AVID XDC GDT STiCA BraVe VATT VideoMAE AudioMAE
Builds on: DINO VideoMAE CrissCross MMD
Closest papers by meaning
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation (CVPR 2024 · similarity 0.88)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.86)
- X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation (NeurIPS 2026 · similarity 0.86)
- Video Event Extraction with Multi-View Interaction Knowledge Distillation (AAAI 2024 · similarity 0.85)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.85)
- Cross-modal Multi-task Learning for Multimedia Event Extraction (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.