AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks

Haoyi Duan, Yan Xia, Mingze Zhou, Li Tang, Jieming Zhu, Zhou Zhao

NeurIPS 2023 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add DG-SCT cross-modal prompts to frozen audio and visual encodersSwin-V2-L + HTS-AT · AVE testaccuracy: 79.3 -> 82.2Improved
Add DG-SCT cross-modal promptsSwin-V2-L + HTS-AT · AVSBench S4mean Jaccard (M_J): 78.0 -> 80.9Improved
Add DG-SCT cross-modal promptsSwin-V2-L + HTS-AT · MUSIC-AVQA testAvg accuracy: 73.6 -> 74.8Improved
Add channel attention (ablation)DG-SCT without spatial or temporal attention · AVEaccuracy: 78.6 -> 81.8Improved
Add spatial attention (ablation)DG-SCT without channel or temporal attention · AVEaccuracy: 78.6 -> 80.9Improved
Add temporal attention alongside spatial and channel attention (ablation)DG-SCT with S and C · AVEaccuracy: 82.0 -> 82.2Improved
Add temporal attention alongside spatial and channel attention (ablation)DG-SCT with S and C · AVSBench MS3mean F-score (M_F): 62.3 -> 64.2Improved
Add A2V guidance (ablation)V2A-only DG-SCT · AVEaccuracy: 81.3 -> 82.2Improved
Add V2A guidance (ablation)A2V-only DG-SCT · AVEaccuracy: 79.2 -> 82.2Improved
Add both A2V and V2A guidance (ablation)DG-SCT with no cross-modal direction · AVEaccuracy: 78.6 -> 82.2Improved
Use bidirectional instead of A2V-only guidance (ablation)DG-SCT · LLP test, segment-levelAV: 53.3 -> 52.8Hurt
Use bidirectional instead of V2A-only guidance (ablation)DG-SCT · LLP test, segment-levelA: 59.4 -> 59.0No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy