Research map / Vision-language and multimodal / Cross-modal learning
Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks
Haoyi Duan, Yan Xia, Mingze Zhou, Li Tang, Jieming Zhu, Zhou Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DG-SCT cross-modal prompts to frozen audio and visual encoders | Swin-V2-L + HTS-AT · AVE test | accuracy: 79.3 -> 82.2 | Improved |
| Add DG-SCT cross-modal prompts | Swin-V2-L + HTS-AT · AVSBench S4 | mean Jaccard (M_J): 78.0 -> 80.9 | Improved |
| Add DG-SCT cross-modal prompts | Swin-V2-L + HTS-AT · MUSIC-AVQA test | Avg accuracy: 73.6 -> 74.8 | Improved |
| Add channel attention (ablation) | DG-SCT without spatial or temporal attention · AVE | accuracy: 78.6 -> 81.8 | Improved |
| Add spatial attention (ablation) | DG-SCT without channel or temporal attention · AVE | accuracy: 78.6 -> 80.9 | Improved |
| Add temporal attention alongside spatial and channel attention (ablation) | DG-SCT with S and C · AVE | accuracy: 82.0 -> 82.2 | Improved |
| Add temporal attention alongside spatial and channel attention (ablation) | DG-SCT with S and C · AVSBench MS3 | mean F-score (M_F): 62.3 -> 64.2 | Improved |
| Add A2V guidance (ablation) | V2A-only DG-SCT · AVE | accuracy: 81.3 -> 82.2 | Improved |
| Add V2A guidance (ablation) | A2V-only DG-SCT · AVE | accuracy: 79.2 -> 82.2 | Improved |
| Add both A2V and V2A guidance (ablation) | DG-SCT with no cross-modal direction · AVE | accuracy: 78.6 -> 82.2 | Improved |
| Use bidirectional instead of A2V-only guidance (ablation) | DG-SCT · LLP test, segment-level | AV: 53.3 -> 52.8 | Hurt |
| Use bidirectional instead of V2A-only guidance (ablation) | DG-SCT · LLP test, segment-level | A: 59.4 -> 59.0 | No clear effect |
Closest papers by meaning
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.88)
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.88)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.88)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.88)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.87)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
