Research map / Vision-language and multimodal / Cross-modal learning
Accommodating Audio Modality in CLIP for Multimodal Processing
Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, Qin Jin
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio modality to CLIP for video retrieval | CLIP4VLA · MSR-VTT-1kA and VATEX | R@1: 45.6 -> 46.2; 57.3 -> 63.5 | Improved |
| Use audio and vision together instead of audio alone (ablation) | CLIP4VLA · Audiocaps | R@1: 28.4 -> 33.6 | Improved |
| Use audio and vision together instead of vision alone (ablation) | CLIP4VLA captioning · MSR-VTT | BLUE4: 44.4 -> 47.8 | Improved |
| Use audio and vision together instead of vision alone (ablation) | CLIP4VLA captioning · VATEX | CIDER: 58.5 -> 59.8 | Improved |
| Initialize the audio backbone with the vision backbone (ablation) | CLIP4VLA audio encoder · MSR-VTT audio retrieval | R@1: 2.5 -> 3.6 | Improved |
| Add inter-modal NCE alignment (ablation) | CLIP4VLA audio encoder · MSR-VTT audio retrieval | R@1: 3.6 -> 6.8 | Improved |
| Add intra-modal NCE learning (ablation) | CLIP4VLA audio encoder · MSR-VTT audio retrieval | R@1: 6.8 -> 10.5 | Improved |
| Add an audio type token (ablation) | CLIP4VLA audio encoder · MSR-VTT audio retrieval | R@1: 10.5 -> 10.6 | Improved |
| Add an audio type token (ablation) | CLIP4VLA audio encoder · VATEX audio retrieval | R@1: 9.3 -> 9.9 | Improved |
| Increase the [NB] proportion in the type-token mixture (ablation) | CLIP4VLA · Audiocaps | audio retrieval | Improved |
| Mix [NB] and [VB] embeddings at a 1:1 ratio (ablation) | CLIP4VLA · MSR-VTT and VATEX | audio retrieval: 1:1 | Improved |
| Use Global-to-Local rather than Global-to-Global fusion (ablation) | CLIP4VLA with vision and audio · MSR-VTT video retrieval | R@1: 44.5 -> 46.8 | Improved |
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.87)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.87)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
