AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Accommodating Audio Modality in CLIP for Multimodal Processing

Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, Qin Jin

AAAI 2023 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 14 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add audio modality to CLIP for video retrievalCLIP4VLA · MSR-VTT-1kA and VATEXR@1: 45.6 -> 46.2; 57.3 -> 63.5Improved
Use audio and vision together instead of audio alone (ablation)CLIP4VLA · AudiocapsR@1: 28.4 -> 33.6Improved
Use audio and vision together instead of vision alone (ablation)CLIP4VLA captioning · MSR-VTTBLUE4: 44.4 -> 47.8Improved
Use audio and vision together instead of vision alone (ablation)CLIP4VLA captioning · VATEXCIDER: 58.5 -> 59.8Improved
Initialize the audio backbone with the vision backbone (ablation)CLIP4VLA audio encoder · MSR-VTT audio retrievalR@1: 2.5 -> 3.6Improved
Add inter-modal NCE alignment (ablation)CLIP4VLA audio encoder · MSR-VTT audio retrievalR@1: 3.6 -> 6.8Improved
Add intra-modal NCE learning (ablation)CLIP4VLA audio encoder · MSR-VTT audio retrievalR@1: 6.8 -> 10.5Improved
Add an audio type token (ablation)CLIP4VLA audio encoder · MSR-VTT audio retrievalR@1: 10.5 -> 10.6Improved
Add an audio type token (ablation)CLIP4VLA audio encoder · VATEX audio retrievalR@1: 9.3 -> 9.9Improved
Increase the [NB] proportion in the type-token mixture (ablation)CLIP4VLA · Audiocapsaudio retrievalImproved
Mix [NB] and [VB] embeddings at a 1:1 ratio (ablation)CLIP4VLA · MSR-VTT and VATEXaudio retrieval: 1:1Improved
Use Global-to-Local rather than Global-to-Global fusion (ablation)CLIP4VLA with vision and audio · MSR-VTT video retrievalR@1: 44.5 -> 46.8Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy