Research map / Video / Video-language understanding
Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVIGATE, a video-text retrieval framework that uses a gated fusion transformer to selectively add audio (AST) features to CLIP frame embeddings, plus an adaptive-margin contrastive loss, reaching SOTA on MSR-VTT, VATEX, and Charades.
Problem
Video-text retrieval often ignores audio, and existing audio-fusion methods blindly use audio even when it is irrelevant (e.g., background noise), which degrades video representations; joint video-text processing also makes retrieval costly.
Approach
A gated fusion transformer whose MHA and FFN outputs are scaled by learned gates computed from pooled audio and frame embeddings, so audio influence is suppressed when uninformative; video and text are encoded independently for efficient retrieval. Adds an adaptive margin in the contrastive loss derived from intra-modal similarities of negative pairs.
Key results
- AVIGATE reaches 50.2 R@1 text-to-video on MSR-VTT with CLIP ViT-B/32 (vs. TEFAL 49.4) and 52.1 with ViT-B/16 (vs. UATVR 50.8).
- On VATEX with ViT-B/16, text-to-video R@1 is 67.5 vs. 64.5 for UATVR.
- Retrieval latency is 0.16 ms similarity time vs. 130.83 ms for TEFAL, reported as over 14x faster than TEFAL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio using a simplified fusion transformer (ablation) | CLIP video-text retrieval baseline on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 45.4 -> 47.1 | Helped |
| Add adaptive margin-based contrastive loss without gated fusion (ablation) | Audio-enhanced CLIP baseline on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 45.4 -> 48.9 | Helped |
| Add gated fusion without a contrastive-loss margin (ablation) | Audio-enhanced CLIP baseline on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 45.4 -> 48.0 | Helped |
| Use a fixed margin of 0.1 instead of an adaptive margin (ablation) | Gated Fusion on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 48.0 -> 49.0 | Helped |
| Replace the fixed margin with an adaptive margin (ablation) | Gated Fusion on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 49.0 -> 50.2 | Helped |
| Add gated fusion and adaptive margin together (ablation) | Audio-enhanced CLIP baseline on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 47.1 -> 50.2 | Helped |
| Combine global and local alignment instead of global alignment only (ablation) | AVIGATE on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 46.1 -> 50.2 | Helped |
| Add global alignment to local alignment (ablation) | AVIGATE on MSR-VTT 9k · MSR-VTT 9k, text-to-video | R@1: 48.5 -> 50.2 | Helped |
| Use AVIGATE instead of TEFAL | Video-text retrieval on MSR-VTT 9k with CLIP ViT-B/32 · MSR-VTT 9k, text-to-video | R@1: 49.4 -> 50.2 | Helped |
| Use AVIGATE instead of TEFAL | Video-text retrieval on MSR-VTT 9k with CLIP ViT-B/32 · MSR-VTT 9k, video-to-text | R@1: 47.1 -> 49.7 | Helped |
| Use AVIGATE instead of UATVR | Video-text retrieval on VATEX with CLIP ViT-B/16 · VATEX, text-to-video | R@1: 64.5 -> 67.5 | Helped |
| Use AVIGATE instead of TEFAL | Video-text retrieval on Charades with CLIP ViT-B/32 · Charades, text-to-video | R@1: 18.2 -> 18.8 | Helped |
Surprising findings
- Naively adding audio improves R@1 by 1.7 points on MSR-VTT, but the paper notes irrelevant audio (e.g., background music) can degrade alignment, motivating gating.
Limitations
- Audio encoder AST is frozen and zero vectors are used for videos without audio
- Gains on Charades and some R@K metrics are small or mixed (e.g., R@10 lower than some baselines)
Experimental setup
Datasets: MSR-VTT VATEX Charades ImageNet AudioSet
Benchmarks: MSR-VTT 9k VATEX Charades
Metrics: R@1 R@5 R@10 RSum latency
Baselines: CLIP4Clip X-Pool TS2-Net UATVR ProST UCoFiA TEFAL ECLIPSE
Builds on: CLIP4Clip CLIP Audio Spectrogram Transformer TEFAL ECLIPSE
Closest papers by meaning
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.87)
- Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions (CVPR 2025 · similarity 0.87)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.86)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.86)
- DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval (AAAI 2024 · similarity 0.86)
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.