Research map / Video / Video-language understanding
SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval
Ruixiang Zhao, Zhihao Xu, Bangxiang Lan, Zijie Xin, Jingyu Liu, Xirong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dedicated speech branch (Whisper ASR + CLIP text encoder) (ablation) | SAVE · MSRVTT-9k | SumR: 211.9 -> 216.2 | Improved |
| add audio (sound) branch (ablation) | SAVE · MSRVTT-9k | SumR: 207.5 -> 216.2 | Improved |
| add soft-ALBEF early vision-audio alignment (ablation) | SAVE · MSRVTT-9k | SumR: 213.8 -> 216.2 | Improved |
| use hard-label ALBEF instead of no alignment (ablation) | SAVE · MSRVTT-9k | SumR: 213.8 -> 211.2 | Hurt |
| use soft labels instead of hard labels in ALBEF (ablation) | SAVE · MSRVTT-9k | SumR: 211.2 -> 216.2 | Improved |
| use Pearson distance instead of MSE (ablation) | soft-ALBEF · MSRVTT-9k | SumR: 215.0 -> 216.2 | Improved |
| use ImageBind instead of AudioCLIP as soft-label teacher (ablation) | soft-ALBEF · MSRVTT-9k | SumR: 215.8 -> 216.2 | Improved |
| use early fusion instead of late fusion for speech (ablation) | SAVE · MSRVTT-9k | SumR: 206.5 -> 216.2 | Improved |
| use parameter-free fusion instead of learnable weights (ablation) | SAVE · MSRVTT-9k | SumR: 215.5 -> 216.2 | No clear effect |
| share CLIP text encoder between speech and query (ablation) | SAVE · MSRVTT-9k | SumR: 214.8 -> 216.2 | Improved |
| use Whisper instead of SenseVoice for ASR (ablation) | SAVE · MSRVTT-9k | SumR: 215.4 -> 216.2 | Improved |
| add speech branch and soft-ALBEF to AVIGATE | AVIGATE · Charades | SumR: 110.6 -> 121.4 | Improved |
Closest papers by meaning
- Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval (CVPR 2025 · similarity 0.88)
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.87)
- Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions (CVPR 2025 · similarity 0.86)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
