Research map / Video / Video-language understanding
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes video-SALMONN, a single end-to-end audio-visual LLM that understands speech, audio events, music and visual frames via a multi-resolution causal Q-Former, diversity loss, and unpaired audio-visual mixed training.
Problem
Existing audio-visual LLMs ignore or poorly model speech in videos, and lack fine-grained temporal audio-visual interaction needed for speech-aware video understanding.
Approach
A multi-resolution causal Q-Former that applies sliding-window Q-Formers at 0.5 s and 5 s scales over time-synchronised Whisper, BEATs and InstructBLIP features, with a causal self-attention mask, a diversity loss on output queries, and mixed unpaired audio-visual training to avoid modality dominance.
Key results
- video-SALMONN 13B reaches 49.6% Video QA accuracy on NExT-QA versus 21.0% for InstructBLIP 13B and 24.7% for the fine-tuned InstructBLIP.
- On AVQA, video-SALMONN 13B reaches 49.8% (Ego4D-QA) and 70.5% (Presentation-QA) versus 18.2% and 21.3% for Video-LLaMA 7B.
- video-SALMONN 13B achieves 2.6% WER on LibriSpeech test-clean and 7.7% WER on AVSR (How2), versus 2.9% and 8.3% for Whisper large-v2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use video-SALMONN instead of InstructBLIP 13B fine-tuned on the same data | NExT-QA test · NExT-QA test | Accuracy: 24.7% -> 49.6% | Helped |
| use video-SALMONN instead of Video-LLaMA 7B | Ego4D-QA · Ego4D-QA | Accuracy: 18.2% -> 49.8% | Helped |
| use video-SALMONN instead of Video-LLaMA 7B | Presentation-QA · Presentation-QA | Accuracy: 21.3% -> 70.5% | Helped |
| remove 5s-resolution (ablation) | video-SALMONN · NExT-QA test | Accuracy: 49.6% -> 47.2% | Hurt |
| remove 5s-resolution (ablation) | video-SALMONN · Ego4D-QA + Presentation-QA | Accuracy: 60.2% -> 57.2% | Hurt |
| remove 0.5s-resolution (ablation) | video-SALMONN · NExT-QA test | Accuracy: 49.6% -> 49.9% | No clear effect |
| remove 0.5s-resolution (ablation) | video-SALMONN · Ego4D-QA + Presentation-QA | Accuracy: 60.2% -> 58.9% | Hurt |
| remove mixed training scheme (ablation) | video-SALMONN · NExT-QA test | Accuracy: 49.6% -> 46.9% | Hurt |
| remove mixed training scheme (ablation) | video-SALMONN · Ego4D-QA + Presentation-QA | Accuracy: 60.2% -> 54.0% | Hurt |
| remove diversity loss (ablation) | video-SALMONN · NExT-QA test | Accuracy: 49.6% -> 49.3% | No clear effect |
| remove diversity loss (ablation) | video-SALMONN · Ego4D-QA + Presentation-QA | Accuracy: 60.2% -> 53.5% | Hurt |
| remove MRC Q-Former (ablation) | video-SALMONN · NExT-QA test | Accuracy: 49.6% -> 42.7% | Hurt |
Surprising findings
- Removing the 5-second resolution level improves ASR/AVSR slightly, while removing the 0.5-second level slightly improves Video QA, showing the two resolutions specialise in speech and high-level visual content.
- Using Llama-2 instead of Vicuna-v1.5 as the backbone drops Video QA from 49.6% to 36.7% and AVQA (Ego4D-QA) from 49.8% to 40.6%.
Limitations
- Strong diversity loss weight causes LLM hallucinations such as high WER insertion rates.
- Evaluated on short videos; higher frame rates are supported but cost more computation and storage.
Experimental setup
Datasets: LibriSpeech AudioCaps LLaVA-150k OCRVQA TextCaps NExT-QA COCO spoken captions VideoChat Ego4D How2 AVSD MSVD-QA
Benchmarks: SAVE LibriSpeech test-clean AudioCaps Flickr30k TextVQA GQA NExT-QA How2 Ego4D-QA Presentation-QA VGGSS SpokenCOCO
Metrics: WER SPIDEr CIDEr accuracy top-1 accuracy
Baselines: Whisper large-v2 InstructBLIP 13B InstructBLIP 13B fine-tuned Video-LLaMA 7B AV-LLM Llama-2 backbone variant
Builds on: Q-Former SALMONN InstructBLIP Video-LLaMA LoRA
Closest papers by meaning
- video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model (ICML 2025 · similarity 0.92)
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.90)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.89)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.