Research map / Video / Video-language understanding
Aligned Better, Listen Better for Audio-Visual Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dolphin, an AV-LLM with spatially fine-grained audio-visual multi-scale adapter and temporally interleaved audio-visual merging, trained on a new 5.2M-sample AVU audio-visual instruction dataset to reduce audio neglect and hallucination.
Problem
Existing Video-LLMs and AV-LLMs under-exploit audio, often ignoring it or hallucinating audio content, due to coarse audio-visual alignment and scarce audio-visual instruction data.
Approach
A multi-scale audio-visual adapter injects audio into visual features across ViT blocks via cross-attention (spatial alignment), and interleaved per-frame audio-visual tokens with bidirectional cross-attention merge them temporally; the AVU dataset deliberately includes samples where audio carries information beyond video and negative samples.
Key results
- Dolphin-13B reaches 74.8% accuracy on MSVD-QA and 63.5% on MSR-VTT-QA, outperforming listed Video-LLMs in the zero-shot table.
- On the AVU audio-visual bench, Dolphin achieves 78.2 accuracy vs 73.2 for Video-LLaMA 2 and 74.7 for video-SALMONN.
- Removing AV inter-merging drops AVU accuracy from 78.2 to 32.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial inter-modal interaction (ablation) | Dolphin · AVU | accuracy: 77.6 -> 78.2 | Helped |
| add the AV multi-scale adapter (ablation) | Dolphin · AVU | accuracy: 75.4 -> 78.2 | Helped |
| add bidirectional context attention (ablation) | Dolphin · AVU | accuracy: 76.1 -> 78.2 | Helped |
| add AV interleaved merging (ablation) | Dolphin · AVU | accuracy: 32.3 -> 78.2 | Helped |
| add the AV multi-scale adapter (ablation) | Dolphin · AVE | accuracy: 51.6 -> 52.1 | Helped |
| add bidirectional context attention (ablation) | Dolphin · AVE | accuracy: 50.3 -> 52.1 | Helped |
| add AV interleaved merging (ablation) | Dolphin · AVE | accuracy: 22.6 -> 52.1 | Helped |
| include AVU-Pretrain (ablation) | Dolphin · MSR-VTT-QA | accuracy: 58.8 -> 62.6 | Helped |
| include AVU-Specific (ablation) | Dolphin · MSR-VTT-QA | accuracy: 59.3 -> 62.6 | Helped |
| include AVU-Negatives (ablation) | Dolphin · MSR-VTT-QA | accuracy: 61.0 -> 62.6 | Helped |
| include AVU-Pretrain (ablation) | Dolphin · AV Understand | accuracy: 69.8 -> 78.2 | Helped |
| include AVU-Specific (ablation) | Dolphin · AV Understand | accuracy: 72.6 -> 78.2 | Helped |
Surprising findings
- Removing the temporal audio-visual inter-merging module collapses performance (AVU accuracy 32.3), far beyond other ablations.
- Using an ImageBind (audio-visually aligned) video encoder instead of CLIP lowers performance, suggesting language-aligned encoders matter more.
Limitations
- Dataset generation relies on proprietary or large expert models (GPT-4, InternVL-34B) and hallucination-prone captioners.
- Audio-visual bench is LLM-judged with GPT-4 scoring, and the bench is built from the authors' own transformed task annotations.
- Speech recognition lags Whisper-based LTU-AS, since the audio encoder is not speech-pretrained.
Experimental setup
Datasets: AVU AudioSet-2M VGG-Sound MUSIC Flickr-SoundNet VGG-SoundSource AVE AVS-dataset LLP LLaVA Valley Video-ChatGPT
Benchmarks: MSVD-QA MSR-VTT-QA ActivityNet-QA TGIF POPE ESC-50 AudioCaps Librispeech IEMOCAP Voxceleb2 GTZAN AVU (MUSIC-based bench)
Metrics: accuracy GPT score (1-5) SPICE WER MAE macro-F1
Baselines: Video-LLaMA Video-LLaMA 2 video-SALMONN AVicuna LTU-AS LTU VideoLLaVA PandaGPT
Builds on: ViT-Adapter LLaVA Video-LLaMA ImageBind
Closest papers by meaning
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.91)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.90)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.90)
- Aligning Audio-Visual Joint Representations with an Agentic Workflow (NeurIPS 2024 · similarity 0.90)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.89)
- SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.