Research map / Vision-language and multimodal / Cross-modal learning
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mechanistic study of how audio and visual information flows through audio-visual LLMs via attention knockout, finding a sequential pathway for AV videos and parallel paths for interleaved multi-input, and showing many multimodal/text tokens can be discarded mid-network without accuracy loss.
Problem
How audio and visual information travels through audio-visual LLMs (AVLLMs) to shape the final answer, for both single audio-visual video and multi-input interleaved audio-visual items, remains poorly understood.
Approach
Applies attention knockout with sliding layer windows to trace audio-visual information flow in AVLLMs, showing attention maps are misleading due to vision sinks in late layers, and that tokens can be discarded once their information has transferred to an aggregation token.
Key results
- Masking video and audio attention at layers 31+ leaves AV-SpeakerBench accuracy unchanged or slightly improved (42.24 to 42.52 for masking video and audio for all text).
- Discarding video and audio tokens at layer 26 leaves Speech Recognition accuracy at 50.25 to 50.75 for discarding video, with prefill latency reduced from 2288.65 s to 2098.75 s (reported in the table units).
- Flow in single AV videos follows Modalities to Question to Last; in multi-input settings, two parallel paths (Candidates+Question to Reference to Last, and Candidates to Option letters to Last) carry the information.
Surprising findings
- Attention to video tokens at late layers is dominated by vision sinks that carry no meaningful information, so attention maps are unreliable indicators of information flow.
- Discarding audio, visual, or question tokens after their information transfer often slightly improves accuracy rather than hurting it.
Limitations
- Analysis restricted to multiple-choice QA where prediction is a single answer letter; open-ended generation not studied.
- Per-task knockout sample sizes are small on WorldSense, making curves noisy.
- Main analysis uses a single 3B model; generality to other AVLLMs is only partly covered.
Experimental setup
Datasets: AV-SpeakerBench AV-Odyssey WorldSense
Benchmarks: AV-SpeakerBench AV-Odyssey WorldSense
Metrics: relative change in answer-letter probability accuracy avg prefill latency
Baselines: Video-SALMONN2 Plus
Builds on: attention knockout information flow in VLMs information flow in VideoLLMs
Closest papers by meaning
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.90)
- AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.90)
- Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.89)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.89)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.