Research map / Vision-language and multimodal / Cross-modal learning

Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual speech separation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mechanistically analyzes trimodal (text-audio-visual) binding in AVLLMs via symbolic temporal/position IDs, finds failures at audio-visual alignment, and proposes ASD-based bounding-box prompting (training-free and LoRA fine-tuned) that improves multi-speaker benchmarks.

Problem

AVLLMs struggle to resolve "who says what" in multi-speaker conversational videos, which requires binding text, audio and visual attributes; the internal mechanism and failure point of this trimodal binding are unclear.

Approach

Identify a three-stage symbolic trimodal binding mechanism (anchor ID retrieval, target ID selection, feature retrieval) using modality-specific symbolic IDs (temporal IDs for audio, position IDs for vision), localize failures to audio-visual target ID selection, and fix it by overlaying red boxes on active speakers from an off-the-shelf ASD model, optionally with LoRA fine-tuning.

Key results

• Training-free ASD raises Qwen2.5-Omni (7B) AVSpeaker accuracy from 44.86% to 45.59%, and ASD-FT reaches 48.79%.
• ASD-FT improves video-SALMONN2+ on DailyOmni from 65.13% to 69.13% and on AVSpeaker from 44.74% to 49.10%.
• Patching correct target-ID attention heads into unprimed runs raises Qwen2.5-Omni (7B) accuracy from 29.6% to 49.0%.

Surprising findings

• Training-free ASD gains are small on AVSpeaker for Qwen2.5-Omni (+0.7 points), while ASD-FT also improves general benchmarks such as WorldSense and DAVE despite no bounding boxes there.
• Inactive-speaker or random bounding boxes degrade performance, so gains are attributed to accurate active-speaker localization.

Limitations

• Analysis uses a toy dataset with animal or synthetic human speakers and four-speaker grids, limiting ecological validity.
• Unprimed accuracy of AVLLMs is below 30%, so mechanistic analysis relies on primed contexts.
• Gains depend on the external ASD model and its bounding box quality; some tasks show mixed results (e.g., DiaDemBench REF drops for video-SALMONN2+ with training-free ASD).

Experimental setup

Datasets: AVSpeaker DailyOmni SocialOmni DiaDemBench OmniBench DAVE WorldSense

Benchmarks: AVSpeaker DailyOmni SocialOmni DiaDemBench OmniBench DAVE WorldSense

Metrics: Accuracy REF ASR RSA correlation CM score

Baselines: AVCD FMD OutRo NumPro VISER

Builds on: visual prompting symbolic binding IDs causal mediation analysis representational similarity analysis

Closest papers by meaning

• When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion? (AAAI 2026 · similarity 0.87)
• From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.87)
• HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.86)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.86)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.86)