Research map / Vision-language and multimodal / Multimodal LLMs

Multi-speaker Attention Alignment for Multimodal Social Interaction

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-free-style attention bias (no new parameters) that selects active cross-modal heads and boosts attention from each speaker's utterance tokens to that speaker's visual region in MLLMs, improving multi-speaker social interaction reasoning.

Problem

MLLMs fail to exploit visual cues in multi-speaker social interaction videos because cross-modal attention between speaker references in text and speaker regions in the image is weak and inconsistent.

Approach

Dynamic cross-modal head selection picks attention heads with high utterance-to-speaker-region attention, then adds an adaptive social-aware bias (scaled by the max existing attention to speaker regions) that shifts attention toward the current speaker's visual tokens, with no extra trainable parameters.

Key results

• Qwen2.5-VL+Ours reaches 87.3 on TVQA+ VideoQA vs 86.1 baseline (VLB) and 66.0 on MMSI mentioned-speaker prediction vs 62.4; average gain up to +3.2 points (InternVL3+Ours).
• InternVL3+Ours achieves 89.1 on TVQA+ and 79.7 on OnlineMMSI pronoun coreference, the best reported in those settings.
• Cross-modal AttnMax on MMSI rises from 4.54 to 17.09 (x10^-2) with the method, versus 9.23 on COCO.

Surprising findings

• Activating all heads with the bias degrades performance, attributed to disrupting heads that handle positional encoding or text modality.
• Adding visual prompts or box coordinates in text gives unstable gains across tasks, while the attention bias improves consistently.

Limitations

• Requires speaker bounding box annotations, which are needed to compute the speaker regions.
• Does not surpass state of the art on speaking target identification on MMSI, attributed to balancing attention between speaker and target.

Experimental setup

Datasets: TVQA+ MMSI OnlineMMSI

Benchmarks: TVQA+ MMSI OnlineMMSI

Metrics: accuracy AttnMax AttnMean

Baselines: MMSI OnlineMMSI Qwen2.5-VL LLaVA-NeXT-Video InternVL3 Qwen2.5-Text ST-VLM TLNet

Builds on: LoRA LLaMA-Factory Qwen2.5-VL LLaVA-NeXT-Video InternVL3

Closest papers by meaning

• Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.88)
• Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models (NeurIPS 2026 · similarity 0.88)
• MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.87)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
• The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.87)