Research map / Vision-language and multimodal / Human and embodied modeling
Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces three multi-party social deduction tasks (speaking target identification, pronoun coreference resolution, mentioned player prediction) with new annotations on YouTube and Ego4D Werewolf data, and a baseline that densely aligns per-player visual pose features with utterances for fusion with BERT/RoBERTa/ELECTRA context.
Problem
Prior multimodal social-behavior work focuses on single persons or uses holistic visual features not aligned to utterances, so it cannot model fine-grained referent dynamics among multiple players in multi-party conversations.
Approach
Track each player with AlphaPose, align tracked player visuals to their names in utterances, and fuse per-utterance speaker kinesics and listener-position features with conversation-context language features in a multimodal transformer, with player-identity permutation training to avoid reliance on fixed IDs.
Key results
- On speaking target identification with BERT, the densely aligned baseline reaches 72.7% on YouTube vs 65.8% for BERT alone.
- On pronoun coreference resolution with RoBERTa, the baseline reaches 73.0% on YouTube vs 69.0% for RoBERTa alone.
- Competing holistic visual features (DINOv2, MViT) give little or no gain over language-only models across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use densely aligned language-visual representations instead of the strongest non-densely-aligned baseline | RoBERTa-based baseline · Speaking Target Identification, YouTube and Ego4D | accuracy: 73.1 -> 74.5; 64.2 -> 66.5 | Helped |
| Use densely aligned language-visual representations instead of the strongest non-densely-aligned baseline | RoBERTa-based baseline · Pronoun Coreference Resolution, YouTube and Ego4D | accuracy: 69.5 -> 73.0; 49.1 -> 52.7 | Helped |
| Use densely aligned language-visual representations instead of the strongest non-densely-aligned baseline | RoBERTa-based baseline · Mentioned Player Prediction, YouTube and Ego4D | accuracy: 60.7 -> 62.5; 51.3 -> 55.1 | Helped |
| Add visual features (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 65.8 -> 72.7; 60.3 -> 65.9; 54.6 -> 58.8 | Helped |
| Add gesture features (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 69.6 -> 72.7; 64.9 -> 65.9; 55.8 -> 58.8 | Helped |
| Add gaze features (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 70.2 -> 72.7; 66.5 -> 65.9; 56.2 -> 58.8 | Mixed |
| Add preceding conversation context (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 40.2 -> 59.1; 51.1 -> 63.4; 36.2 -> 47.3 | Helped |
| Add following conversation context to preceding context (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 59.1 -> 72.7; 63.4 -> 65.9; 47.3 -> 58.8 | Helped |
| Add player permutation learning (ablation) | BERT-based baseline · YouTube: Speaking Target Identification, Pronoun Coreference Resolution, Mentioned Player Prediction | accuracy: 70.8 -> 72.7; 51.5 -> 65.9; 52.7 -> 58.8 | Helped |
Surprising findings
- Holistic visual features from DINOv2 and MViT often fail to improve over language-only baselines and sometimes slightly hurt performance.
Limitations
- Evaluation limited to social deduction game settings (Werewolf and Avalon).
- Annotations for Ego4D use only 83 of 101 videos where individuals could be visually identified.
- Mentioned player prediction annotations are automatic and the Ego4D annotations are fewer.
Experimental setup
Datasets: YouTube Werewolf dataset (Lai et al. [28]) Ego4D Social subset (Werewolf and Avalon)
Benchmarks: YouTube Werewolf (new social tasks) Ego4D Social (new social tasks)
Metrics: accuracy Krippendorff's alpha
Baselines: BERT RoBERTa ELECTRA BERT + DINOv2 BERT + MViT (Lai et al. [28]) RoBERTa + DINOv2 RoBERTa + MViT ELECTRA + DINOv2 ELECTRA + MViT
Builds on: BERT RoBERTa ELECTRA AlphaPose MViT
Closest papers by meaning
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.90)
- Omni-MMSI: Toward Identity-attributed Social Interaction Understanding (CVPR 2026 · similarity 0.86)
- VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence (AAAI 2025 · similarity 0.86)
- Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.86)
- MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.86)
- SocialGesture: Delving into Multi-person Gesture Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.