Research map / Vision-language and multimodal / Multimodal LLMs

Omni-MMSI: Toward Identity-attributed Social Interaction Understanding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Omni-MMSI, a task for multi-party social interaction understanding from raw audio-video, and Omni-MMSI-R, a reference-guided pipeline that combines tool-extracted identity-attributed cues with LoRA-tuned Qwen2.5-Omni CoT reasoning.

Problem

Existing multi-party social interaction understanding assumes oracle-preprocessed, identity-attributed social cues; on raw audio-video, identity attribution (who speaks what, where people are) fails for extractors and Omni-LLMs, degrading referent prediction.

Approach

Use per-participant reference audio-image pairs to guide identity attribution via tools (Whisper, SpeechBrain, YOLO, OSNet), feed the attributed cues plus references and raw video to a fine-tuned Omni-LLM, and supervise with structured two-step CoT (last speaker confirmation, referent inference).

Key results

• Omni-MMSI-R reaches 43.06% average accuracy on Ego4D and 47.04% on YouTube, versus 37.70% and 44.80% for Gemini 2.5 Pro.
• Identity attribution average accuracy on Ego4D is 78.79% for Omni-MMSI-R versus 47.17% for the best referential Omni-LLM baseline (Gemini 2.5 Pro).
• Ablation on Ego4D: adding both references and tool-extracted cues plus 2-step CoT raises average accuracy from 33.97% (baseline) to 43.06%.

Surprising findings

• Adding a 3-step reasoning stage that requires the model to extract social cues itself drops average accuracy to 34.43%.
• Gemini 2.5 Pro attains higher PCR accuracy than Omni-MMSI-R on YouTube in the referential comparison table (48.72 vs 47.04 average), and Qwen3 Omni 30B's attribution drops when references are added.

Limitations

• Evaluated only on controlled Werewolf game datasets with all participants visible and a fixed setting.
• Relies on manually constructed reference pairs (69 profiles) and tool accuracy; small Omni-LLMs cannot exploit references well.
• LLM-based cue generation adds considerable inference latency.

Experimental setup

Datasets: Werewolf Among Us (YouTube subset) Werewolf Among Us (Ego4D subset)

Benchmarks: Werewolf Among Us Ego4D (STI, PCR) Werewolf Among Us YouTube (STI, PCR)

Metrics: accuracy verbal attribution accuracy non-verbal attribution accuracy

Baselines: Qwen2.5 Omni 7B Phi-4-Multimodal HumanOmni R1-Omni OmniVinci Qwen3 Omni 30B Gemini 2.5 Pro Lee et al. MMSI pipeline Li et al. MMSI pipeline

Builds on: Werewolf Among Us MMSI benchmark Qwen2.5-Omni LLaMA-Factory LoRA

Closest papers by meaning

• MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.87)
• Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.86)
• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.86)
• Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection (NeurIPS 2025 · similarity 0.86)
• SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.86)
• MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.85)