Research map / Vision-language and multimodal / Multimodal LLMs
Read the Room: Video Social Reasoning with Mental-Physical Causal Chains
Lixing Niu, Jiapeng Li, Xingping Yu, Xinyi Dong, Shu Wang, Ruining Feng, Bo Wu, Ping Wei, Yisen Wang, Lifeng Fan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces R^3-Bench, a video social reasoning benchmark built around mental-physical causal chains for inferring others' mental states from social cues.
Problem
Existing social reasoning datasets are limited in complexity, scale, and mental-state coverage, falling short of the causal dynamics of real-life interactions, so AI systems struggle to "read the room" from video.
Approach
A video social reasoning benchmark (R^3-Bench) whose items are structured around causal chains linking mental states to physical cues, intended to be richer in complexity, scale, and mental-state coverage than prior datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| SFT on R3-FDT | Qwen2-VL-7B · R3-Bench-DX | overall accuracy: 74.90 -> 88.74 | Improved |
| GRPO (RLFT) on R3-FDT | Qwen2-VL-7B · R3-Bench-DX | overall accuracy: 74.90 -> 86.88 | Improved |
| SFT on R3-FDT | Qwen2-VL-7B · R3-Bench-Hard | accuracy: 31.96 -> 42.09 | Improved |
| GRPO (RLFT) on R3-FDT | Qwen2-VL-7B · R3-Bench-Hard | accuracy: 31.96 -> 39.87 | Improved |
| SFT on R3-FDT | Qwen2-VL-7B · R3-Bench-DX chain consistency | chain consistency: 12.68 -> 35.73 | Improved |
| GRPO (RLFT) on R3-FDT | Qwen2-VL-7B · Social-IQ 2.0 | accuracy: 61.51 -> 68.40 | Improved |
| SFT on R3-FDT | Qwen2-VL-7B · Social-IQ 2.0 | accuracy: 61.51 -> 65.38 | Improved |
| GRPO (RLFT) on R3-Bench-FDT | Qwen2-VL-7B · IntentQA | accuracy: 82.38 -> 90.58 | Improved |
| SFT on R3-FDT | Qwen2-VL-7B · IntentQA | accuracy: 82.38 -> 87.77 | Improved |
| add subtitles to input | Gemini 2.5 Pro · R3-Bench-DX | overall accuracy: 80.79 -> 86.34 | Improved |
| add subtitles to input | GPT-4o · R3-Bench-DX | overall accuracy: 82.23 -> 82.64 | Improved |
| per-question accuracy vs chain-level consistency | GPT-4o · R3-Bench-DX | chain consistency: 82.64 accuracy vs 25.36 consistency | Hurt |
Experimental setup
Benchmarks: R^3-Bench
Closest papers by meaning
- Language Models Represent Beliefs of Self and Others (ICML 2024 · similarity 0.88)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.88)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.88)
- VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence (AAAI 2025 · similarity 0.88)
- Spot The Ball: A Benchmark for Visual Social Inference (CVPR 2026 · similarity 0.87)
- MuMA-ToM: Multi-modal Multi-Agent Theory of Mind (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
