Research map / Large language models / AI and society
MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MimeQA, a 806-question open-ended video QA benchmark built from mime videos to test nonverbal social reasoning, showing state-of-the-art VideoLLMs score 20-38% versus 86% for humans.
Problem
Current social intelligence benchmarks for AI are language-dominant, leaving models weak at understanding nonverbal social interactions; mime performances offer a language-free testbed.
Approach
Use mime videos (no speech, no props) as a novel source for a hierarchical video QA benchmark (grounding the imagined, scene-level, global-level social reasoning), with text-only ablations and title/OCR analyses to expose language bias.
Key results
- Open-source VideoLLMs reach roughly 20% average accuracy and GPT-4o 31.3%, Gemini-2.5-Pro 38.3%, versus 86.0% human accuracy on MimeQA.
- Fine-tuning Qwen2.5-VL-72B on 80% of MimeQA raised held-out accuracy from 22.5% to 26.6%, with Working Memory rising from 23.5% to 47.1%.
- Removing video (text-only) changes accuracy only 1-7% for open-source models, and the LLM grader agreed with humans on 92.0% of 352 questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-VL on 80% of MimeQA | Qwen2.5-VL · MimeQA test | accuracy: 22.5% to 26.6% | Helped |
| Add timestamped PoseC3D action labels (ablation) | Qwen2.5-VL-72B · MimeQA subset, overall | accuracy: 16.84% to 16.16% | Hurt |
| Add timestamped PoseC3D action labels (ablation) | Qwen2.5-VL-72B · MimeQA subset, Grounding the Imagined | accuracy: 2.33% to 6.98% | Helped |
| Add timestamped PoseC3D action labels (ablation) | Qwen2.5-VL-72B · MimeQA subset, Working Memory | accuracy: 14.29% to 23.81% | Helped |
| Add timestamped PoseC3D action labels (ablation) | Qwen2.5-VL-72B · MimeQA subset, Theory of Mind | accuracy: 52.17% to 34.78% | Hurt |
| Add timestamped PoseC3D action labels (ablation) | Qwen2.5-VL-72B · MimeQA subset, Social Judgment | accuracy: 34.48% to 24.14% | Hurt |
| Include video title in the prompt (ablation) | Qwen2.5-VL · MimeQA | accuracy: 20.1 to 24.3 | Helped |
| Add chain-of-thought prompting (ablation) | Qwen2.5-VL · MimeQA | accuracy: 20.1% to 22.2% | Helped |
| Add chain-of-thought prompting (ablation) | LLaVA-Video · MimeQA | accuracy: 19.4% to 17.7% | Hurt |
| Add chain-of-thought prompting (ablation) | InternVL2.5 · MimeQA | accuracy: 21.6% to 18.7% | Hurt |
| Add chain-of-thought prompting (ablation) | VideoLLaMA3 · MimeQA | accuracy: 22.2% to 17.7% | Hurt |
| Add chain-of-thought prompting (ablation) | GPT-4o · MimeQA | accuracy: 31.3% to 29.5% | Hurt |
Surprising findings
- Chain-of-thought prompting gave no meaningful accuracy gain and decreased performance for most models.
- Several open-source models scored higher on text-only input than with video on grounding questions, indicating they fail to use visual information.
Limitations
- Dataset is small (806 QA over 101 videos), limiting training-based methods.
- Videos primarily represent Western-style or select cultural contexts; cultural coverage is limited.
- Mime-specific genre; findings may not transfer to other expressive performance forms.
Experimental setup
Datasets: MimeQA Social-IQ 2.0 IntentQA NTU RGB+D
Benchmarks: MimeQA Social-IQ 2.0 IntentQA
Metrics: accuracy LLM-judge accuracy
Baselines: Qwen2.5-VL LLaVA-Video InternVL2.5 VideoLLaMA3 GPT-4o Gemini-1.5-Pro Gemini-2.5-Pro GPT-o3
Builds on: Social-IQ 2.0 IntentQA chain-of-thought prompting
Closest papers by meaning
- SocialGesture: Delving into Multi-person Gesture Understanding (CVPR 2025 · similarity 0.87)
- Omni-MMSI: Toward Identity-attributed Social Interaction Understanding (CVPR 2026 · similarity 0.87)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.87)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.86)
- VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence (AAAI 2025 · similarity 0.86)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.