Research map / Vision-language and multimodal / Visual question answering
SocialGesture: Delving into Multi-person Gesture Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SocialGesture, a multi-person video dataset of four deictic social gestures with temporal, bounding-box, interaction and VQA annotations, and benchmarks action recognition, localization and VLMs on it.
Problem
Existing gesture datasets are single-person or controlled, lacking natural multi-person social interactions, which limits understanding of gestures in social context and alignment with language.
Approach
A large-scale multi-person gesture dataset sourced from YouTube and Ego4D covering pointing, showing, giving and reaching gestures, with multi-level annotations and a novel gesture VQA benchmark for VLMs.
Key results
- Best four-class gesture classification reached only 56.18% top-1 (VideoSwin-L); per-person cropping raised UniFormerV2-B/16 to 64.72%.
- Temporal localization averaged only 14.73 mAP (VideoMAEV2) and 10.73 (I3D) across thresholds 0.3-0.7.
- Zero-shot VLMs struggled on gesture localization (e.g., Qwen2.5-VL-72B 54.82% GestureLoc; no model above 70% on human counting); LoRA fine-tuning of Qwen2-VL-7B raised TargetLoc to 84.52%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce I3D sliding-window stride from 16 to 4 (ablation) | ActionFormer temporal localization · SocialGesture | Avg mAP: 5.94 -> 19.19 | Helped |
| Use per-person ground-truth bounding-box crops (ablation) | SocialGesture four-gesture classification · SocialGesture | Top1 accuracy (%): 55.51 -> 64.72 | Helped |
| Fine-tune Qwen2-VL-7B with LoRA SFT (ablation) | Qwen2-VL-7B · SocialGesture VQA | accuracy (%) across subtasks: 60.18 -> 82.55; 74.15 -> 33.07; 67.18 -> 84.52; 9.58 -> 49.21; 59.19 -> 41.19 | Mixed |
| Fine-tune Qwen2-VL-7B with full SFT (ablation) | Qwen2-VL-7B · SocialGesture VQA | accuracy (%) across subtasks: 60.18 -> 83.64; 74.15 -> 33.07; 67.18 -> 81.13; 9.58 -> 51.25; 59.19 -> 38.51 | Mixed |
| Use VideoMAEV2 features instead of I3D features | ActionFormer temporal localization · SocialGesture | Avg mAP: 10.73 -> 14.73 | Helped |
Surprising findings
- No evaluated VLM exceeded 70% accuracy on counting humans in social scenes, despite its simplicity.
- Pretrained video models that excel on THUMOS14 and ActivityNet perform poorly on social gesture localization.
Limitations
- Gesture localization and recognition remain hard for current models, with low mAP and accuracy.
- VQA annotations partly generated by GPT-4o and reviewed by humans, which may introduce bias.
- Dataset is restricted to four deictic gesture categories from YouTube and Ego4D sources.
Experimental setup
Datasets: SocialGesture Kinetics-400 THUMOS14 ActivityNet-1.3
Benchmarks: SocialGesture
Metrics: mAP@[0.3:0.1:0.7] Accuracy Top-1 accuracy
Baselines: TSN TANet SlowFast TimeSformer MViTv2 VideoSwin UniFormerV2 Qwen2-VL Qwen2.5-VL InternVL-2.5 LLaVA-NeXT-Video Claude 3.7 Sonnet
Builds on: ActionFormer LLaVA LoRA
Closest papers by meaning
- MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.87)
- Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception (NeurIPS 2026 · similarity 0.86)
- Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.86)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.85)
- Pose Priors from Language Models (CVPR 2025 · similarity 0.85)
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.