Research map / Detection and segmentation / Open-set and few-shot detection
Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a part-aware bottom-up group reasoning DETR-style model that detects individuals, enriches them with pose-supervised body-part features, and associates individuals to groups via similarity, reaching SOTA on NVI-DET.
Problem
Fine-grained social interaction detection (NVI-DET): detecting individuals, their social groups, and nonverbal interactions as triplets, where prior methods use holistic person features and directly regress group boxes without modeling person-to-person relations.
Approach
Detect individuals first, enrich their embeddings with learnable body-part queries supervised by ViTPose-derived keypoint masks (pose as privileged information, used only in training), then build groups bottom-up via a learned similarity matrix between group queries and individual embeddings instead of direct group box regression.
Key results
- On NVI test, the method achieves 78.52 AR vs. 74.67 for NVI-DEHR (+3.85), and 63.59 mR@25 vs. 59.46.
- On Café (split by view), Group mAP1.0 of 18.23 vs. 14.36 for Café-base, and Group mAP0.5 46.88 vs. 37.52.
- LLaVA-1.6-vicuna-7B with ground-truth group crops reaches only 37.14 AR on NVI test; LoRA fine-tuning gives no additional gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use part-aware bottom-up group reasoning instead of the NVI-DEHR approach | NVI-DET · NVI val | AR: 71.20 -> 74.49 | Helped |
| Use part-aware bottom-up group reasoning instead of the NVI-DEHR approach | NVI-DET · NVI test | AR: 74.67 -> 78.52 | Helped |
| Apply the method frame-wise without temporal modeling | Café group activity detection · Café split by view | Group mAP@0.5: 37.52 -> 46.88 | Helped |
| Apply the method frame-wise without temporal modeling | Café group activity detection · Café split by place | Group mAP@0.5: 28.72 -> 39.03 | Helped |
| Fine-tune LLaVA with LoRA rank 8 | LLaVA-1.6-vicuna-7B · NVI test | AR | No clear effect |
| Add pose-guided supervision instead of CLIP text guidance (ablation) | Part-aware bottom-up model · NVI val | AR | Helped |
| Add the individual embedding enhancer for part-aware enrichment (ablation) | Part-aware bottom-up model · NVI val | AR | Helped |
| Use similarity-based association instead of conventional guided embedding (ablation) | Part-aware bottom-up model · NVI val | AR | Helped |
| Add the association loss (ablation) | Part-aware bottom-up model · NVI val | AR: 48.32 -> 74.49 | Helped |
| Add the part loss (ablation) | Part-aware bottom-up model · NVI val | AR: 73.58 -> 74.49 | Helped |
| Use triplet NMS instead of no NMS (ablation) | Part-aware bottom-up model · NVI val | AR: 63.53 -> 74.49 | Helped |
| Set pose-keypoint perturbation to ε=2.0 instead of ε=0.0 (ablation) | Part-aware bottom-up model · NVI val | AR: 74.49 -> 72.19 | Hurt |
Surprising findings
- Removing the part loss (lambda_p=0) yields only a slight drop, and the model performs well across lambda_p from 0 to 10.
- Using CLIP text-prompted body-part guidance is less effective than pose-based supervision.
Limitations
- Depends on an external pretrained pose estimator and predefined keypoints.
- Café evaluated frame-wise without temporal modeling.
Experimental setup
Datasets: NVI Café
Benchmarks: NVI Café
Metrics: mean Recall@K AR Group mAP Outlier mIoU
Baselines: m-QPIC m-CDN m-GEN-VLKT NVI-DEHR Joint JRDB-base HGC Café-base LLaVA-1.6-vicuna-7B CLIP-guided variant
Builds on: DETR NVI-DEHR guided embedding ViTPose
Closest papers by meaning
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.87)
- Omni-MMSI: Toward Identity-attributed Social Interaction Understanding (CVPR 2026 · similarity 0.86)
- Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.85)
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (NeurIPS 2024 · similarity 0.85)
- SocialGesture: Delving into Multi-person Gesture Understanding (CVPR 2025 · similarity 0.84)
- Spot The Ball: A Benchmark for Visual Social Inference (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.