Research map / Video / Video-language understanding
Interacted Object Grounding in Spatio-Temporal Human-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GIO, a third-view spatio-temporal HOI benchmark with 1,098 interacted-object classes and 290K boxes, and a 4D question-answering framework (4D-QA) that selects SAM mask proposals using video, SlowFast, and reconstructed 4D human-object layout features.
Problem
Existing video HOI benchmarks provide few predefined object classes, which limits open-world interacted-object understanding; the paper targets localizing diverse, rare interacted objects in third-person videos given the human tracklet.
Approach
Reformulate interacted object localization as a multi-option question answering over SAM mask candidates, using 2D video features plus 3D human-scene layout features (SMPL humans, ZoeDepth scene, BPS encoding) processed by 2D and 3D transformer decoders, with a distance prior between humans and masks.
Key results
- 4D-QA reaches 23.38 mAP@0.5 and 29.71 mIoU_w on GIO, versus 17.53 mAP@0.5 and 20.41 mIoU_w for the best baseline Grounding DINO.
- Removing the 3D human-object feature drops 4D-QA to 22.67 mAP@0.5 and 28.76 mIoU_w.
- On VidHOI, GIO-pretrained 4D-QA reaches 25.35 mAP and 29.61 mIoU_w after 10-epoch finetuning, while Grounding DINO gets 15.87 mAP and 17.57 mIoU_w.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 4D-QA instead of the strongest listed baseline, Grounding DINO | GIO interacted object grounding · GIO test | mAP@0.5: 17.53 -> 23.38 | Helped |
| Use 4D-QA instead of the strongest listed baseline, Grounding DINO | GIO interacted object grounding · GIO test | mIoU_w: 20.41 -> 29.71 | Helped |
| Finetune GIO-pretrained 4D-QA for 10 epochs | 4D-QA on VidHOI · VidHOI test | mAP: 14.23 -> 25.35 | Helped |
| Add the 3D feature (ablation) | 4D-QA · GIO test | mAP@0.5: 22.67 -> 23.38 | Helped |
| Use GIoU distance instead of L2 distance (ablation) | 4D-QA · GIO test | mAP@0.5: 22.34 -> 23.38 | Helped |
| Add human-object distance scoring (ablation) | 4D-QA · GIO test | mAP@0.5: 22.07 -> 23.38 | Helped |
| Replace the Grounding DINO interaction feature with Bert embedding (ablation) | 4D-QA · GIO test | mAP@0.5: 20.02 -> 23.38 | Hurt |
| Replace the Grounding DINO interaction feature with CLIP embedding (ablation) | 4D-QA · GIO test | mAP@0.5: 20.00 -> 23.38 | Hurt |
| Add the interaction/action feature (ablation) | 4D-QA · GIO test | mAP@0.5: 19.04 -> 23.38 | Helped |
| Use ground-truth rather than predicted human boxes, with action input retained (ablation) | 4D-QA · GIO test | mAP@0.5: 18.92 -> 23.38 | Helped |
| Use SAM-generated mask candidates instead of direct box regression (ablation) | 4D-QA · GIO test | mAP@0.5: 18.82 -> 23.38 | Helped |
Surprising findings
- Qwen-VL gets decent mIoU_w (20.28) but poor mAP, indicating it localizes approximate positions but not precise boxes.
- Grounding DINO slightly outperforms zero-shot 4D-QA on mAP on VidHOI because that dataset has only about 1.22 HOIs per frame.
Limitations
- Performance is sub-optimal for small objects, attributed to mismatch between annotations and SAM masks.
- The 3D component is limited by over-flat monocular depth estimates and imprecise human reconstruction.
- LLM-based grounding (Qwen-VL) performs relatively poorly on interactiveness reasoning.
Experimental setup
Datasets: AVA 2.2 VidHOI COCO
Benchmarks: GIO VidHOI
Metrics: mAP@0.5 mAP@0.6 mAP@0.7 mAP@0.8 mAP@0.9 weighted mIoU (mIoU_w)
Baselines: PViC Gaze Detic CG-STVG Grounding DINO Qwen-VL
Builds on: SAM SlowFast Grounding DINO DJ-RN PHALP
Closest papers by meaning
- ContextHOI: Spatial Context Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.87)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.86)
- BOP-ASK: Object-Interaction Reasoning for Vision-Language Models (CVPR 2026 · similarity 0.86)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.86)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.86)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.