Research map / Video / Video-language understanding
Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add supervised fine-tuning on the gesture training data (ablation) | InternVL3-8B · Reference task | accuracy: 66.1 -> 68.5 | Improved |
| add Hand Intent Token on top of SFT (ablation) | InternVL3-8B · Reference task | accuracy: 68.5 -> 75.0 | Improved |
| add Hand Intent Token on top of SFT (ablation) | InternVL3-8B · Temporal task | accuracy: 60.7 -> 66.1 | Improved |
| add synthetic videos to real training data (ablation) | HINT InternVL3-8B · Reference task | accuracy: 67.3 -> 75.0 | Improved |
| add real videos to synthetic-only training data (ablation) | HINT InternVL3-8B · Reference task | accuracy: 69.0 -> 75.0 | Improved |
| replace visual keypoint overlay with learned keypoint encoder (HINT) (ablation) | InternVL3-8B · Reference task | accuracy: 57.1 -> 75.0 | Improved |
| draw visual keypoints on frames (ablation) | SFT InternVL3-8B · Reference task | accuracy: 68.5 -> 57.1 | Hurt |
| replace text keypoint input with learned keypoint encoder (ablation) | InternVL3-8B · Reference task | accuracy: 69.0 -> 75.0 | Improved |
| raise hand detection threshold from 0.5 to 0.7 (ablation) | HINT InternVL3-8B · Reference task | accuracy: 75.0 -> 64.9 | Hurt |
| use oracle keyframes instead of uniform 32 frames (ablation) | InternVL3-8B · Reference task | accuracy: 66.1 -> 61.3 | Hurt |
| remove the pointing hand from video (ablation) | HINT InternVL3-8B · Reference task | accuracy: 75.0 -> 41.7 | Hurt |
| scale InternVL3 from 8B to 78B | InternVL3 · Reference task | accuracy: 66.1 -> 71.4 | Improved |
Closest papers by meaning
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.88)
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.88)
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.88)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.87)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.87)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
