Research map / Video / Video-language understanding
EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EgoRe-5M, a 5M-QA egocentric dataset with chain-of-thought and hand-object/temporal grounding annotations, and trains Qwen2-VL-7B into EgoThinker via SFT followed by GRPO-based reinforcement fine-tuning.
Problem
Current MLLMs reason well about visible third-person events but struggle with egocentric video, which requires inferring the unobservable camera wearer's intentions, fine-grained hand-object interactions, and multi-minute temporal context.
Approach
Curate a large egocentric QA corpus from web and existing egocentric videos with explicit CoT rationales and dense spatio-temporal grounding, then apply SFT followed by rule-based-reward GRPO (IoU plus format reward) to sharpen hand-object and temporal localization.
Key results
- EgoThinker reaches 64.4 on EgoTaskQA (vs 57.9 for Qwen2-VL-7B) and 47.1 on EgoPlan-Val (vs 38.3), and 54.0 on VLN-QA (vs 42.0).
- On EK-Visor hand-object grounding, EgoThinker gets Loc-Acc 80.3 vs 71.7 for Qwen2.5-VL-72B, though its mIoU (53.7) is below Qwen2.5-VL-72B's 64.1.
- On EgoExoLearn temporal grounding, EgoThinker reaches mIoU 25.2 and R1@0.05 63.9, versus 21.1 and 49.9 for Qwen2.5-VL-72B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2-VL-7B · EgoPlan-Val | accuracy: 38.3 -> 47.1 | Helped |
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2-VL-7B · VLN-QA | accuracy: 42.0 -> 54.0 | Helped |
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2-VL-7B · EgoExoLearn | R1@0.05: 5.4 -> 63.9 | Helped |
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2VL · VideoMME | score: +0.0 | No clear effect |
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2VL · VideoHallucer | score: 47.6 -> 47.0 | Hurt |
| Train EgoThinker with the full SFT-RFT curriculum | Qwen2VL · POPE | score: 83.6 -> 86.8 | Helped |
| Add the EgoRe-5M short-term split in SFT (ablation) | EgoThinker · EgoTaskQA | accuracy: 57.7 -> 61.6 | Helped |
| Add the EgoRe-5M long-term split to short-term SFT (ablation) | EgoThinker · EgoSchema(val) | accuracy: 69.1 -> 71.1 | Helped |
| Add the CoT split to short-term and long-term SFT (ablation) | EgoThinker · QAEgo4D | accuracy: 63.7 -> 67.2 | Helped |
| Add the fine-grained grounding split and train with RFT (ablation) | EgoThinker · EK-Visor | mIoU: 28.5 -> 53.7 | Helped |
| Use RFT rather than SFT on EgoRe-5M-FG (ablation) | EgoThinker Stage1 · EgoExoLearn | mIoU: 9.84 -> 25.2 | Helped |
| Add grounding data (ablation) | EgoThinker · ERQA | accuracy: 40.1 -> 41.8 | Helped |
Surprising findings
- EgoThinker's EK-Visor mIoU (53.7) is lower than Qwen2.5-VL-72B (64.1) despite higher Loc-Acc, and it does not improve over Qwen2-VL-7B on VideoMME (both 62.9).
- Hallucination results are mixed: VideoHallucer drops slightly (47.6 to 47.0) while POPE improves (83.6 to 86.8).
Limitations
- Relies on extensive annotations and offline fine-tuning; no real-time inference in resource-constrained settings (stated in conclusion).
- Grounding data did not significantly improve high-level reasoning benchmarks, which the baseline already handles to some degree.
- Human validation covered only 500 sampled QA pairs (over 95% judged acceptable).
Experimental setup
Datasets: EgoRe-5M HowTo100M HTM-AA Howto-Interlink7M Ego4D EPIC-Kitchens EgoExoLearn EgoExo4D EK-Visor llavar-gpt4-20k ShareGPT4o ShareGPT4Video
Benchmarks: EgoTaskQA QAEgo4D ERQA EgoPlan-Val EgoSchema EgoMCQ VLN-QA RES EK-Visor EgoExoLearn MVBench Perception Test
Metrics: accuracy mIoU Loc-Acc R1@0.05
Baselines: Qwen2-VL-7B LLaVA-Video-7B LLaVA-OneVision-7B VideoLLaMA3 VideoChat2-HD InternVL2-8B Exo2Ego-7B Qwen2.5-VL-72B GPT-4o Grounding-DINO Gemini 2.5-Pro MiniCPM-V2.6
Builds on: GRPO DeepSeek-R1 Qwen2-VL HowTo100M Videochat2
Closest papers by meaning
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.91)
- EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.90)
- Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.90)
- EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding (ICML 2026 · similarity 0.89)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.