Research map / Vision-language and multimodal / Visual question answering
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoThink, a 700-image, open-ended VQA benchmark from Ego4D frames testing first-person thinking across six capabilities and twelve dimensions, scoring 21-ish VLMs with GPT-4 as judge.
Problem
Existing VLM evaluation benchmarks focus mainly on third-person perspective, and first-person (egocentric) thinking capabilities needed for embodied agents and robotics remain largely unexplored.
Approach
A manually annotated egocentric VQA benchmark organized into six core capabilities (object, activity, localization, reasoning, forecasting, planning) with twelve fine-grained dimensions, evaluated with open-ended answers graded by GPT-4 single-answer grading (0/0.5/1) and validated against humans and other LLM judges.
Key results
- GPT-4V achieves the best average score of 65.5 on EgoThink, leading in most dimensions but trailing BLIP-2-11B on localization.
- Counting is the hardest dimension, with the best score being GPT-4V's 42.0; forecasting's best is 55.0.
- GPT-4 judge correlates with humans at Pearson 0.68, matching Claude-2 (0.68) and above GPT-3.5-Turbo (0.43).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the LLM parameter scale from ∼7B to ∼13B in LLaVA | LLaVA · EgoThink | Average score | Helped |
| Increase the LLM parameter scale from ∼7B to ∼13B in LLaVA-1.5 | LLaVA-1.5 · EgoThink | Average score | Helped |
| Increase LLM scale from ∼7B to ∼13B in PandaGPT and InstructBLIP | PandaGPT and InstructBLIP · EgoThink | Average score | No clear effect |
| Apply instruction tuning and additional instruction-aware tokens (ablation) | BLIP-2-11B · EgoThink | Average score: 49.6 -> 51.5 | Helped |
| Add segmentation marks as visual grounding information | GPT-4V · EgoThink | Object localization correctness | Helped |
Surprising findings
- Enlarging the trainable parameters of the LLM has the largest effect on performance, while simply scaling LLMs without better alignment may not help (PandaGPT, InstructBLIP).
- GPT-4V is outperformed by the open-source BLIP-2-11B on localization, including location and spatial relationship dimensions.
Limitations
- Only images are used because most VLMs do not support video input; video-based evaluation is deferred to future work.
- Benchmark size is small (700 images), constrained by the cost of open-ended GPT-4 evaluation.
- GPT-4 judge is less reliable for planning answers that are long and detailed.
Experimental setup
Datasets: Ego4D
Benchmarks: EgoThink
Metrics: single-answer grading score (0/0.5/1) Pearson correlation with human judgments Cohen's kappa
Baselines: GPT-4V OpenFlamingo-7B BLIP-2-6.7B VideoChat-7B LLaVA-1.5-7B MiniGPT-4-7B InstructBLIP-7B LLaVA-13B-Llama2
Builds on: Ego4D Set-of-Mark
Closest papers by meaning
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.90)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark (CVPR 2025 · similarity 0.89)
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.88)
- LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.88)
- WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.