Research map / Vision-language and multimodal / Multimodal LLMs

ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ECBench, a manually annotated egocentric video QA benchmark with 4,324 questions across 30 embodied cognition dimensions, plus the ECEval scoring scheme, and evaluates proprietary, open-source, and embodied LVLMs.

Problem

Existing embodied video QA benchmarks lack systematic coverage of embodied cognition, robot self-centric questions, scene dynamics, and embodied hallucination, so LVLMs' egocentric embodied abilities are poorly measured.

Approach

A hierarchical 30-dimension taxonomy (static scene-based and robot-centric, dynamic scene, and hallucination) with category-independent human annotation, GPT-4o blind-testing filtering of common-sense-answerable questions, and a hybrid ECEval scoring (binary for closed-ended, 0.5-anchored multi-level for open-ended).

Key results

• GPT-4o with 32 frames achieves the highest overall ECBench score of 50.35, versus 24.09 for blind GPT-4o.
• Best open-source model Qwen2VL-72B with 20 frames scores 44.62, and best native video LVLM LongVA-7B scores 40.47.
• Human performance reaches 94.96 overall, and all models perform poorly on dynamic scenes and user-input hallucination (UI scores near zero).

Measured effects reported in the paper

Surprising findings

• Blind GPT-4o without video scores 24.09, indicating a large share of answers can be guessed from common sense despite the filtering.
• Embodied LVLMs AlanaVLM (34.75) and GeLM (21.54) underperform general LVLMs, and no evaluated model handles user-input ambiguity or missing references well.

Limitations

• Could not test all proprietary models (e.g., Claude-3.5) due to budget.
• Video inputs are limited to under five minutes, so long-horizon visual memory is not well tested.
• Dynamic-scene videos are relatively few (40 videos, 248 QA) and lack scene and object diversity.

Experimental setup

Datasets: ScanNet MultiScan HM3D

Benchmarks: ECBench

Metrics: ECEval score (binary for closed-ended, 0.5-anchored 0-1 scal blind LLM baseline

Baselines: GPT-4o GPT-4o-mini GPT-4v Qwen2VL-72B LongVA-7B InternVL2-40B Idefics3-8B Kangaroo-8B Video-LLaVA-7B Video-LLaMA2-7B AlanaVLM-7B GeLM-7B

Builds on: OpenEQA MVBench MMBench-Video

Closest papers by meaning

• EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.89)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
• EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
• NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.88)
• Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.88)