Research map / Vision-language and multimodal / Multimodal LLMs
EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EOC-Bench, a human-annotated egocentric video QA benchmark (3,277 QA pairs over 656 videos) testing object-centric Past/Present/Future cognition with visual object prompts and a multi-scale temporal accuracy metric, and evaluates proprietary, open-source, and object-level MLLMs.
Problem
Existing embodied video QA benchmarks focus on static scenes and appearance/spatial attributes, and do not systematically evaluate object-level dynamics, memory, and forecasting in dynamic egocentric interactions.
Approach
A taxonomy of 11 object-centric tasks across Past, Present and Future, with mixed question formats (true/false, single-choice, multiple-choice, open-ended timestamp questions), visual prompts (point, box, mask) for object referencing, and a multi-scale relative-tolerance temporal accuracy metric (MSTA) calibrated by human error analysis.
Key results
- GPT-4o achieves the best overall mean of 61.83% on EOC-Bench, versus 94.63% for human participants.
- The best open-source model, InternVL2.5-78B, reaches 52.33% mean, trailing GPT-4o by about 9.5 points.
- Absolute time perception remains weak: the best open-source open-ended score is 24.02% (LLaVA-Video-72B), just above random-level performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase input from 1 frame to 32 frames (ablation) | GPT-4o · EOC-Bench | Mean accuracy: 49.6 -> 61.8; 24.6% | Helped |
| increase input from 1 frame to 32 frames (ablation) | GPT-4o · EOC-Bench Past | Accuracy: 36.8 -> 54.9; 49.2% | Helped |
| increase input from 1 frame to 32 frames (ablation) | Gemini-2.0-flash · EOC-Bench | Mean accuracy: 47.8 -> 57.4; 20.1% | Helped |
| increase input from 1 frame to 32 frames (ablation) | Gemini-2.0-flash · EOC-Bench Past | Accuracy: 29.9 -> 47.9; 60.2% | Helped |
| increase input from 1 frame to 32 frames (ablation) | InternVL2.5-78B · EOC-Bench | Mean accuracy: 47.6 -> 52.3; 9.9% | Helped |
| increase input from 1 frame to 32 frames (ablation) | VideoLLaMA3-7B · EOC-Bench | Mean accuracy: 42.1 -> 46.2; 10.4% | Helped |
| use point object prompt instead of box prompt (ablation) | InternVL2.5-8B · EOC-Bench | Mean accuracy: 42.05 -> 41.83 | Hurt |
| use mask object prompt instead of box prompt (ablation) | InternVL2.5-8B · EOC-Bench | Mean accuracy: 42.05 -> 42.12 | Helped |
| use point object prompt instead of box prompt (ablation) | Qwen2.5-VL-7B · EOC-Bench | Mean accuracy: 43.13 -> 41.21 | Hurt |
| use mask object prompt instead of box prompt (ablation) | Qwen2.5-VL-7B · EOC-Bench | Mean accuracy: 43.13 -> 41.31 | Hurt |
| use point object prompt instead of box prompt (ablation) | VideoLLaMA3-7B · EOC-Bench | Mean accuracy: 46.04 -> 45.24 | Hurt |
| use mask object prompt instead of box prompt (ablation) | VideoLLaMA3-7B · EOC-Bench | Mean accuracy: 46.04 -> 46.10 | No clear effect |
Surprising findings
- Many smaller MLLMs score below random guessing on multiple-choice questions, suggesting overfitting to single-choice formats.
- Multi-frame input gives large gains for GPT-4o on past-oriented questions (49.2% relative gain from 1 to 32 frames), yet many open models gain little.
Limitations
- Videos are limited to under six minutes, so long-term memory is not well tested.
- Evaluation uses fixed frame sampling (1 to 32 frames), which may miss memory-relevant content.
- Human error analysis for metric thresholds is based on only three volunteers.
Experimental setup
Datasets: EPIC-KITCHENS Ego4D Charades-Ego MECCANO
Benchmarks: EOC-Bench
Metrics: accuracy MSTA (multi-scale temporal accuracy)
Baselines: Random Human GPT-4o-mini Gemini-2.0-flash VideoLLaVA LongVA NVILA VideoLLaMA2 Qwen2.5-VL InternVL2.5 LLaVA-OneVision LLaVA-Video
Builds on: SoM
Closest papers by meaning
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.90)
- EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.89)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.88)
- Is `Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning (CVPR 2025 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.