Research map / Video / Video-language understanding
Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoTempo, a 500-question free-form egocentric VideoQA benchmark across 10 temporal reasoning categories, showing current MLLMs score low without multi-frame temporal input.
Problem
Existing egocentric VideoQA datasets can be answered from text-only or single-frame input (commonsense shortcuts), so they do not truly test temporal understanding of multimodal LLMs.
Approach
Builds a free-form OpenQA benchmark from Ego4D clips using Gemini 1.5 Pro for QA generation followed by manual review that removes questions answerable from a single frame, and diagnoses benchmarks via text-only, single-frame, and sampling-rate comparisons.
Key results
- Gemini 1.5 Flash reaches 9.1% accuracy on EgoTempo with a single frame and 39.1% at 1FPS, a 4.3x gain versus 1.4x on EgoSchema.
- Text-only Gemini Flash achieves 57.8% on EgoTaskQA and 31.3% on EgoSchema, showing these benchmarks can be answered from commonsense.
- GPT-4o reaches 40.1% average accuracy on EgoTempo with 32 frames, and performance plateaus around 16-32 frames for weaker models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide multiple video frames instead of a single frame | Gemini Flash on EgoTempo · EgoTempo | accuracy: 9.2 -> 39.1 | Helped |
| provide multiple video frames instead of a single frame (ablation) | GPT-4o on EgoTempo · EgoTempo | accuracy: 12.9 -> 44.4 | Helped |
| provide multiple video frames instead of a single frame (ablation) | Gemini 1.5 Flash on EgoTempo · EgoTempo | accuracy: 9.1 -> 38.9 | Helped |
| use text-only input instead of a single frame (ablation) | Gemini Flash on EgoTempo · EgoTempo | accuracy: 10.0 vs 9.2 | No clear effect |
| use uniformly sampled frames in chronological order instead of random sampling (ablation) | Gemini Flash · EgoTempo | accuracy: 30.5 -> 39.1 | Helped |
| preserve chronological order instead of shuffling uniformly sampled frames (ablation) | Gemini Flash · EgoTempo | accuracy: 32.3 -> 39.1 | Helped |
| provide more frames (S = 1) rather than a single frame | Gemini Flash on EgoSchema · EgoSchema | accuracy: 50.6 -> 70.5 | Helped |
| provide more frames (S = 1) rather than a single frame | Gemini Flash on EgoTaskQA · EgoTaskQA | accuracy: 40.6 -> 54.9 | Helped |
Surprising findings
- Single-frame Gemini Flash achieves 51% on EgoSchema and 41% on EgoTaskQA, meaning about half the questions are answerable without the video.
- Adding more frames does not improve counting tasks (action and object counting) for either GPT-4o or Gemini.
Limitations
- Small benchmark with 500 question-answer pairs across 10 categories (50 each).
- Question generation relies on Gemini 1.5 Pro with manual review, so generator bias may persist.
- Evaluation relies on an LLM judge (Gemini 1.5 Pro) for open-ended answers.
Experimental setup
Datasets: Ego4D EgoTaskQA EgoSchema EgoThink
Benchmarks: EgoTempo EgoSchema EgoTaskQA EgoThink
Metrics: accuracy
Baselines: Gemini 1.5 Flash GPT-4o Claude-3.5-Sonnet Qwen2-VL LLaVA-OneVision InternLM-XC2.5 MiniCPM-V2.6 LLaVA-NeXT-Video-34B VideoLLaVA
Builds on: Ego4D EgoSchema EgoTaskQA EgoVLP
Closest papers by meaning
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.91)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.91)
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.90)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.90)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.89)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.