Research map / Video / Video-language understanding
MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 7M-sample egocentric QA dataset from Ego4D narrations via a GPT-4o text-only data engine, a 629-video EgoMemoria benchmark, and MM-Ego, a LLaVA-OV-7B-based video MLLM with Memory Pointer Prompting for question-aware frame selection.
Problem
Egocentric video understanding of long videos (seconds to one hour) lacks large-scale QA training data, suitable benchmarks, and efficient MLLM designs that can handle many frames without prohibitive token counts.
Approach
Generate episodic-memory QA from human narrations with a text-only LLM, then use a two-step "global glimpse" (question-aware pointer scoring over compressed per-frame embeddings) and "fallback" (feeding top-k high-resolution frame embeddings to the LLM) mechanism, plus a debiased evaluation that removes questions answerable without video.
Key results
- MM-Ego achieves 61.27 MDA on EgoMemoria versus 47.32 for LLaVA-OV 7B, with the largest gains in medium and long videos.
- MM-Ego reaches 69.03 on EgoSchema, a +8.18 gain over LLaVA-NeXT-Video-32B-Qwen (60.85).
- On Video-MME (w/o subs), MM-Ego scores 57.00 overall versus 58.30 for LLaVA-OV 7B, roughly on par.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA-OV on the MM-Ego SFT data mixture | LLaVA-OV · EgoMemoria MDA | accuracy: +8.65 | Helped |
| Add Memory Pointer Prompting | Ego SFT · EgoMemoria MDA | accuracy: +13.95 | Helped |
| Train using egocentric QA data only instead of the comprehensive data mixture (ablation) | LLaVA-OV · EgoMemoria MDA | accuracy: 55.97 -> 58.11 | Helped |
| Train using egocentric QA data only instead of the comprehensive data mixture (ablation) | MM-Ego · EgoMemoria MDA | accuracy: 61.27 -> 63.78 | Helped |
| Reduce sampled frames from 32 to 16 (ablation) | MM-Ego · EgoMemoria MDA | accuracy: 61.27 -> 60.93 | Hurt |
| Reduce sampled frames from 32 to 8 (ablation) | MM-Ego · EgoMemoria MDA | accuracy: 61.27 -> 58.55 | Hurt |
| Reduce sampled frames from 32 to 4 (ablation) | MM-Ego · EgoMemoria MDA | accuracy: 61.27 -> 58.46 | Hurt |
| Increase sampled frames from 4 to 8 (ablation) | MM-Ego · EgoMemoria MDA | accuracy: 58.46 -> 58.55 | Helped |
Surprising findings
- Without visual input, many questions are answerable from text alone, so original accuracy overstates egocentric understanding (LLaVA-OV drops from 65.45 to 47.32 after debiasing).
Limitations
- Debiasing relies on excluding questions answerable without video, and the benchmark is generated by GPT-4o from narrations rather than human-written QA.
- Video-MME performance is below LLaVA-OV 7B, and the training data mixture is less diverse.
- Model currently handles a few hundred frames; scaling to thousands of frames is future work.
Experimental setup
Datasets: Ego4D narrations Ego4D episodic memory LLaVA-NeXT SFT collection ShareGPT4V ShareGPT-4o ALLaVA ShareGPT4Video NExT-QA ActivityNet-QA ScienceQA
Benchmarks: EgoMemoria EgoSchema Video-MME
Metrics: accuracy mean debiased accuracy (MDA)
Baselines: LLaVA-OV 7B Ego SFT GPT-4o LLaVA-NeXT-Video-7B-DPO LLaVA-NeXT-Video-32B-Qwen
Builds on: LLaVA-OV Ego4D episodic memory Pointer Networks SigLIP
Closest papers by meaning
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.91)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.89)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos (ICLR 2025 · similarity 0.88)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.