Research map / Video / Video-language understanding
EgoLife: Towards Egocentric Life Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the EgoLife project: a week-long, six-person multimodal egocentric dataset, the EgoLifeQA long-context QA benchmark, and the EgoButler system (EgoGPT fine-tuned omni-modal captioner plus EgoRAG retrieval).
Problem
Current egocentric datasets and models cover only short clips and single perspectives, so they cannot support ultra-long, multi-person, life-oriented assistance such as recalling past events, tracking habits, and recognizing individuals over days.
Approach
Collect a week-long, multi-person, multi-view egocentric dataset with audio, transcripts and dense visual-audio captions, pair it with long-context life-oriented QA, and combine an egocentric audio-visual model (EgoGPT) with hierarchical retrieval memory (EgoRAG) to answer questions over days of footage.
Key results
- EgoGPT (EgoIT-99K+D1) reaches 36.0 average on EgoLifeQA versus 36.9 for Gemini-1.5-Pro and 36.2 for GPT-4o, with EgoGPT best on RelationMap (39.7 TaskMaster and 39.2 EntityLog are its top entries).
- EgoGPT+EgoRAG scores 35.4 on questions with certificate length over 24 hours, versus 25.0 for EgoGPT alone and 18.4 for Gemini-1.5-Pro.
- EgoGPT (EgoIT-99K+D1) reaches 75.4 on EgoSchema with 32 frames, the highest among compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EgoLife Day-1 fine-tuning (ablation) | EgoGPT (EgoIT) · EgoSchema | score: 73.2 -> 75.4 | Helped |
| Add EgoLife Day-1 fine-tuning (ablation) | EgoGPT (EgoIT) · EgoPlan | score: 32.4 -> 33.4 | Helped |
| Add EgoLife Day-1 fine-tuning (ablation) | EgoGPT (EgoIT) · EgoThink | score: 61.7 -> 61.4 | Hurt |
| Add EgoRAG retrieval (ablation) | EgoGPT · EgoLifeQA >24h | score: 25.0 -> 35.4 | Helped |
| Use audio-only captions instead of visual-and-audio captions (ablation) | EgoGPT · EgoLifeQA | Average: 33.1 -> 27.2 | Hurt |
| Use visual-only captions instead of visual-and-audio captions (ablation) | EgoGPT · EgoLifeQA | Average: 33.1 -> 31.2 | Hurt |
| Add audio to visual-only captions (ablation) | EgoGPT (EgoIT-99K) · EgoLifeQA | Average: 31.2 -> 33.1 | Helped |
| Add Day-1 fine-tuning (ablation) | EgoGPT (Audio Only) · EgoLifeQA | Average: 27.2 -> 28.1 | Helped |
| Add Day-1 fine-tuning (ablation) | EgoGPT (Visual Only) · EgoLifeQA | Average: 31.2 -> 33.6 | Helped |
| Add Day-1 fine-tuning (ablation) | EgoGPT (Visual+Audio) · EgoLifeQA | Average: 33.1 -> 36.0 | Helped |
| Use human visual-audio captions rather than EgoGPT captions (ablation) | EgoButler · EgoLifeQA | Average: 36.0 -> 45.5 | Helped |
| Use transcript-only memory rather than EgoGPT visual-and-audio captions (ablation) | EgoButler · EgoLifeQA | Average: 36.0 -> 29.6 | Hurt |
Surprising findings
- Gemini-1.5-Pro scores highest on the EgoLifeQA average (36.9) in Table 5, above EgoGPT (36.0), despite EgoGPT's personalization advantages.
- Audio-only captions give the weakest retrieval performance (27.2), and human-written narrations alone score lower (31.5) than human-verified visual-audio captions (45.5).
Limitations
- EgoGPT struggles with speech comprehension, especially emotions and laughter, and overfits to Day-1 data, misidentifying individuals.
- EgoRAG lacks multi-step reasoning and fails when key information is missing from retrieved clips.
- Evaluation on EgoLifeQA is a quick run on one participant's 500 QAs in this version.
Experimental setup
Datasets: EgoIT-99K EgoLife Ego4D Charades-Ego HoloAssist EGTEA Gaze+ IndustReal EgoTaskQA EgoProceL Epic-Kitchens ADL LibriSpeech
Benchmarks: EgoLifeQA EgoSchema EgoPlan-Bench EgoThink
Metrics: accuracy
Baselines: GPT-4o Gemini-1.5-Pro LLaVA-OV LLaVA-Next-Video LongVA Qwen2-VL Oryx IXC-2.5 InternVideo2
Builds on: LLaVA-OneVision Ola EgoSchema Ego4D
Closest papers by meaning
- LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.88)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.88)
- ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.87)
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.86)
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.86)
- EgoBlind: Towards Egocentric Visual Assistance for the Blind (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.