Research map / Video / Video-language understanding
Egocentric video understanding
Benchmarks and trains multimodal models for first-person video reasoning and grounded question answering.
46 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Video-language understanding, keyword cluster “video understanding · question · long”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
13 papers in 2023–24 and 30 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.8% to 4.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (CVPR 2017 · 73 citations)
- 2022 · Episodic Memory Question Answering (CVPR 2022 · 25 citations)
- 2023 · Egocentric Video Task Translation (CVPR 2023 · 13 citations)
- 2024 · Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · 105 citations)
- 2025 · EgoLife: Towards Egocentric Life Assistant (CVPR 2025 · 22 citations)
- 2026 · Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding (AAAI 2026 · 3 citations)
Earliest papers here
- Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (CVPR 2017 · 73 citations)
- Episodic Memory Question Answering (CVPR 2022 · 25 citations)
- EgoTaskQA: Understanding Human Tasks in Egocentric Videos (NeurIPS 2022 · 13 citations)
What has been tried
30 papers here report 321 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 7%, hurt 25%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use Abductive-CLIP instead of Sequential-CLIP trained only on positive co-occurrence pairs | MM-AU · CLIP score | 27.14 -> 27.24 | Helped | Abductive Ego-View Accident Video Understanding for Safe Driving Perception (CVPR 2024 · 34 citations) |
| Add CloseQA training to OpenQA training | QAEGO4D · Sentence Similarity | 54.9 -> 54.8 | No clear effect | Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · 24 citations) |
| Add EgoLife Day-1 fine-tuning | EgoSchema · score | 73.2 -> 75.4 | Helped | EgoLife: Towards Egocentric Life Assistant (CVPR 2025 · 22 citations) |
| Add scene-text input to video and question | EgoTextVQA-Outdoor · accuracy | 30.3 -> 52.9 | Helped | EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · 14 citations) |
| Combine action segments and salient objects as context | Ego4D v1 validation · Noun-Verb mAP | 6.14 -> 7.55 | Helped | Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation (CVPR 2024 · 14 citations) |
| provide multiple video frames instead of a single frame | EgoTempo · accuracy | 12.9 -> 44.4 | Helped | Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · 11 citations) |
| Increase tracking window from 60 to 120 frames | Nymeria test, 3pts tracking · Full joint position error (mm) | 83.88 -> 79.61 | Helped | EgoLM: Multi-Modal Language Model of Egocentric Motions (CVPR 2025 · 4 citations) |
| Add a cross-perspective reference video | Action Prediction · accuracy | 31.7 -> 39.9 (+8.2) | Helped | EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · 4 citations) |
| add sound-text module to EgoExo | EgoExoLearn Ex2Eg · top-1 accuracy | 45.30 -> 50.18 | Helped | Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues (CVPR 2025 · 2 citations) |
| Add the EgoRe-5M short-term split in SFT | EgoTaskQA · accuracy | 57.7 -> 61.6 | Helped | EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · 2 citations) |
| Add token selection | Break Eggs · F1 score | 71.84 -> 73.53 | Helped | Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations (CVPR 2025 · 1 citation) |
| Use 9 input frames rather than fewer or more frames | EgoGazeVQA · accuracy (Average) | 57.0; 57.1; 57.2; 57.1; 56.3 | Mixed | In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Ego4D · 7 of 19 papers
- Charades-Ego · 3 of 19 papers
- EPIC-Kitchens · 3 of 19 papers
- EgoExoLearn · 3 of 19 papers
- EgoExo4D · 3 of 19 papers
- EGTEA Gaze+ · 2 of 19 papers
- EgoTaskQA · 2 of 19 papers
- EgoSchema · 2 of 19 papers
Baselines
- GPT-4o · 6 of 19 papers
- Qwen2-VL · 4 of 19 papers
- LaViLa · 4 of 19 papers
- EgoVLP · 4 of 19 papers
- LLaVA-NeXT-Video · 3 of 19 papers
- MiniCPM-V2.6 · 3 of 19 papers
- Gemini-1.5-Pro · 2 of 19 papers
- LLaVA-OV · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 43 papers; from OpenAlex and the papers' affiliations).
- Meta · 5 papers
- University of Texas, Austin · 4 papers
- Zhejiang University · 4 papers
- Shanghai AI Laboratory · 3 papers
- University of Texas at Austin · 2 papers
- Politecnico di Torino · 2 papers
- ETH Zurich · 2 papers
- NVIDIA · 2 papers
- Nanyang Technological University · 2 papers
- Harbin Institute of Technology · 2 papers
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Event and anomaly reasoning in video (184 papers)
- Video LLMs with fine-grained understanding (229 papers)
- Long video understanding (236 papers)
- Video question answering (64 papers)
Elsewhere in the topic
- Temporal video grounding (88 papers)
- Video-language pretraining (82 papers)
- Dense video captioning (65 papers)
- Text-video retrieval (59 papers)
- Encoder-decoder video captioning (52 papers)
- Video moment retrieval (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).