Research map / Video / Video-language understanding
Retrieval-Augmented Egocentric Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoInstructor, a retrieval-augmented captioner that retrieves third-person instructional videos via a cross-view retriever trained on automatically mined ego-exo pairs with a new EgoExoNCE loss, and uses them as references to caption egocentric videos.
Problem
Egocentric video captioning is limited by small egocentric datasets, while large third-person instructional video corpora are underused because of the ego-exo viewpoint and domain gap.
Approach
Learn a unified ego-exo representation space from unpaired data by pairing videos through language alignment of refined ASR narrations and Ego4D captions, train a retriever with EgoExoNCE (aligning both views to shared text features), and condition a visually-conditioned LLM on retrieved exo videos and their captions.
Key results
- Retrieval module with EgoExoNCE and refined captions improves CharadesEgo Ego2Exo R@1 from 18.3 (InfoNCE baseline) to 23.7 relative to baseline in ablation (ID-3 vs ID-7 gains on cross-view benchmarks).
- Adding retrieved exo videos (K=8, MPT-1B) raises Ego4D cooking CIDEr from 46.8 (0-shot) to 210.7, versus LaViLA-LLaMA7B at 108.9.
- On EgoLearner, one-shot LLaMA-7B with GT paired exo videos reaches CIDEr 25.1 vs 20.4 with cross-view retrieved videos and 5.2 with random videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add one retrieved exocentric instructional video as caption-generation context | EgoInstructor on Ego4d cooking validation · Ego4d cooking validation | CIDER: 54.1 -> 173.8 | Helped |
| Increase the retrieved reference count from 1 to 8 | MPT-1B captioning model · Ego4d cooking validation | CIDER: 197.3 -> 210.7 | Helped |
| Use original ASR transcripts instead of refined captions (ablation) | LLaMA-7B with one retrieved video · Ego4d cooking validation | CIDER: 166.5 vs. 173.8 | Hurt |
| Select a random exocentric video instead of retrieving a relevant one (ablation) | LLaMA-7B captioning model · EgoLearner | CIDER: 8.2 -> 5.2 | Hurt |
| Use one retrieved exocentric video as reference | LLaMA-7B captioning model · EgoLearner | CIDER: 8.2 -> 20.4 | Helped |
| Replace InfoNCE with EgoExoNCE and use refined captions (ablation) | Cross-view retrieval module · EgoLearner | Ego2Exo: 38.7 -> 46.4 | Helped |
| Replace InfoNCE with EgoExoNCE and use refined captions (ablation) | Cross-view retrieval module · YouCook2-Clip | R@1: 21.3 -> 23.7 | Helped |
| Replace InfoNCE with EgoNCE (ablation) | Cross-view retrieval module · EK100-MIR | mAP: 28.1 -> 29.0 | Helped |
| Replace InfoNCE with EgoExoNCE (ablation) | Cross-view retrieval module · EgoLearner and CharadesEgo | cross-view retrieval: ID-3 vs. ID-6 | Helped |
| Refine HowTo100M ASR transcripts into captions (ablation) | Cross-view retrieval module · EgoLearner Ego2Exo | accuracy: 41.3 -> 46.4 | Helped |
| Add cross-view visual representation alignment | VideoMAE retrieval backbone · CharadesEgo | Ego2Exo Avg. R@1/@5/@10: 18.3 -> 25.1 | Helped |
| Add cross-view visual representation alignment | VideoMAE retrieval backbone · EK100-MIR | mAP: 28.1 -> 31.6 | Helped |
Surprising findings
- Randomly selected exocentric references yield performance similar to 0-shot captioning, and can mislead the model.
Limitations
- Zero-shot transfer to EgoLearner is weak without retrieval, reflecting distribution mismatch between datasets.
- Egocentric captioning evaluated mainly on cooking subset of Ego4D.
- Exo2Ego retrieval gains are minimal on EgoLearner because ego videos come from similar environments.
Experimental setup
Datasets: Ego4D HowTo100M EK100 MIR EgoMCQ SummMCQ YouCook2 CharadesEgo EgoLearner
Benchmarks: EK100 MIR EgoMCQ SummMCQ YouCook2-Clip YouCook2-Video CharadesEgo EgoLearner Ego4D cooking captioning
Metrics: mAP nDCG accuracy R@1 R@5 R@10 BLEU-4 METEOR ROUGE-L CIDEr
Baselines: CLIP EgoVLP LaViLa InternVideo VideoMAE HierVL GPT-4V LaViLA-GPT2
Builds on: InfoNCE EgoNCE EgoVLP Flamingo-style gated cross-attention LaViLa
Closest papers by meaning
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.89)
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.89)
- Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning (CVPR 2025 · similarity 0.89)
- X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization (CVPR 2024 · similarity 0.88)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.88)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.