Research map / Video / Video-language understanding
Grounded Question-Answering in Long Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GroundVQA, a unified Flan-T5 model that jointly grounds and answers questions in long egocentric videos, plus an LLM-based pipeline (EgoTimeQA, 303K samples) and a close-ended CloseQA evaluation.
Problem
Question answering in long egocentric videos requires temporally grounding the query and generating answers, but chained grounding-then-answering suffers from error propagation, annotated grounded QA data is scarce, and open-ended answers are hard to evaluate.
Approach
A single multi-task model jointly trains video-language grounding (temporal localizer) and open-ended/close-ended answer generation over shared visual-language encoder features, with training data synthesized from Ego4D narrations by an LLM (Llama2-13B) and a multiple-choice CloseQA task with LLM-generated distractors for evaluation.
Key results
- GroundVQA-Base reaches 58.2 sentence similarity and 50.2 CloseQA accuracy on QAEGO4D, versus 41.4 for the best reproduced baseline (SimpleVQA+*).
- EgoTimeQA pre-training raises GroundVQA-Small sentence similarity from 55.6 to 57.7 and CloseQA accuracy from 40.8 to 48.7 on QAEGO4D/QAEGO4DClose.
- On NLQv2 test, GroundVQA-Base with NaQ and NLQv2 pre-training obtains Mean R@1 of 22.15, close to GroundNLQ's 20.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CloseQA training to OpenQA training (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 54.9 -> 54.8 | No clear effect |
| Add VLG training (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 54.8 -> 55.6 | Helped |
| Add EGOTIMEQA training data (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 54.8 -> 56.1 | Helped |
| Add EGOTIMEQA training data and VLG training (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 56.1 -> 57.7 | Helped |
| Add VLG training (ablation) | GroundVQAS · QAEGO4DClose | Accuracy: 47.2±0.5 -> 48.7±0.4 | Helped |
| Use question-relevant video segments as input, bypassing temporal grounding (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 57.7 -> 58.4 | Helped |
| Apply VLG supervision to decoder cross-attention (ablation) | SimpleVQA+S · QAEGO4D | Sentence Similarity: 54.9 -> 54.7 | Hurt |
| Train one unified model instead of separate grounding and QA models (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 54.7 -> 55.6 | Helped |
| Train one unified model instead of separate grounding and QA models, with EGOTIMEQA (ablation) | GroundVQAS · QAEGO4D | Sentence Similarity: 56.0 -> 57.7 | Helped |
| Add EGOTIMEQA to QAEGO4D training (ablation) | GroundVQAS · NLQv2 val | Mean Recall@1: 8.8 -> 18.4 | Helped |
| Add NLQv2 and NaQ training data (ablation) | GroundVQAS · NLQv2 val | Mean Recall@1: 18.4 -> 20.9 | Helped |
Surprising findings
- Adding the VLG loss directly to the LM decoder's cross-attention (SimpleVQA+) hurt QA performance, suggesting the grounding task exceeds the cross-attention capacity.
Limitations
- Open-ended answers remain hard to evaluate, and the QAEGO4D and NLQ annotations often have multiple plausible answers or segments.
- Performance gap to the oracle upper bound is mainly due to imperfect temporal grounding.
Experimental setup
Datasets: Ego4D NLQv2 NaQ QAEGO4D EgoTimeQA
Benchmarks: QAEGO4D QAEGO4DClose NLQv2
Metrics: sentence similarity ROUGE-L METEOR Accuracy Recall@1 Recall@5 IoU
Baselines: BlindVQA SimpleVQA SimpleVQA+ Rehearsal Memory VSLNet EgoVLP ReLER NaQ++ GroundNLQ
Builds on: ActionFormer GroundNLQ EgoVLP Flan-T5 QAEGO4D
Closest papers by meaning
- Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos (AAAI 2025 · similarity 0.92)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.90)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.90)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.89)
- Can I Trust Your Answer? Visually Grounded Video Question Answering (CVPR 2024 · similarity 0.88)
- Multi-Factor Adaptive Vision Selection for Egocentric Video Question Answering (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.