Research map / Vision-language and multimodal / Visual question answering
Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces multi-hop video QA with temporal evidence grounding in long egocentric videos, builds an automatically generated instruction-tuning dataset and a manually verified benchmark (MULTIHOP-EGOQA), and proposes GeLM, an MLLM with grounding tokens and a temporal grounding module.
Problem
Existing egocentric video QA and grounding benchmarks assume single-span evidence, so models cannot answer multi-hop questions that require localizing and reasoning over multiple scattered time intervals in long videos.
Approach
Mine multi-hop QA triplets from Ego4D narrations via syntax-tree-based action scene graphs and LLM generation, then add special grounding token pairs to an MLLM whose hidden states drive a saliency and a similarity branch to predict multiple evidence intervals.
Key results
- GeLM-7B achieves mIoU 16.7 and IoU@0.3 18.2 on MULTIHOP-EGOQA, versus 12.2 mIoU for GPT-4o and 9.7 for the caption-LLM pipeline, with QA score 4.8 (human 7.5).
- GeLM-7B reaches 35.4 mIoU and 31.0 P@0.5 on ActivityNet-RTL, exceeding LITA-13B (28.6 mIoU, 25.9 P@0.5).
- Adding the NCE and BCE losses with the similarity branch lifts IoU@0.3 from 13.8 to 18.2 and mIoU from 14.2 to 16.7 compared to single-loss variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GeLM on the automatically collected multi-hop instruction data | GeLM-7B · MultiHop-EgoQA | mIoU: 16.7 | Helped |
| Fine-tune TimeChat on the collected instructions (ablation) | TimeChat · MultiHop-EgoQA | mIoU: 3.6 -> 8.1 | Helped |
| Increase GeLM training data from 25% to 50% (ablation) | GeLM · MultiHop-EgoQA | mIoU: 11.3 -> 16.1 | Helped |
| Increase GeLM training data from 50% to 100% (ablation) | GeLM · MultiHop-EgoQA | mIoU: 16.1 -> 16.7 | Helped |
| Add BCE saliency loss alongside NCE similarity loss (ablation) | GeLM similarity branch · MultiHop-EgoQA | IoU@0.3: 14.1 to 18.2 | Helped |
| Add BCE saliency loss alongside NCE similarity loss (ablation) | GeLM similarity branch · MultiHop-EgoQA | mIoU: 13.4 to 16.7 | Helped |
| Use saliency rather than similarity inference strategy with BCE + NCE (ablation) | GeLM · MultiHop-EgoQA | IoU@0.3: 18.2 -> 19.2 | Helped |
| Use saliency rather than similarity inference strategy with BCE + NCE (ablation) | GeLM · MultiHop-EgoQA | mIoU: 16.7 -> 14.7 | Hurt |
| Use saliency rather than similarity inference strategy with BCE + NCE (ablation) | GeLM · MultiHop-EgoQA | Score: 4.8 -> 4.7 | Hurt |
| Use the similarity branch rather than the saliency branch on ActivityNet-RTL (ablation) | GeLM-7B · ActivityNet-RTL | mIoU: 31.8 -> 35.4 | Helped |
| Use the similarity branch rather than the saliency branch on ActivityNet-RTL (ablation) | GeLM-7B · ActivityNet-RTL | R. Score: 45.3 -> 45.1 | No clear effect |
| Train GeLM on ActivityNet-RTL rather than use LITA-13B | GeLM-7B Similarity · ActivityNet-RTL | mIoU: 28.6 -> 35.4 | Helped |
Surprising findings
- Instruction tuning on single-hop data (TimeChat, VTimeLLM) does not guarantee better multi-hop grounding, and LLaVA-NeXT-Video could not produce time-related responses.
- Only Sentence Similarity aligned with GPT-4o judgments among the open-ended metrics; CIDEr favored responses that paraphrase the question.
Limitations
- Training data are not manually refined and the benchmark is small (360 clips, 1,080 triplets).
- Grounding performance remains far below human performance (mIoU 16.7 vs human 61.8).
Experimental setup
Datasets: Ego4D narrations ActivityNet-RTL
Benchmarks: MULTIHOP-EGOQA ActivityNet-RTL
Metrics: mIoP mIoG IoU@0.3 mIoU GPT-4o score Sentence similarity P@0.5 R. Score
Baselines: GPT-4o InternVL2-8B LLaVA-NeXT-Video-7B TimeChat-7B VTimeLLM-7B LLaVA-NeXT-7B + Llama-3.1-8B pipeline LITA-7B LITA-13B
Builds on: TimeChat LITA MIL-NCE InternVideo
Closest papers by meaning
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.92)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.89)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.88)
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks (NeurIPS 2025 · similarity 0.88)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.