Research map / Video / Video-language understanding
Multi-Factor Adaptive Vision Selection for Egocentric Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MFAS, an egocentric VideoQA framework combining multi-scale patch partition/merging in a TimeSformer, prior-guided patch selection, and a question-guided hierarchical cross-attention aggregation network, evaluated on EgoTaskQA and QAEgo4D.
Problem
Egocentric video question answering faces small object recognition, noise from redundant regions, and difficulty with spatial-temporal reasoning from a first-person view.
Approach
Extends TimeSformer with a twin sub-patch/patch branch (multi-scale fusion with learnable alpha/beta), selects sub-patches using a hand-object gaze prior combined with attention scores and temporal neighborhood propagation, and aggregates video and question features via a multi-layer spatial-temporal cross-attention network.
Key results
- On EgoTaskQA indirect split, MFAS reaches 45.40 overall accuracy vs 42.28 for EgoVLPv2 (+3.12).
- On EgoTaskQA direct split, MFAS reaches 48.69 overall accuracy vs 46.26 for EgoVLPv2 (+2.43).
- On QAEgo4D, MFAS† (with ranking loss) reaches 12.7 accuracy vs 11.9 for EgoVLPv2†; ablation removing prior-guided selection drops indirect accuracy by 1.45 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MFAS instead of EgoVLPv2 | EgoTaskQA direct split · EgoTaskQA direct split | All accuracy: 46.26 -> 48.69 | Helped |
| use MFAS instead of EgoVLPv2 | EgoTaskQA indirect split · EgoTaskQA indirect split | All accuracy: 42.28 -> 45.40 | Helped |
| use MFAS† instead of EgoVLPv2† | QAEgo4D discriminative setting · QAEgo4D discriminative | Accuracy: 11.9 -> 12.7 | Helped |
| use MFAS† instead of SimpleVQA† | QAEgo4D generative setting · QAEgo4D generative | Accuracy: 9.7 -> 11.9 | Helped |
| add the patch partition and merging module (ablation) | MFAS · EgoTaskQA indirect split | All accuracy: 44.71 -> 45.40 | Helped |
| add the prior-guided patch selection module (ablation) | MFAS · EgoTaskQA indirect split | All accuracy: 43.95 -> 45.40 | Helped |
| add the hierarchical aggregation network (ablation) | MFAS · EgoTaskQA indirect split | All accuracy: 44.52 -> 45.40 | Helped |
Limitations
- Evaluated only on two egocentric VideoQA datasets
- Generative answers can diverge from ground truth despite being contextually plausible
- Prior matrix is hand-designed from gaze/hand-activity observations, which may not transfer across settings
Experimental setup
Datasets: EgoTaskQA QAEgo4D
Benchmarks: EgoTaskQA QAEgo4D
Metrics: accuracy BLEU METEOR ROUGE
Baselines: HCRN CMCIR EgoVLP EgoVLPv2 PSAC HME VisualBERT ClipBERT Longformer JustAsk SimpleVQA BlindVQA
Builds on: TimeSformer EgoVLPv2 RoBERTa HCRN
Closest papers by meaning
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.88)
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.88)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.87)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.86)
- Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos (AAAI 2025 · similarity 0.86)
- Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.