Research map / Video / Video-language understanding
Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a model-agnostic VA3 framework (hierarchical video aligner plus QDG-based answer aggregator with contrastive losses) that improves accuracy and compositional consistency of existing VidQA models, along with new c-F1 consistency metrics and an LLM-based question decomposition pipeline.
Problem
Existing video question-answering models act as black boxes and show poor compositional consistency, answering main questions correctly without correctly answering their sub-questions, and existing consistency metrics (CA, RWR, Delta) are asymmetric and unstable.
Approach
A hierarchical video aligner (object, appearance, motion levels) selects question-relevant clips, and a graph attention answer aggregator passes information along question decomposition graph edges, trained with a contrastive loss on edge types; metrics are redefined as balanced cP/cR/c-F1.
Key results
- VA3(HME) improves overall main accuracy by 2.71% (41.59 to 44.30) and c-F1 by 3.54% on AGQA-Decomp.
- VA3 improves MSVD accuracy by 3.23% to 4.76% and NExT-QA by 3.58% to 4.51% across the three baselines.
- On the novel composition setting, VA3(HGA) raises c-F1 by 5.03% over HGA (35.44 to 40.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VA3 to HME | HME · AGQA-Decomp balanced | overall main-question accuracy: +2.71 | Helped |
| Add VA3 to HGA | HGA · AGQA-Decomp balanced | overall main-question accuracy: +1.27 | Helped |
| Add VA3 to HQGA | HQGA · AGQA-Decomp balanced | overall main-question accuracy: +1.23 | Helped |
| Add VA3 to HME | HME · AGQA-Decomp novel composition | accuracy: +1.91 | Helped |
| Add VA3 to HGA | HGA · AGQA-Decomp novel composition | accuracy: +1.87 | Helped |
| Add VA3 to HQGA | HQGA · AGQA-Decomp novel composition | accuracy: +2.12 | Helped |
| Add VA3 to HME | HME · AGQA-Decomp more composition steps | accuracy: +1.79 | Helped |
| Add VA3 to HGA | HGA · AGQA-Decomp more composition steps | accuracy: +1.12 | Helped |
| Add VA3 to HQGA | HQGA · AGQA-Decomp more composition steps | accuracy: +1.27 | Helped |
| Add VA3 with automatic question decomposition | HQGA · MSVD | accuracy: +3.23 | Helped |
| Add VA3 with automatic question decomposition | HQGA · NExT-QA | accuracy: +3.58 | Helped |
| Replace EIGV with the proposed hierarchical video aligner (ablation) | HME · AGQA-Decomp balanced | overall main-question accuracy | Helped |
Surprising findings
- Video aligner alone substantially improves accuracy but helps little on compositional consistency.
- Existing CA, RWR and Delta metrics give inconsistent rankings for models with identical consistency, motivating the new c-F1 metrics.
Limitations
- Gains on the main question are smaller than on sub-questions, since the aligner struggles with main questions that need the whole video.
- Automatic decomposition quality depends on LLM prompting with hand-built candidate examples from AGQA-Decomp.
Experimental setup
Datasets: AGQA-Decomp MSVD NExT-QA
Benchmarks: AGQA-Decomp MSVD NExT-QA
Metrics: accuracy consistency precision (cP) consistency recall (cR) consistency F1 (c-F1) negative consistency F1 (Nc-F1) CA RWR Delta
Baselines: HME HGA HQGA EIGV
Builds on: HME HGA HQGA EIGV AGQA-Decomp
Closest papers by meaning
- Video-Context Aligned Transformer for Video Question Answering (AAAI 2024 · similarity 0.90)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.90)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.88)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.88)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.