Research map / Video / Video-language understanding
Video-Context Aligned Transformer for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V-CAT, a transformer VideoQA model that uses all questions about a video as a context to align video and text semantically (contrastive loss) and content (cross-attention), then answers the current question, reaching SOTA on MSVD-QA and MSRVTT-QA.
Problem
Video question answering suffers from imbalanced information between a richly informative video and a short question, causing unstable semantic and content alignment and reasoning.
Approach
Concatenate all questions related to a video into a context that balances textual and visual information; align video and context globally with contrastive learning, use the context to select relevant video content, then decode the answer from the current question attending to refined video features.
Key results
- V-CAT reaches 45.2% accuracy on MSVD-QA and 43.3% on MSRVTT-QA, exceeding the prior best EIGV by 2.6 and 4 points.
- On MSVD-QA, V-CAT exceeds CoMVT by 2.6 points and on MSRVTT-QA it exceeds MERLOT by 0.2 points, without any dataset pretraining.
- Using only the current question as context drops accuracy to 31.3 (MSVD) and 31.5 (MSRVTT); removing context entirely gives 35.7 and 34.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use V-CAT instead of the strongest baseline EIGV | Video question answering · MSVD-QA | accuracy: 42.6 -> 45.2 | Helped |
| use V-CAT instead of the strongest baseline MERLOT | Video question answering · MSRVTT-QA | accuracy: 43.1 -> 43.3 | Helped |
| add video context (ablation) | V-CAT · MSVD-QA | accuracy: 35.7 -> 45.2 | Helped |
| add video context (ablation) | V-CAT · MSRVTT-QA | accuracy: 34.2 -> 43.3 | Helped |
| use video context rather than the current question as context (ablation) | V-CAT · MSVD-QA | accuracy: 31.3 -> 45.2 | Helped |
| use video context rather than the current question as context (ablation) | V-CAT · MSRVTT-QA | accuracy: 31.5 -> 43.3 | Helped |
| add semantic alignment (ablation) | V-CAT · MSVD-QA | accuracy: 45.1 -> 45.2 | Helped |
| add semantic alignment (ablation) | V-CAT · MSRVTT-QA | accuracy: 43.0 -> 43.3 | Helped |
| add content alignment (ablation) | V-CAT · MSVD-QA | accuracy: 37.2 -> 45.2 | Helped |
| add content alignment (ablation) | V-CAT · MSRVTT-QA | accuracy: 36.6 -> 43.3 | Helped |
| add contrastive learning loss (ablation) | V-CAT · MSVD-QA | accuracy: 45.1 -> 45.2 | Helped |
| add contrastive learning loss (ablation) | V-CAT · MSRVTT-QA | accuracy: 43.0 -> 43.3 | Helped |
Surprising findings
- Using only the current question as context performs worse than using no context at all.
- Replacing the contrastive loss with L1, L2, or KL divergence reduces accuracy below the no-contrastive-loss variant.
Limitations
- Relies on existing question sets to construct context; alternative context sources are not explored.
- Uses traditional frame features (ResNet/ResNeXt) rather than modern video encoders, and evaluation is limited to two open-ended datasets.
- Hyperparameters such as layer counts and loss weight were searched per dataset, so results depend on dataset-specific tuning.
Experimental setup
Datasets: MSVD-QA MSRVTT-QA
Benchmarks: MSVD-QA MSRVTT-QA
Metrics: accuracy
Baselines: HCRN B2A HAIR CLIP-QA MHN HQGA EIGV CoVGT PMT HD-VILA MERLOT CoMVT
Builds on: Transformer CLIP reading comprehension-inspired VideoQA
Closest papers by meaning
- Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering (CVPR 2024 · similarity 0.90)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.88)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.88)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.87)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.